first commit
JustinLin610
1 year ago
commit
ba2d85a13b
@ -0,0 +1,11 @@
|
||||
__pycache__
|
||||
*.so
|
||||
build
|
||||
.coverage_*
|
||||
*.egg-info
|
||||
*~
|
||||
.vscode/
|
||||
.idea/
|
||||
.DS_Store
|
||||
|
||||
/private/
|
||||
@ -0,0 +1,27 @@
|
||||
------------- LICENSE FOR NVIDIA Megatron-LM code --------------
|
||||
|
||||
Copyright (c) 2022, NVIDIA CORPORATION. All rights reserved.
|
||||
|
||||
Redistribution and use in source and binary forms, with or without
|
||||
modification, are permitted provided that the following conditions
|
||||
are met:
|
||||
* Redistributions of source code must retain the above copyright
|
||||
notice, this list of conditions and the following disclaimer.
|
||||
* Redistributions in binary form must reproduce the above copyright
|
||||
notice, this list of conditions and the following disclaimer in the
|
||||
documentation and/or other materials provided with the distribution.
|
||||
* Neither the name of NVIDIA CORPORATION nor the names of its
|
||||
contributors may be used to endorse or promote products derived
|
||||
from this software without specific prior written permission.
|
||||
|
||||
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY
|
||||
EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
||||
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
|
||||
PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
|
||||
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
|
||||
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
|
||||
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
|
||||
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
|
||||
OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
|
||||
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
||||
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 107 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 28 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 302 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 615 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 376 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 444 KiB |
Binary file not shown.
Binary file not shown.
|
After Width: | Height: | Size: 139 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 1.3 MiB |
@ -0,0 +1,45 @@
|
||||
## 评测复现
|
||||
|
||||
- CEVAL
|
||||
|
||||
```Shell
|
||||
wget https://huggingface.co/datasets/ceval/ceval-exam/resolve/main/ceval-exam.zip
|
||||
mkdir data/ceval
|
||||
mv ceval-exam.zip data/ceval
|
||||
cd data/ceval; unzip ceval-exam.zip
|
||||
cd ../../
|
||||
python evaluate_ceval.py -d data/ceval/
|
||||
```
|
||||
|
||||
- MMLU
|
||||
|
||||
```Shell
|
||||
wget https://people.eecs.berkeley.edu/~hendrycks/data.tar
|
||||
mkdir data/mmlu
|
||||
mv data.tar data/mmlu
|
||||
cd data/mmlu; tar xf data.tar
|
||||
cd ../../
|
||||
python evaluate_mmlu.py -d data/mmlu/data/
|
||||
```
|
||||
|
||||
- HumanEval
|
||||
|
||||
Get the HumanEval.jsonl file from [here](https://github.com/openai/human-eval/tree/master/data)
|
||||
|
||||
```Shell
|
||||
python evaluate_humaneval.py -f HumanEval.jsonl -o HumanEval_res.jsonl
|
||||
git clone https://github.com/openai/human-eval
|
||||
pip install -e human-eval
|
||||
evaluate_functional_correctness HumanEval_res.jsonl
|
||||
```
|
||||
|
||||
When installing package human-eval, please note its following disclaimer:
|
||||
|
||||
This program exists to run untrusted model-generated code. Users are strongly encouraged not to do so outside of a robust security sandbox. The execution call in execution.py is deliberately commented out to ensure users read this disclaimer before running code in a potentially unsafe manner. See the comment in execution.py for more information and instructions.
|
||||
|
||||
|
||||
- GSM8K
|
||||
|
||||
```Shell
|
||||
python evaluate_gsm8k.py
|
||||
```
|
||||
@ -0,0 +1,263 @@
|
||||
import os
|
||||
import pandas as pd
|
||||
import numpy as np
|
||||
import argparse
|
||||
import datasets
|
||||
import torch
|
||||
|
||||
from typing import List
|
||||
from tqdm import tqdm
|
||||
from transformers.trainer_utils import set_seed
|
||||
|
||||
|
||||
'''
|
||||
wget https://huggingface.co/datasets/ceval/ceval-exam/resolve/main/ceval-exam.zip
|
||||
mkdir data/ceval
|
||||
mv ceval-exam.zip data/ceval
|
||||
cd data/ceval; unzip ceval-exam.zip
|
||||
cd ../../
|
||||
python evaluate_ceval.py -d data/ceval/
|
||||
'''
|
||||
|
||||
def load_models_tokenizer(args):
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
from transformers.generation import GenerationConfig
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(args.checkpoint_path, trust_remote_code=True)
|
||||
model = AutoModelForCausalLM.from_pretrained(args.checkpoint_path, device_map="auto", trust_remote_code=True).eval()
|
||||
model.generation_config = GenerationConfig.from_pretrained(args.checkpoint_path, trust_remote_code=True)
|
||||
return model, tokenizer
|
||||
|
||||
|
||||
def format_example(line, include_answer=True):
|
||||
example = '问题：' + line['question']
|
||||
for choice in choices:
|
||||
example += f'\n{choice}. {line[f"{choice}"]}'
|
||||
|
||||
if include_answer:
|
||||
example += '\n答案：' + line["answer"] + '\n\n'
|
||||
else:
|
||||
example += '\n答案：'
|
||||
return example
|
||||
|
||||
|
||||
def generate_few_shot_prompt(k, subject, dev_df):
|
||||
prompt = ''
|
||||
if k == -1:
|
||||
k = dev_df.shape[0]
|
||||
for i in range(k):
|
||||
prompt += format_example(
|
||||
dev_df.iloc[i, :],
|
||||
include_answer=True,
|
||||
)
|
||||
return prompt
|
||||
|
||||
|
||||
def get_logits(tokenizer, model, inputs: List[str]):
|
||||
input_ids = tokenizer(inputs, padding=False)['input_ids']
|
||||
input_ids = torch.tensor(input_ids, device=model.device)
|
||||
tokens = {'input_ids': input_ids}
|
||||
|
||||
outputs = model(input_ids)['logits']
|
||||
logits = outputs[:, -1, :]
|
||||
log_probs = torch.nn.functional.softmax(logits, dim=-1)
|
||||
return log_probs, {'tokens': tokens}
|
||||
|
||||
|
||||
@torch.no_grad()
|
||||
def eval_subject(
|
||||
model,
|
||||
tokenizer,
|
||||
subject_name,
|
||||
test_df,
|
||||
k=5,
|
||||
dev_df=None,
|
||||
few_shot=False,
|
||||
save_result_dir=None,
|
||||
**kwargs
|
||||
):
|
||||
result = []
|
||||
score = []
|
||||
|
||||
few_shot_prompt = generate_few_shot_prompt(
|
||||
k, subject_name, dev_df) if few_shot else []
|
||||
all_probs = {'prob_A': [], 'prob_B': [], 'prob_C': [], 'prob_D': []}
|
||||
if args.debug: print(f"few_shot_prompt: {few_shot_prompt}")
|
||||
|
||||
for _, row in tqdm(test_df.iterrows(), total=len(test_df)):
|
||||
question = format_example(row, include_answer=False)
|
||||
full_prompt = few_shot_prompt + question
|
||||
|
||||
output, input_info = get_logits(tokenizer, model, [full_prompt])
|
||||
assert output.shape[0] == 1
|
||||
logits = output.flatten()
|
||||
|
||||
softval = torch.nn.functional.softmax(
|
||||
torch.tensor(
|
||||
[
|
||||
logits[tokenizer("A")['input_ids']],
|
||||
logits[tokenizer("B")['input_ids']],
|
||||
logits[tokenizer("C")['input_ids']],
|
||||
logits[tokenizer("D")['input_ids']],
|
||||
]
|
||||
),
|
||||
dim=0,
|
||||
)
|
||||
if softval.dtype in {torch.bfloat16, torch.float16}:
|
||||
softval = softval.to(dtype=torch.float32)
|
||||
probs = softval.detach().cpu().numpy()
|
||||
|
||||
for i, choice in enumerate(choices):
|
||||
all_probs[f'prob_{choice}'].append(probs[i])
|
||||
pred = {0: "A", 1: "B", 2: "C", 3: "D"}[np.argmax(probs)]
|
||||
|
||||
if 'answer' in row:
|
||||
correct = 1 if pred == row['answer'] else 0
|
||||
score.append(correct)
|
||||
if args.debug: print(f'{question} pred: {pred} ref: {row["answer"]}')
|
||||
result.append(pred)
|
||||
|
||||
if score:
|
||||
correct_ratio = 100 * sum(score) / len(score)
|
||||
if args.debug: print(subject_name, correct_ratio)
|
||||
else:
|
||||
correct_ratio = 0
|
||||
if save_result_dir:
|
||||
test_df['model_output'] = result
|
||||
for i, choice in enumerate(choices):
|
||||
test_df[f'prob_{choice}'] = (all_probs[f'prob_{choice}'])
|
||||
if score:
|
||||
test_df["correctness"] = score
|
||||
os.makedirs(save_result_dir, exist_ok=True)
|
||||
test_df.to_csv(os.path.join(
|
||||
save_result_dir, f'{subject_name}_result.csv'), encoding="utf-8", index=False)
|
||||
|
||||
return correct_ratio
|
||||
|
||||
|
||||
def cal_ceval(res):
|
||||
acc_sum_dict = dict()
|
||||
acc_norm_sum_dict = dict()
|
||||
cnt_dict = dict()
|
||||
acc_sum = 0.
|
||||
cnt = 0
|
||||
hard_cnt = 0
|
||||
hard_acc_sum = 0.
|
||||
for tt in res.keys():
|
||||
name = tt.split('-')[-1]
|
||||
acc_sum += float(res[tt])
|
||||
cnt += 1
|
||||
class_ = TASK_NAME_MAPPING[name][2]
|
||||
if class_ not in acc_sum_dict:
|
||||
acc_sum_dict[class_] = 0.
|
||||
acc_norm_sum_dict[class_] = 0.
|
||||
cnt_dict[class_] = 0.
|
||||
if name in hard_list:
|
||||
hard_cnt += 1
|
||||
hard_acc_sum += float(res[tt])
|
||||
acc_sum_dict[class_] += float(res[tt])
|
||||
cnt_dict[class_] += 1
|
||||
print('\n\n\n')
|
||||
for k in ['STEM', 'Social Science', 'Humanities', 'Other']:
|
||||
if k in cnt_dict:
|
||||
print('%s acc: %.2f ' % (
|
||||
k, acc_sum_dict[k] / cnt_dict[k]))
|
||||
if hard_cnt > 0:
|
||||
print('Hard acc:%.2f ' % (hard_acc_sum / hard_cnt))
|
||||
print('AVERAGE acc:%.2f ' % (acc_sum / cnt))
|
||||
|
||||
|
||||
TASK_NAME_MAPPING = {
|
||||
"computer_network": ["Computer Network", "\u8ba1\u7b97\u673a\u7f51\u7edc", "STEM"],
|
||||
"operating_system": ["Operating System", "\u64cd\u4f5c\u7cfb\u7edf", "STEM"],
|
||||
"computer_architecture": ["Computer Architecture", "\u8ba1\u7b97\u673a\u7ec4\u6210", "STEM"],
|
||||
"college_programming": ["College Programming", "\u5927\u5b66\u7f16\u7a0b", "STEM"],
|
||||
"college_physics": ["College Physics", "\u5927\u5b66\u7269\u7406", "STEM"],
|
||||
"college_chemistry": ["College Chemistry", "\u5927\u5b66\u5316\u5b66", "STEM"],
|
||||
"advanced_mathematics": ["Advanced Mathematics", "\u9ad8\u7b49\u6570\u5b66", "STEM"],
|
||||
"probability_and_statistics": ["Probability and Statistics", "\u6982\u7387\u7edf\u8ba1", "STEM"],
|
||||
"discrete_mathematics": ["Discrete Mathematics", "\u79bb\u6563\u6570\u5b66", "STEM"],
|
||||
"electrical_engineer": ["Electrical Engineer", "\u6ce8\u518c\u7535\u6c14\u5de5\u7a0b\u5e08", "STEM"],
|
||||
"metrology_engineer": ["Metrology Engineer", "\u6ce8\u518c\u8ba1\u91cf\u5e08", "STEM"],
|
||||
"high_school_mathematics": ["High School Mathematics", "\u9ad8\u4e2d\u6570\u5b66", "STEM"],
|
||||
"high_school_physics": ["High School Physics", "\u9ad8\u4e2d\u7269\u7406", "STEM"],
|
||||
"high_school_chemistry": ["High School Chemistry", "\u9ad8\u4e2d\u5316\u5b66", "STEM"],
|
||||
"high_school_biology": ["High School Biology", "\u9ad8\u4e2d\u751f\u7269", "STEM"],
|
||||
"middle_school_mathematics": ["Middle School Mathematics", "\u521d\u4e2d\u6570\u5b66", "STEM"],
|
||||
"middle_school_biology": ["Middle School Biology", "\u521d\u4e2d\u751f\u7269", "STEM"],
|
||||
"middle_school_physics": ["Middle School Physics", "\u521d\u4e2d\u7269\u7406", "STEM"],
|
||||
"middle_school_chemistry": ["Middle School Chemistry", "\u521d\u4e2d\u5316\u5b66", "STEM"],
|
||||
"veterinary_medicine": ["Veterinary Medicine", "\u517d\u533b\u5b66", "STEM"],
|
||||
"college_economics": ["College Economics", "\u5927\u5b66\u7ecf\u6d4e\u5b66", "Social Science"],
|
||||
"business_administration": ["Business Administration", "\u5de5\u5546\u7ba1\u7406", "Social Science"],
|
||||
"marxism": ["Marxism", "\u9a6c\u514b\u601d\u4e3b\u4e49\u57fa\u672c\u539f\u7406", "Social Science"],
|
||||
"mao_zedong_thought": ["Mao Zedong Thought", "\u6bdb\u6cfd\u4e1c\u601d\u60f3\u548c\u4e2d\u56fd\u7279\u8272\u793e\u4f1a\u4e3b\u4e49\u7406\u8bba\u4f53\u7cfb\u6982\u8bba", "Social Science"],
|
||||
"education_science": ["Education Science", "\u6559\u80b2\u5b66", "Social Science"],
|
||||
"teacher_qualification": ["Teacher Qualification", "\u6559\u5e08\u8d44\u683c", "Social Science"],
|
||||
"high_school_politics": ["High School Politics", "\u9ad8\u4e2d\u653f\u6cbb", "Social Science"],
|
||||
"high_school_geography": ["High School Geography", "\u9ad8\u4e2d\u5730\u7406", "Social Science"],
|
||||
"middle_school_politics": ["Middle School Politics", "\u521d\u4e2d\u653f\u6cbb", "Social Science"],
|
||||
"middle_school_geography": ["Middle School Geography", "\u521d\u4e2d\u5730\u7406", "Social Science"],
|
||||
"modern_chinese_history": ["Modern Chinese History", "\u8fd1\u4ee3\u53f2\u7eb2\u8981", "Humanities"],
|
||||
"ideological_and_moral_cultivation": ["Ideological and Moral Cultivation", "\u601d\u60f3\u9053\u5fb7\u4fee\u517b\u4e0e\u6cd5\u5f8b\u57fa\u7840", "Humanities"],
|
||||
"logic": ["Logic", "\u903b\u8f91\u5b66", "Humanities"],
|
||||
"law": ["Law", "\u6cd5\u5b66", "Humanities"],
|
||||
"chinese_language_and_literature": ["Chinese Language and Literature", "\u4e2d\u56fd\u8bed\u8a00\u6587\u5b66", "Humanities"],
|
||||
"art_studies": ["Art Studies", "\u827a\u672f\u5b66", "Humanities"],
|
||||
"professional_tour_guide": ["Professional Tour Guide", "\u5bfc\u6e38\u8d44\u683c", "Humanities"],
|
||||
"legal_professional": ["Legal Professional", "\u6cd5\u5f8b\u804c\u4e1a\u8d44\u683c", "Humanities"],
|
||||
"high_school_chinese": ["High School Chinese", "\u9ad8\u4e2d\u8bed\u6587", "Humanities"],
|
||||
"high_school_history": ["High School History", "\u9ad8\u4e2d\u5386\u53f2", "Humanities"],
|
||||
"middle_school_history": ["Middle School History", "\u521d\u4e2d\u5386\u53f2", "Humanities"],
|
||||
"civil_servant": ["Civil Servant", "\u516c\u52a1\u5458", "Other"],
|
||||
"sports_science": ["Sports Science", "\u4f53\u80b2\u5b66", "Other"],
|
||||
"plant_protection": ["Plant Protection", "\u690d\u7269\u4fdd\u62a4", "Other"],
|
||||
"basic_medicine": ["Basic Medicine", "\u57fa\u7840\u533b\u5b66", "Other"],
|
||||
"clinical_medicine": ["Clinical Medicine", "\u4e34\u5e8a\u533b\u5b66", "Other"],
|
||||
"urban_and_rural_planner": ["Urban and Rural Planner", "\u6ce8\u518c\u57ce\u4e61\u89c4\u5212\u5e08", "Other"],
|
||||
"accountant": ["Accountant", "\u6ce8\u518c\u4f1a\u8ba1\u5e08", "Other"],
|
||||
"fire_engineer": ["Fire Engineer", "\u6ce8\u518c\u6d88\u9632\u5de5\u7a0b\u5e08", "Other"],
|
||||
"environmental_impact_assessment_engineer": ["Environmental Impact Assessment Engineer", "\u73af\u5883\u5f71\u54cd\u8bc4\u4ef7\u5de5\u7a0b\u5e08", "Other"],
|
||||
"tax_accountant": ["Tax Accountant", "\u7a0e\u52a1\u5e08", "Other"],
|
||||
"physician": ["Physician", "\u533b\u5e08\u8d44\u683c", "Other"]
|
||||
}
|
||||
hard_list = ['advanced_mathematics', 'discrete_mathematics', 'probability_and_statistics', 'college_physics', 'college_chemistry', 'high_school_mathematics', 'high_school_physics', 'high_school_chemistry']
|
||||
choices = ["A", "B", "C", "D"]
|
||||
|
||||
|
||||
def main(args):
|
||||
model, tokenizer = load_models_tokenizer(args)
|
||||
|
||||
dev_result = {}
|
||||
for subject_name in tqdm(TASK_NAME_MAPPING.keys()):
|
||||
val_file_path = os.path.join(args.eval_data_path, 'val', f'{subject_name}_val.csv')
|
||||
dev_file_path = os.path.join(args.eval_data_path, 'dev', f'{subject_name}_dev.csv')
|
||||
# test_file_path = os.path.join(args.eval_data_path, 'test', f'{subject_name}_test.csv')
|
||||
val_df = pd.read_csv(val_file_path)
|
||||
dev_df = pd.read_csv(dev_file_path)
|
||||
# test_df = pd.read_csv(test_file_path)
|
||||
|
||||
score = eval_subject(model, tokenizer, subject_name, val_df, dev_df=dev_df, k=5, few_shot=True,
|
||||
save_result_dir=f"outs/ceval_eval_result")
|
||||
dev_result[subject_name] = score
|
||||
cal_ceval(dev_result)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
parser = argparse.ArgumentParser(description='Test HF checkpoint.')
|
||||
parser.add_argument('-c', '--checkpoint-path', type=str, help='Checkpoint path', default="Qwen/Qwen-7B")
|
||||
parser.add_argument('-s', '--seed', type=int, default=1234, help='Random seed')
|
||||
|
||||
"""Provide extra arguments required for tasks."""
|
||||
group = parser.add_argument_group(title='Evaluation options')
|
||||
group.add_argument('-d', '--eval_data_path', type=str, required=True,
|
||||
help='Path to eval data')
|
||||
group.add_argument("--max-seq-len", type=int, default=2048,
|
||||
help='Size of the output generated text.')
|
||||
group.add_argument("--debug", action='store_true', default=False,
|
||||
help='Print infos.')
|
||||
|
||||
args = parser.parse_args()
|
||||
set_seed(args.seed)
|
||||
|
||||
main(args)
|
||||
@ -0,0 +1,110 @@
|
||||
import random
|
||||
import tqdm
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import torch
|
||||
import numpy as np
|
||||
import jsonlines
|
||||
import argparse
|
||||
import jsonlines
|
||||
import datasets
|
||||
from datasets import load_from_disk,load_dataset
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
from transformers.generation import GenerationConfig
|
||||
|
||||
|
||||
ANS_RE = re.compile(r"#### (\-?[0-9\.\,]+)")
|
||||
INVALID_ANS = "[invalid]"
|
||||
|
||||
def doc_to_text(doc):
|
||||
return fewshot_prompt + "\nQuestion: " + doc["question"] + "\nLet's think step by step\n"
|
||||
|
||||
def decode(tokens_list, tokenizer, raw_text_len):
|
||||
sents = []
|
||||
# print(len(tokens_list))
|
||||
for tokens in tokens_list:
|
||||
tokens = tokens.cpu().numpy().tolist()
|
||||
sent = tokenizer.tokenizer.decode(
|
||||
tokens[raw_text_len:])
|
||||
sent = sent.split('<|endoftext|>')[0]
|
||||
sent = sent.split('\n\n\n')[0]
|
||||
sent = sent.split("\n\n")[0]
|
||||
sent = sent.split("Question:")[0]
|
||||
sents.append(sent)
|
||||
return sents
|
||||
|
||||
def generate_sample(model, tokenizer, input_txt):
|
||||
input_ids = tokenizer.tokenizer.encode(input_txt)
|
||||
raw_text_len = len(input_ids)
|
||||
context_enc = torch.tensor(
|
||||
[input_ids]).to(model.device)
|
||||
print(f"Input text: {input_txt}\n")
|
||||
outputs = model.generate(context_enc)
|
||||
output_text = decode(outputs,tokenizer,raw_text_len)[0]
|
||||
print(f"\nOutput text: {output_text}\n")
|
||||
return output_text
|
||||
|
||||
|
||||
def extract_answer_hf(completion):
|
||||
match = ANS_RE.search(completion)
|
||||
if match:
|
||||
match_str = match.group(1).strip()
|
||||
match_str = match_str.replace(",", "")
|
||||
return eval(match_str)
|
||||
else:
|
||||
return INVALID_ANS
|
||||
|
||||
def extract_answer(completion):
|
||||
try:
|
||||
last_number = re.findall(r'\d+', completion)[-1]
|
||||
return eval(last_number)
|
||||
except:
|
||||
return INVALID_ANS
|
||||
|
||||
def is_correct( completion, answer):
|
||||
gold = extract_answer_hf(answer)
|
||||
assert gold != INVALID_ANS, "No ground truth answer found in the document."
|
||||
return extract_answer(completion) == gold
|
||||
|
||||
if __name__ == '__main__':
|
||||
|
||||
parser = argparse.ArgumentParser(description='Test HF checkpoint.')
|
||||
parser.add_argument("-c", "--checkpoint-path", type=str, help="Checkpoint path", default="Qwen/Qwen-7B")
|
||||
parser.add_argument("-f","--sample-input-file", type=str, default=None)
|
||||
parser.add_argument("-o","--sample-output-file", type=str, default="gsm8k_res.jsonl")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
fewshot_prompt = open("gsm8k_prompt.txt").read()
|
||||
if args.sample_input_file is not None:
|
||||
dataset = load_from_disk(args.sample_input_file)
|
||||
else:
|
||||
config = datasets.DownloadConfig(resume_download=True, max_retries=100)
|
||||
dataset = load_dataset("gsm8k", 'main', download_config=config)
|
||||
|
||||
test = dataset["test"]
|
||||
|
||||
print('Loading tokenizer ...')
|
||||
tokenizer = AutoTokenizer.from_pretrained(args.checkpoint_path, trust_remote_code=True)
|
||||
|
||||
print('Loading model ...')
|
||||
model = AutoModelForCausalLM.from_pretrained(args.checkpoint_path, device_map="auto", trust_remote_code=True).eval()
|
||||
model.generation_config = GenerationConfig.from_pretrained(args.checkpoint_path, trust_remote_code=True)
|
||||
model.generation_config.do_sample = False
|
||||
|
||||
f_output = jsonlines.Writer(open(args.sample_output_file, 'w', encoding='utf-8'))
|
||||
tot_length = test.num_rows
|
||||
acc_res = []
|
||||
for doc in test:
|
||||
context = doc_to_text(doc)
|
||||
completion = generate_sample(model, tokenizer, context)
|
||||
answer= doc["answer"]
|
||||
acc = is_correct(completion, answer)
|
||||
doc["completion"]=completion
|
||||
doc["acc"]=acc
|
||||
f_output.write(doc)
|
||||
acc_res.append(acc)
|
||||
|
||||
f_output.close()
|
||||
print("Acc: ",np.mean(acc_res))
|
||||
@ -0,0 +1,70 @@
|
||||
import random
|
||||
import tqdm
|
||||
import os
|
||||
import sys
|
||||
import torch
|
||||
import jsonlines
|
||||
import argparse
|
||||
import jsonlines
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
from transformers.generation import GenerationConfig
|
||||
|
||||
"""
|
||||
git clone https://github.com/openai/human-eval
|
||||
$ pip install -e human-eval
|
||||
evaluate_functional_correctness sample-output-file
|
||||
"""
|
||||
|
||||
def decode(tokens_list, tokenizer, raw_text_len):
|
||||
sents = []
|
||||
# print(len(tokens_list))
|
||||
for tokens in tokens_list:
|
||||
tokens = tokens.cpu().numpy().tolist()
|
||||
sent = tokenizer.tokenizer.decode(
|
||||
tokens[raw_text_len:])
|
||||
sent = sent.split('<|endoftext|>')[0]
|
||||
sent = sent.split('\n\n\n')[0]
|
||||
sent = sent.split("\n\n")[0]
|
||||
sent = sent.split("def ")[0]
|
||||
sents.append(sent)
|
||||
return sents
|
||||
|
||||
def generate_sample(model, tokenizer, input_txt):
|
||||
input_ids = tokenizer.tokenizer.encode(input_txt)
|
||||
raw_text_len = len(input_ids)
|
||||
context_enc = torch.tensor([input_ids] ).to(model.device)
|
||||
print(f"Input text: {input_txt}\n")
|
||||
outputs = model.generate(context_enc)
|
||||
output_text = decode(outputs,tokenizer,raw_text_len)[0]
|
||||
print(f"\nOutput text: \n{output_text}\n")
|
||||
return output_text
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
|
||||
parser = argparse.ArgumentParser(description='Test HF checkpoint.')
|
||||
parser.add_argument("-c", "--checkpoint-path", type=str, help='Checkpoint path', default="Qwen/Qwen-7B")
|
||||
parser.add_argument("-f","--sample-input-file", type=str, default=None, help="data path to HumanEval.jsonl")
|
||||
parser.add_argument("-o","--sample-output-file", type=str, default="HumanEval_res.jsonl")
|
||||
|
||||
|
||||
args = parser.parse_args()
|
||||
print('Loading tokenizer ...')
|
||||
tokenizer = AutoTokenizer.from_pretrained(args.checkpoint_path, trust_remote_code=True)
|
||||
|
||||
print('Loading model ...')
|
||||
model = AutoModelForCausalLM.from_pretrained(args.checkpoint_path, device_map="auto", trust_remote_code=True).eval()
|
||||
model.generation_config = GenerationConfig.from_pretrained(args.checkpoint_path, trust_remote_code=True)
|
||||
model.generation_config.do_sample = False
|
||||
|
||||
f_output = jsonlines.Writer(open(args.sample_output_file, 'w', encoding='utf-8'))
|
||||
|
||||
f = jsonlines.open(args.sample_input_file)
|
||||
with f_output as output:
|
||||
for jobj in tqdm.tqdm(f, desc='task_idx'):
|
||||
prompt = jobj['prompt']
|
||||
task_id = jobj['task_id']
|
||||
gen_sents = generate_sample(model, tokenizer, prompt)
|
||||
gen_jobjs = {'task_id': task_id, "completion": gen_sents}
|
||||
output.write(gen_jobjs)
|
||||
f_output.close()
|
||||
@ -0,0 +1,218 @@
|
||||
import os
|
||||
import pandas as pd
|
||||
import numpy as np
|
||||
import argparse
|
||||
import datasets
|
||||
import torch
|
||||
|
||||
from typing import List
|
||||
from tqdm import tqdm
|
||||
from transformers.trainer_utils import set_seed
|
||||
|
||||
|
||||
'''
|
||||
wget https://people.eecs.berkeley.edu/~hendrycks/data.tar
|
||||
mkdir data/mmlu
|
||||
mv data.tar data/mmlu
|
||||
cd data/mmlu; tar xf data.tar
|
||||
cd ../../
|
||||
python eval/evaluate_mmlu.py -d data/mmlu/data/
|
||||
'''
|
||||
|
||||
|
||||
def load_models_tokenizer(args):
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
from transformers.generation import GenerationConfig
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(args.checkpoint_path, trust_remote_code=True)
|
||||
model = AutoModelForCausalLM.from_pretrained(args.checkpoint_path, device_map="auto", trust_remote_code=True).eval()
|
||||
model.generation_config = GenerationConfig.from_pretrained(args.checkpoint_path, trust_remote_code=True)
|
||||
return model, tokenizer
|
||||
|
||||
|
||||
def format_example(line, include_answer=True):
|
||||
example = 'Question: ' + line['question']
|
||||
for choice in choices:
|
||||
example += f'\n{choice}. {line[f"{choice}"]}'
|
||||
|
||||
if include_answer:
|
||||
example += '\nAnswer: ' + line["answer"] + '\n\n'
|
||||
else:
|
||||
example += '\nAnswer:'
|
||||
return example
|
||||
|
||||
|
||||
def generate_few_shot_prompt(k, subject, dev_df):
|
||||
|
||||
def format_subject(subject):
|
||||
l = subject.split("_")
|
||||
s = ""
|
||||
for entry in l:
|
||||
s += " " + entry
|
||||
return s.strip()
|
||||
|
||||
prompt = "The following are multiple choice questions (with answers) about {}.\n\n".format(format_subject(subject))
|
||||
|
||||
if k == -1:
|
||||
k = dev_df.shape[0]
|
||||
for i in range(k):
|
||||
prompt += format_example(
|
||||
dev_df.iloc[i, :],
|
||||
include_answer=True,
|
||||
)
|
||||
return prompt
|
||||
|
||||
|
||||
def get_logits(tokenizer, model, inputs: List[str]):
|
||||
input_ids = tokenizer(inputs, padding=False)['input_ids']
|
||||
input_ids = torch.tensor(input_ids, device=model.device)
|
||||
|
||||
if input_ids.shape[1] > args.max_seq_len:
|
||||
input_ids = input_ids[:, input_ids.shape[1]-args.max_seq_len+1:]
|
||||
tokens = {'input_ids': input_ids}
|
||||
|
||||
outputs = model(input_ids)['logits']
|
||||
logits = outputs[:, -1, :]
|
||||
log_probs = torch.nn.functional.softmax(logits, dim=-1)
|
||||
return log_probs, {'tokens': tokens}
|
||||
|
||||
|
||||
@torch.no_grad()
|
||||
def eval_subject(
|
||||
model,
|
||||
tokenizer,
|
||||
subject_name,
|
||||
test_df,
|
||||
k=5,
|
||||
dev_df=None,
|
||||
few_shot=False,
|
||||
save_result_dir=None,
|
||||
**kwargs
|
||||
):
|
||||
result = []
|
||||
score = []
|
||||
|
||||
few_shot_prompt = generate_few_shot_prompt(
|
||||
k, subject_name, dev_df) if few_shot else []
|
||||
all_probs = {'prob_A': [], 'prob_B': [], 'prob_C': [], 'prob_D': []}
|
||||
if args.debug: print(f"few_shot_prompt: {few_shot_prompt}")
|
||||
|
||||
for _, row in tqdm(test_df.iterrows(), total=len(test_df)):
|
||||
question = format_example(row, include_answer=False)
|
||||
full_prompt = few_shot_prompt + question
|
||||
|
||||
output, input_info = get_logits(tokenizer, model, [full_prompt])
|
||||
assert output.shape[0] == 1
|
||||
logits = output.flatten()
|
||||
|
||||
softval = torch.nn.functional.softmax(
|
||||
torch.tensor(
|
||||
[
|
||||
logits[tokenizer(" A")['input_ids']],
|
||||
logits[tokenizer(" B")['input_ids']],
|
||||
logits[tokenizer(" C")['input_ids']],
|
||||
logits[tokenizer(" D")['input_ids']],
|
||||
]
|
||||
),
|
||||
dim=0,
|
||||
)
|
||||
if softval.dtype in {torch.bfloat16, torch.float16}:
|
||||
softval = softval.to(dtype=torch.float32)
|
||||
probs = softval.detach().cpu().numpy()
|
||||
|
||||
for i, choice in enumerate(choices):
|
||||
all_probs[f'prob_{choice}'].append(probs[i])
|
||||
pred = {0: "A", 1: "B", 2: "C", 3: "D"}[np.argmax(probs)]
|
||||
|
||||
if 'answer' in row:
|
||||
correct = 1 if pred == row['answer'] else 0
|
||||
score.append(correct)
|
||||
if args.debug: print(f'{question} pred: {pred} ref: {row["answer"]}')
|
||||
result.append(pred)
|
||||
|
||||
if save_result_dir:
|
||||
test_df['model_output'] = result
|
||||
for i, choice in enumerate(choices):
|
||||
test_df[f'prob_{choice}'] = (all_probs[f'prob_{choice}'])
|
||||
if score:
|
||||
test_df["correctness"] = score
|
||||
os.makedirs(save_result_dir, exist_ok=True)
|
||||
test_df.to_csv(os.path.join(
|
||||
save_result_dir, f'{subject_name}_result.csv'), encoding="utf-8", index=False)
|
||||
|
||||
return score
|
||||
|
||||
|
||||
def cal_mmlu(res):
|
||||
acc_sum_dict = dict()
|
||||
acc_norm_sum_dict = dict()
|
||||
cnt_dict = dict()
|
||||
acc_sum = 0.
|
||||
cnt = 0
|
||||
hard_cnt = 0
|
||||
hard_acc_sum = 0.
|
||||
|
||||
for class_ in TASK_NAME_MAPPING.keys():
|
||||
acc_sum_dict[class_] = 0.
|
||||
acc_norm_sum_dict[class_] = 0.
|
||||
cnt_dict[class_] = 0.
|
||||
|
||||
for tt in TASK_NAME_MAPPING[class_]:
|
||||
acc_sum += sum(res[tt])
|
||||
cnt += len(res[tt])
|
||||
|
||||
acc_sum_dict[class_] += sum(res[tt])
|
||||
cnt_dict[class_] += len(res[tt])
|
||||
|
||||
print('\n\n\n', 'total cnt:', cnt, '\n')
|
||||
for k in TASK_NAME_MAPPING.keys():
|
||||
if k in cnt_dict:
|
||||
print('%s ACC: %.2f ' % (
|
||||
k, acc_sum_dict[k] / cnt_dict[k] * 100))
|
||||
print('AVERAGE ACC:%.2f ' % (acc_sum / cnt * 100))
|
||||
|
||||
|
||||
def main(args):
|
||||
model, tokenizer = load_models_tokenizer(args)
|
||||
|
||||
dev_result = {}
|
||||
for subject_name in tqdm(SUBJECTS):
|
||||
# val_file_path = os.path.join(args.eval_data_path, 'val', f'{subject_name}_val.csv')
|
||||
dev_file_path = os.path.join(args.eval_data_path, 'dev', f'{subject_name}_dev.csv')
|
||||
test_file_path = os.path.join(args.eval_data_path, 'test', f'{subject_name}_test.csv')
|
||||
# val_df = pd.read_csv(val_file_path, names=['question','A','B','C','D','answer'])
|
||||
dev_df = pd.read_csv(dev_file_path, names=['question','A','B','C','D','answer'])
|
||||
test_df = pd.read_csv(test_file_path, names=['question','A','B','C','D','answer'])
|
||||
|
||||
score = eval_subject(model, tokenizer, subject_name, test_df, dev_df=dev_df, k=5, few_shot=True,
|
||||
save_result_dir=f"outs/mmlu_eval_result")
|
||||
dev_result[subject_name] = score
|
||||
cal_mmlu(dev_result)
|
||||
|
||||
|
||||
TASK_NAME_MAPPING = {'stem': ['abstract_algebra', 'anatomy', 'astronomy', 'college_biology', 'college_chemistry', 'college_computer_science', 'college_mathematics', 'college_physics', 'computer_security', 'conceptual_physics', 'electrical_engineering', 'elementary_mathematics', 'high_school_biology', 'high_school_chemistry', 'high_school_computer_science', 'high_school_mathematics', 'high_school_physics', 'high_school_statistics', 'machine_learning'],
|
||||
'Humanities': ['formal_logic', 'high_school_european_history', 'high_school_us_history', 'high_school_world_history', 'international_law', 'jurisprudence', 'logical_fallacies', 'moral_disputes', 'moral_scenarios', 'philosophy', 'prehistory', 'professional_law', 'world_religions'],
|
||||
'other': ['business_ethics', 'college_medicine', 'human_aging', 'management', 'marketing', 'medical_genetics', 'miscellaneous', 'nutrition', 'professional_accounting', 'professional_medicine', 'virology', 'global_facts', 'clinical_knowledge'],
|
||||
'social': ['econometrics', 'high_school_geography', 'high_school_government_and_politics', 'high_school_macroeconomics', 'high_school_microeconomics', 'high_school_psychology', 'human_sexuality', 'professional_psychology', 'public_relations', 'security_studies', 'sociology', 'us_foreign_policy']}
|
||||
SUBJECTS = [v for vl in TASK_NAME_MAPPING.values() for v in vl]
|
||||
choices = ["A", "B", "C", "D"]
|
||||
|
||||
if __name__ == '__main__':
|
||||
parser = argparse.ArgumentParser(description='Test HF checkpoint.')
|
||||
parser.add_argument('-c', '--checkpoint-path', type=str, help='Checkpoint path', default="Qwen/Qwen-7B")
|
||||
parser.add_argument('-s', '--seed', type=int, default=1234, help='Random seed')
|
||||
parser.add_argument('--gpu', type=int, default=0, help='gpu id')
|
||||
|
||||
"""Provide extra arguments required for tasks."""
|
||||
group = parser.add_argument_group(title='Evaluation options')
|
||||
group.add_argument('-d', '--eval_data_path', type=str,
|
||||
help='Path to eval data')
|
||||
group.add_argument("--max-seq-len", type=int, default=2048,
|
||||
help='Size of the output generated text.')
|
||||
group.add_argument("--debug", action='store_true', default=False,
|
||||
help='Print infos.')
|
||||
|
||||
args = parser.parse_args()
|
||||
set_seed(args.seed)
|
||||
|
||||
main(args)
|
||||
@ -0,0 +1,340 @@
|
||||
# Introducing Qwen-7B: Open foundation and human-aligned models (of the state-of-the-arts)
|
||||
|
||||
Large language models have recently attracted an extremely large amount of
|
||||
attention.
|
||||
The boom of [ChatGPT](https://openai.com/blog/chatgpt) rocketed the development of artificial general intelligence and indicates that large language models compress world knowledge into neural networks, and the alignment to human cognition can lead to powerful conversational agents that can provide assistance by interacting with human users.
|
||||
Now, the latest version of ChatGPT based on [GPT-4](https://arxiv.org/abs/2303.08774) demonstrates tremendously exciting performance across unlimited capabilities, say, language understanding, logical reasoning, planning, etc., and its incorporation with external tools, including tools and models, releases the power of an agent capable of understanding instructions, executing code, using tools, and so on, to reach the objectives set up by human users.
|
||||
|
||||
These significant progresses indicate the importance of large language models as _the foundation of AI services_.
|
||||
|
||||
We are happy to release the 7B-parameter models of our large pretrained model series Qwen (abbr. Tongyi Qianwen), Qwen-7B.
|
||||
This release includes model weights and codes for pretrained and human-aligned language models of 7B parameters:
|
||||
|
||||
- `Qwen-7B` is the pretrained language model, and `Qwen-7B-Chat` is fine-tuned to align with human intent.
|
||||
- `Qwen-7B` is pretrained on over 2.2 trillion tokens with a context length of 2048. On the series of benchmarks we tested, Qwen-7B generally performs better than existing open models of similar scales and appears to be on par with some of the larger models.
|
||||
- `Qwen-7B-Chat` is fine-tuned on curated data, including not only task-oriented data but also specific security- and service-oriented data, which seems insufficient in existing open models.
|
||||
- Example codes for fine-tuning, evaluation, and inference are included. There are also guides on long-context and tool use in inference.
|
||||
|
||||
**Goal of release**:
|
||||
We believe that while the recent waves of releases of LLMs may have deepened our understanding of model behaviors under standard regimes, it is yet to be revealed how the accompanied techniques of nowadays LLMs, such as 1) quantization and fine-tuning after quantization, 2) training-free long-context inference, and 3) fine-tuning with service-oriented data, including search and tool uses, affect the models as a whole.
|
||||
The open release of Qwen-7B marks our first step towards fully understanding the real-world application of such techniques.
|
||||
It is our hope that it will enable the community to analyze and continue to improve the safety of those models, striving to establish responsible development and deployment of LLMs.
|
||||
|
||||
> **Disclaimer**:
|
||||
> We must note that even though the weights and codes are released in an open manner and commercial use is not prohibited, similar to other pretrained language models, Qwen-7B comes with potential risks influenced by complex factors, including but not limited to over-diversified, inaccurate, or misleading generation.
|
||||
> Developers and stakeholders should perform their own red teaming and provide related security measures before deployment, and they must abide by and comply with local governance and regulations.
|
||||
> In no event shall the authors be held liable for any claim, damages, or other liability arising from the use of the released weights or codes.
|
||||
|
||||
The remainder of this document describes our pretraining and fine-tuning methodology.
|
||||
|
||||
## Pretraining
|
||||
|
||||
Qwen-7B is a transformer-based decoder-only language model with an architecture similar to the [LLaMA](https://github.com/facebookresearch/llama) series of models.
|
||||
It is pretrained on over 2.2 trillion tokens with 2048 context length from publicly available data, covering general and professional fields with a focus on the English and Chinese languages.
|
||||
|
||||
### Data
|
||||
|
||||
**Pretraining data**:
|
||||
Our training data includes a mix of data from publicly available sources, consisting mainly of web documents and code files.
|
||||
Besides, the data are multilingual, with most of them in English and Chinese.
|
||||
We made an effort and employed an ensemble of models to exclude data of low quality or deemed unfit for pretraining, such as NSFW content.
|
||||
The final data underwent global fuzzy deduplication.
|
||||
The mix of pretraining corpora has been optimized through numerous ablation experiments.
|
||||
|
||||
**Tokenization**:
|
||||
Compared to the current mainstream open models based on Chinese and English vocabularies, we use a vocabulary of 151,851 tokens.
|
||||
It first considers efficient encoding of Chinese, English, and code data, and is also more friendly to multilingual languages, enabling users to directly enhance the capability of some languages without expanding the vocabulary.
|
||||
It segments numbers by single digits and calls the [tiktoken](https://github.com/openai/tiktoken) tokenizer library for efficient tokenization.
|
||||
After tokenization, the data amounts to over 2.2 trillion tokens.
|
||||
|
||||
<figure>
|
||||
<img src="assets/tokenizer.png"
|
||||
alt="Tokenization efficiency"
|
||||
width="1200px">
|
||||
<figcaption>We randomly selected 1 million document corpora of each language to test and compare the encoding compression rates of different models (with XLM-R, which supports 100 languages, as the base value 1, not shown in the figure). As can be seen, while ensuring the efficient decoding of Chinese, English, and code, Qwen-7B also achieves a high compression rate for many other languages (such as th, he, ar, ko, vi, ja, tr, id, pl, ru, nl, pt, it, de, es, fr etc.), equipping the model with strong scalability as well as high training and inference efficiency in these languages.</figcaption>
|
||||
</figure>
|
||||
|
||||
### Model
|
||||
|
||||
**Model architecture**:
|
||||
Qwen-7B is built with architecture similar to LLaMA.
|
||||
The following are the main differences from the standard transformer: 1) using untied embedding, 2) using rotary positional embedding, 3) no biases except for QKV in attention, 4) RMSNorm instead of LayerNorm, 5) SwiGLU instead of ReLU, and 6) adopting flash attention to accelerate training.
|
||||
The model has 32 layers, the embedding dimension is 4096, and the number of attention heads is 32.
|
||||
|
||||
**Training details**:
|
||||
The model is trained using the AdamW optimizer, with $\beta_1=0.9, \beta_2=0.95, \epsilon=10^{-6}$.
|
||||
The sequence length is 2048, and the batch size is 2048, which means each optimization step accumulates over 4 million tokens.
|
||||
We use a cosine learning rate schedule, with a warm-up of 2000 steps, a peak learning rate of $3 \times 10^{-4}$, and a minimum learning rate of 10% of the peak learning rate.
|
||||
We use a weight decay of 0.1 and gradient clipping of 1.0.
|
||||
The training adopts mixed precision training with `bfloat16`.
|
||||
|
||||
|
||||
### Evaluation
|
||||
|
||||
We report results of Qwen-7B on standard benchmarks.
|
||||
|
||||
#### World knowledge
|
||||
|
||||
[C-Eval](https://arxiv.org/abs/2305.08322) is a common evaluation benchmark for testing the common-sense capability of pretrained models in Chinese. It covers 52 subjects in four major directions: humanities, social sciences, STEM, and other specialties. According to standard practice, we use the development set samples as the source of few-shot prompts to evaluate the 5-shot validation set and test set accuracy of the Qwen-7B pretrained model.
|
||||
|
||||
The accuracy comparison of the Qwen-7B model and other models on the C-Eval validation set is as follows:
|
||||
|
||||
| Model | Average |
|
||||
| :---------- | -------: |
|
||||
| Alpaca-7B | 28.9 |
|
||||
| Vicuna-7B | 31.2 |
|
||||
| ChatGLM-6B | 37.1 |
|
||||
| Baichuan-7B | 42.7 |
|
||||
| ChatGLM2-6B | 50.9 |
|
||||
| InternLM-7B | 53.4 |
|
||||
| ChatGPT | 53.5 |
|
||||
| Claude-v1.3 | 55.5 |
|
||||
| **Qwen-7B** | **60.8** |
|
||||
|
||||
The performance comparison of the Qwen-7B pretrained model and other models on the C-Eval test set is shown in the following table:
|
||||
|
||||
| Model | Avg. | Avg. (Hard) | STEM | Social Sciences | Humanities | Others |
|
||||
| :---------------------- | -------- | ----------: | ---: | --------------: | ---------: | -----: |
|
||||
| ChatGLM-6B | 38.9 | 29.2 | 33.3 | 48.3 | 41.3 | 38.0 |
|
||||
| Chinese-Alpaca-Plus-13B | 41.5 | 30.5 | 36.6 | 49.7 | 43.1 | 41.2 |
|
||||
| Baichuan-7B | 42.8 | 31.5 | 38.2 | 52.0 | 46.2 | 39.3 |
|
||||
| WestlakeLM-19B | 44.6 | 34.9 | 41.6 | 51.0 | 44.3 | 44.5 |
|
||||
| AndesLM-13B | 46.0 | 29.7 | 38.1 | 61.0 | 51.0 | 41.9 |
|
||||
| BatGPT-15B-sirius | 47.0 | 31.9 | 42.7 | 57.5 | 48.6 | 43.6 |
|
||||
| ChatGLM2-6B | 51.7 | 37.1 | 48.6 | 60.5 | 51.3 | 49.8 |
|
||||
| InternLM-7B | 52.8 | 37.1 | 48.0 | 67.4 | 55.4 | 45.8 |
|
||||
| Baichuan-13B | 53.6 | 36.7 | 47.0 | 66.8 | 57.3 | 49.8 |
|
||||
| Claude-v1.3 | 54.2 | 39.0 | 51.9 | 61.7 | 52.1 | 53.7 |
|
||||
| ChatGPT | 54.4 | 41.4 | 52.9 | 61.8 | 50.9 | 53.6 |
|
||||
| **Qwen-7B** | **59.6** | 41.0 | 52.8 | 74.1 | 63.1 | 55.2 |
|
||||
|
||||
As can be seen, Qwen-7B achieves the best performance out of all existing models of similar scale and even surpasses larger-scale models.
|
||||
|
||||
MMLU is currently one of the most recognized benchmarks for evaluating English comprehension abilities, covering 57 subtasks across different academic fields and difficulty levels. The MMLU 5-shot accuracy performance of the Qwen-7B is shown in the following table:
|
||||
|
||||
| Model | Average | STEM | Social Sciences | Humanities | Others |
|
||||
| :----------- | -------: | ---: | --------------: | ---------: | -----: |
|
||||
| LLaMA-7B | 35.1 | 30.5 | 38.3 | 34.0 | 38.1 |
|
||||
| Baichuan-7B | 42.3 | 35.6 | 48.9 | 38.4 | 48.1 |
|
||||
| LLaMA2-7B | 45.3 | 36.4 | 51.2 | 42.9 | 52.2 |
|
||||
| LLaMA-13B | 46.9 | 35.8 | 53.8 | 45.0 | 53.3 |
|
||||
| ChatGLM2-6B | 47.9 | 41.2 | 54.4 | 43.7 | 54.5 |
|
||||
| InternLM-7B | 51.0 | - | - | - | - |
|
||||
| Baichuan-13B | 51.6 | 41.6 | 60.9 | 47.4 | 58.5 |
|
||||
| LLaMA2-13B | 54.8 | 44.1 | 62.6 | 52.8 | 61.1 |
|
||||
| ChatGLM2-12B | 56.2 | 48.2 | 65.1 | 52.6 | 60.9 |
|
||||
| **Qwen-7B** | **56.7** | 47.6 | 65.9 | 51.5 | 64.7 |
|
||||
|
||||
In terms of English, Qwen-7B also surpasses other similar open pretrained models, and is competitive when compared to larger versions of other models.
|
||||
|
||||
#### Coding
|
||||
|
||||
We compared the code capabilities of pretrained models on [HumanEval](https://github.com/openai/human-eval), and the results are as follows:
|
||||
|
||||
| Model | Pass@1 |
|
||||
| :----------- | -------: |
|
||||
| Baichuan-7B | 9.2 |
|
||||
| ChatGLM2-6B | 9.2 |
|
||||
| InternLM-7B | 10.4 |
|
||||
| LLaMA-7B | 10.5 |
|
||||
| LLaMA2-7B | 12.8 |
|
||||
| Baichuan-13B | 12.8 |
|
||||
| LLaMA-13B | 15.8 |
|
||||
| MPT-7B | 18.3 |
|
||||
| LLaMA2-13B | 18.3 |
|
||||
| **Qwen-7B** | **24.4** |
|
||||
|
||||
#### Math
|
||||
|
||||
We compared the math capabilities of pretrained models on [GSM8K](https://github.com/openai/human-eval) (8-shot), and the results are as follows:
|
||||
|
||||
| Model | Accuracy |
|
||||
| :----------- | -------: |
|
||||
| MPT-7B | 6.8 |
|
||||
| Falcon-7B | 6.8 |
|
||||
| Baichuan-7B | 9.7 |
|
||||
| LLaMA-7B | 11.0 |
|
||||
| LLaMA2-7B | 14.6 |
|
||||
| LLaMA-13B | 17.8 |
|
||||
| Baichuan-13B | 26.6 |
|
||||
| LLaMA2-13B | 28.7 |
|
||||
| InternLM-7B | 31.2 |
|
||||
| ChatGLM2-6B | 32.4 |
|
||||
| ChatGLM2-12B | 40.9 |
|
||||
| **Qwen-7B** | **51.6** |
|
||||
|
||||
#### Natural language processing
|
||||
|
||||
We compared the translation capabilities of pre-trained models on WMT22 zh-en and en-zh (5-shot BLEU), and the results are as follows:
|
||||
|
||||
| Model | Average | zh-en | en-zh |
|
||||
| :---------- | -------: | -------: | -------: |
|
||||
| InternLM-7B | 11.8 | 9.0 | 14.5 |
|
||||
| LLaMA-7B | 12.7 | 16.7 | 8.7 |
|
||||
| LLaMA-13B | 15.8 | 19.5 | 12.0 |
|
||||
| LLaMA2-7B | 19.9 | 21.9 | 17.9 |
|
||||
| Bloom-7B | 20.3 | 19.1 | 21.4 |
|
||||
| LLaMA2-13B | 23.3 | 22.4 | 24.2 |
|
||||
| PolyLM-13B | 23.6 | 20.2 | 27.0 |
|
||||
| Baichuan-7B | 24.6 | 22.6 | 26.6 |
|
||||
| **Qwen-7B** | **27.5** | **24.3** | **30.6** |
|
||||
|
||||
#### Long-context inference
|
||||
|
||||
We include support for training-free long-context inference based on ntk-aware interpolation, LogN attention scaling, and local window attention.
|
||||
The context can be expanded from 2048 to over 8192.
|
||||
The following are the test results on arXiv in terms of perplexity (PPL).
|
||||
|
||||
<table>
|
||||
<tr>
|
||||
<th rowspan="2">Model</th><th colspan="5" align="center">Sequence Length</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<th align="center">1024</th><th align="center">2048</th><th align="center">4096</th><th align="center">8192</th><th align="center">16384</th>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Qwen-7B</td><td align="right"><b>4.23</b></td><td align="right"><b>3.78</b></td><td align="right">39.35</td><td align="right">469.81</td><td align="right">2645.09</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>+ dynamic_ntk</td><td align="right"><b>4.23</b></td><td align="right"><b>3.78</b></td><td align="right">3.59</td><td align="right">3.66</td><td align="right">5.71</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>+ dynamic_ntk + logn</td><td align="right"><b>4.23</b></td><td align="right"><b>3.78</b></td><td align="right"><b>3.58</b></td><td align="right">3.56</td><td align="right">4.62</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>+ dynamic_ntk + logn + local_attn</td><td align="right"><b>4.23</b></td><td align="right"><b>3.78</b></td><td align="right"><b>3.58</b></td><td align="right"><b>3.49</b></td><td align="right"><b>4.32</b></td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
## Fine-tuning
|
||||
|
||||
`Qwen-7B-Chat` embodies our practice in alignment with human intents, ensuring internalized safety, and building intelligent agents for services.
|
||||
|
||||
### Data
|
||||
|
||||
**Alignment data**:
|
||||
The data includes common instruction-style conversations, and security- and service-oriented data, which involves substantial annotation efforts.
|
||||
Instruction data covers broad abilities, such as writing, question answering, brainstorming and planning, content understanding, summarization, natural language processing, and coding.
|
||||
Security data tries to prevent the model from generating harmful and inappropriate content.
|
||||
Service data tries to enhance the model with specific conversation patterns that can be parsed to invoke and incorporate external systems.
|
||||
|
||||
**Data formatting**:
|
||||
Since the data consists of conversation turns, we arrange them into texts using the [ChatML](https://github.com/openai/openai-python/blob/main/chatml.md) format, which is a meta language that can describe both the metadata (e.g., roles) and the content of a turn.
|
||||
Currently, existing roles include system, user, and assistant.
|
||||
|
||||
### Model
|
||||
|
||||
**Training details**:
|
||||
The causal language modeling objective is used to fine-tune the model, except for the tokens in the content of user's turns.
|
||||
The model is trained using the AdamW optimizer, with $\beta_1=0.9, \beta_2=0.95, \epsilon=10^{-6}$.
|
||||
The sequence length is limited to 2048, and the batch size is 128.
|
||||
The model is trained for 4000 steps, and over the first 1430 steps, the learning rate is warmed up to $1 \times 10^{-5}$.
|
||||
We use weight decay of 0.1, dropout of 0.1, and gradient clipping of 1.0.
|
||||
|
||||
### Evaluation
|
||||
|
||||
Evaluation of human-aligned models is non-trivial and often non-standardized, since such models often target specific applications.
|
||||
We evaluate Qwen-7B-Chat from multiple perspectives.
|
||||
|
||||
#### World knowledge
|
||||
|
||||
As fine-tuning uses a much smaller dataset than pretraining and humans' understanding of world knowledge may be limited, we also evaluate the world knowledge of Qwen-7B-Chat using C-Eval and MMLU in a zero-shot and generative manner.
|
||||
|
||||
We demonstrate the zero-shot accuracy of Qwen-7B-Chat on the C-Eval validation set.
|
||||
|
||||
| Model | Avg. Acc. |
|
||||
| :---------------------- | --------: |
|
||||
| LLaMA2-7B-Chat | 31.9 |
|
||||
| LLaMA2-13B-Chat | 40.6 |
|
||||
| Chinese-Alpaca-2-7B | 41.3 |
|
||||
| Chinese-Alpaca-Plus-13B | 43.3 |
|
||||
| Baichuan-13B-Chat | 50.4 |
|
||||
| ChatGLM2-6B-Chat | 50.7 |
|
||||
| InternLM-7B-Chat | 53.2 |
|
||||
| **Qwen-7B-Chat** | **54.2** |
|
||||
|
||||
The zero-shot accuracy of Qwen-7B-Chat on C-Eval testing set is provided below
|
||||
|
||||
| Model | Avg. | STEM | Social Sciences | Humanities | Others |
|
||||
| :---------------------- | -------: | ---: | --------------: | ---------: | -----: |
|
||||
| Chinese-Alpaca-Plus-13B | 41.5 | 36.6 | 49.7 | 43.1 | 41.2 |
|
||||
| Chinese-Alpaca-2-7B | 40.3 | - | - | - | - |
|
||||
| ChatGLM2-6B-Chat | 50.1 | 46.4 | 60.4 | 50.6 | 46.9 |
|
||||
| Baichuan-13B-Chat | 51.5 | 43.7 | 64.6 | 56.2 | 49.2 |
|
||||
| **Qwen-7B-Chat** | **54.6** | 47.8 | 67.6 | 59.3 | 50.6 |
|
||||
|
||||
Compared with other models with comparable model sizes, the human-aligned Qwen-7B-Chat performs well in C-Eval accuracy.
|
||||
|
||||
The zero-shot accuracy of Qwen-7B-Chat on MMLU is provided below.
|
||||
The performance of Qwen-7B-Chat is still on top among other human-aligned models with comparable size.
|
||||
|
||||
| Model | Avg. Acc. |
|
||||
| :---------------- | --------: |
|
||||
| ChatGLM2-6B-Chat | 45.5 |
|
||||
| LLaMA2-7B-Chat | 47.0 |
|
||||
| InternLM-7B-Chat | 50.8 |
|
||||
| Baichuan-13B-Chat | 52.1 |
|
||||
| ChatGLM2-12B-Chat | 52.1 |
|
||||
| **Qwen-7B-Chat** | **53.9** |
|
||||
|
||||
#### Coding
|
||||
|
||||
The zero-shot Pass@1 of Qwen-7B-Chat on [HumanEval](https://github.com/openai/human-eval) is demonstrated below
|
||||
|
||||
| Model | Pass@1 |
|
||||
| :---------------- | -------: |
|
||||
| LLaMA2-7B-Chat | 12.2 |
|
||||
| InternLM-7B-Chat | 14.0 |
|
||||
| Baichuan-13B-Chat | 16.5 |
|
||||
| LLaMA2-13B-Chat | 18.9 |
|
||||
| **Qwen-7B-Chat** | **21.3** |
|
||||
|
||||
#### Math
|
||||
|
||||
The accuracy of Qwen-7B-Chat on GSM8K is shown below
|
||||
|
||||
| Model | Zero-shot Acc. | 4-shot Acc. |
|
||||
| :---------------- | -------------: | ----------: |
|
||||
| ChatGLM2-6B-Chat | - | 28.0 |
|
||||
| LLaMA2-7B-Chat | 20.4 | 28.2 |
|
||||
| LLaMA2-13B-Chat | 29.4 | 36.7 |
|
||||
| InternLM-7B-Chat | 32.6 | 34.5 |
|
||||
| Baichuan-13B-Chat | - | 36.3 |
|
||||
| ChatGLM2-12B-Chat | - | 38.1 |
|
||||
| **Qwen-7B-Chat** | **41.1** | **43.5** |
|
||||
|
||||
#### Service
|
||||
|
||||
LLMs have shown capability in coordinating multiple external systems to achieve the given instructions, which creates new opportunities in traditional online services, the most notable being web search.
|
||||
|
||||
Qwen supports calling plugins/tools/APIs through [ReAct Prompting](https://arxiv.org/abs/2210.03629).
|
||||
ReAct is also one of the main approaches used by the [LangChain](https://python.langchain.com/) framework.
|
||||
For how to write and use prompts for ReAct Prompting, please refer to [the ReAct examples](examples/react_prompt.md).
|
||||
In the soon-to-be-released evaluation benchmark for assessing tool usage capabilities, Qwen's performance is as follows:
|
||||
|
||||
| Model | Tool Selection (Acc.↑) | Tool Input (Rouge-L↑) | False Positive Error↓ |
|
||||
| :---------- | --------------------------: | -------------------------: | -------------------------: |
|
||||
| GPT-4 | 95% | **0.90** | 15.0% |
|
||||
| GPT-3.5 | 85% | 0.88 | 75.0% |
|
||||
| **Qwen-7B** | **99%** | 0.89 | **8.5%** |
|
||||
|
||||
> The plugins that appear in the evaluation set do not appear in the training set of Qwen.
|
||||
> This benchmark evaluates the accuracy of the model in selecting the correct plugin from multiple candidate plugins, the rationality of the parameters passed into the plugin, and the false positive rate.
|
||||
> False Positive: Incorrectly invoking a plugin when it should not have been called when responding to a query.
|
||||
|
||||
Qwen also has the capability to be used as a [HuggingFace Agent](https://huggingface.co/docs/transformers/transformers_agents).
|
||||
Its performance on the benchmark provided by HuggingFace is as follows:
|
||||
|
||||
| Model | Tool Selection↑ | Tool Used↑ | Code↑ |
|
||||
| :-------------- | -------------------: | --------------: | ---------: |
|
||||
| GPT-4 | **100.00** | **100.00** | **97.41** |
|
||||
| GPT-3.5 | 95.37 | 96.30 | 87.04 |
|
||||
| StarCoder-15.5B | 87.04 | 87.96 | 68.89 |
|
||||
| **Qwen-7B** | 90.74 | 92.59 | 74.07 |
|
||||
|
||||
## Conclusion
|
||||
|
||||
In this document, we describe Qwen-7B, including a pretrained model and a human-aligned model.
|
||||
These models have demonstrated exciting performance compared to existing open models of similar or even larger scales.
|
||||
As part of our ongoing commitment to the concept of Model as a Service, the release also includes practical pieces such as long context inference and external system integration, which we hope would facilitate developers realizing their own ideas and concepts.
|
||||
We believe that the open release of Qwen-7B models would further our understanding of variables and techniques introduced in realistic settings and help to drive progress in this important area together with the community.
|
||||
Loading…
Reference in New Issue