From 73f4040abb8e4cf3f92bc577d161611cea78b652 Mon Sep 17 00:00:00 2001 From: JustinLin610 Date: Tue, 12 Sep 2023 23:10:59 +0800 Subject: [PATCH 1/3] add gradient checkpointing --- finetune/finetune_lora_ds.sh | 1 + finetune/finetune_lora_single_gpu.sh | 1 + finetune/finetune_qlora_ds.sh | 1 + finetune/finetune_qlora_single_gpu.sh | 1 + 4 files changed, 4 insertions(+) diff --git a/finetune/finetune_lora_ds.sh b/finetune/finetune_lora_ds.sh index 12efc6e..d877030 100644 --- a/finetune/finetune_lora_ds.sh +++ b/finetune/finetune_lora_ds.sh @@ -44,4 +44,5 @@ torchrun $DISTRIBUTED_ARGS finetune.py \ --model_max_length 2048 \ --lazy_preprocess True \ --use_lora \ + --gradient_checkpointing \ --deepspeed finetune/ds_config_zero2.json \ No newline at end of file diff --git a/finetune/finetune_lora_single_gpu.sh b/finetune/finetune_lora_single_gpu.sh index 47b7237..9333b7d 100644 --- a/finetune/finetune_lora_single_gpu.sh +++ b/finetune/finetune_lora_single_gpu.sh @@ -32,4 +32,5 @@ python finetune.py \ --report_to "none" \ --model_max_length 2048 \ --lazy_preprocess True \ + --gradient_checkpointing \ --use_lora \ No newline at end of file diff --git a/finetune/finetune_qlora_ds.sh b/finetune/finetune_qlora_ds.sh index debea4f..58c05e9 100644 --- a/finetune/finetune_qlora_ds.sh +++ b/finetune/finetune_qlora_ds.sh @@ -46,4 +46,5 @@ torchrun $DISTRIBUTED_ARGS finetune.py \ --lazy_preprocess True \ --use_lora \ --q_lora \ + --gradient_checkpointing \ --deepspeed finetune/ds_config_zero2.json \ No newline at end of file diff --git a/finetune/finetune_qlora_single_gpu.sh b/finetune/finetune_qlora_single_gpu.sh index 3b89ee4..bf65b9f 100644 --- a/finetune/finetune_qlora_single_gpu.sh +++ b/finetune/finetune_qlora_single_gpu.sh @@ -32,5 +32,6 @@ python finetune.py \ --report_to "none" \ --model_max_length 2048 \ --lazy_preprocess True \ + --gradient_checkpointing \ --use_lora \ --q_lora \ No newline at end of file From fb52dd330837638afd65fb332c523d83c451966f Mon Sep 17 00:00:00 2001 From: Junyang Lin Date: Wed, 13 Sep 2023 16:53:34 +0800 Subject: [PATCH 2/3] Update README.md --- README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.md b/README.md index 757ab30..7f1ec7e 100644 --- a/README.md +++ b/README.md @@ -15,6 +15,10 @@



+__Will be back soon...__ + +--- + We opensource **Qwen-7B** and **Qwen-7B-Chat** on both **🤖 ModelScope** and **🤗 Hugging Face** (Click the logos on top to the repos with codes and checkpoints). This repo includes the brief introduction to Qwen-7B, the usage guidance, and also a technical memo [link](tech_memo.md) that provides more information. Qwen-7B is the 7B-parameter version of the large language model series, Qwen (abbr. Tongyi Qianwen), proposed by Alibaba Cloud. Qwen-7B is a Transformer-based large language model, which is pretrained on a large volume of data, including web texts, books, codes, etc. Additionally, based on the pretrained Qwen-7B, we release Qwen-7B-Chat, a large-model-based AI assistant, which is trained with alignment techniques. The features of the Qwen-7B series include: From 0e07532cd0a28ca3e3af1850b8297bc095e44749 Mon Sep 17 00:00:00 2001 From: Dayiheng Liu <2689514749@qq.com> Date: Fri, 15 Sep 2023 10:03:22 +0800 Subject: [PATCH 3/3] Update EVALUATION.md fix typo --- eval/EVALUATION.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/eval/EVALUATION.md b/eval/EVALUATION.md index 1381e69..75ea021 100644 --- a/eval/EVALUATION.md +++ b/eval/EVALUATION.md @@ -59,7 +59,7 @@ pip install -e human-eval python evaluate_humaneval.py -f HumanEval.jsonl -o HumanEval_res.jsonl evaluate_functional_correctness HumanEval_res.jsonl # Qwen-7B-Chat -python evaluate_chat_mmlu.py -f HumanEval.jsonl -o HumanEval_res_chat.jsonl +python evaluate_chat_humaneval.py -f HumanEval.jsonl -o HumanEval_res_chat.jsonl evaluate_functional_correctness HumanEval_res_chat.jsonl ```