|
|
# Inference Qwen Using 🤖 ModelScope and 🤗 Transformers
|
|
|
|
|
|
Below, we provide simple examples to show how to inference Qwen with 🤖 ModelScope and 🤗 Transformers.
|
|
|
|
|
|
## Requirements
|
|
|
|
|
|
* python 3.8 and above
|
|
|
* pytorch 1.12 and above, 2.0 and above are recommended
|
|
|
* transformers 4.32 and above
|
|
|
* CUDA 11.4 and above are recommended (this is for GPU users, flash-attention users, etc.)
|
|
|
<br>
|
|
|
|
|
|
## Installation
|
|
|
|
|
|
You can use our pre-built docker images to skip most of the environment setup steps, see Section ["Using Pre-built Docker Images"](https://github.com/QwenLM/Qwen?tab=readme-ov-file#-docker) for more details.
|
|
|
|
|
|
If not using docker, please make sure you have setup the environment and installed the required packages. Make sure you meet the above requirements, and then install the dependent libraries.
|
|
|
|
|
|
```bash
|
|
|
pip install -r Qwen/requirements.txt
|
|
|
```
|
|
|
|
|
|
If your device supports fp16 or bf16, we recommend installing [flash-attention](https://github.com/Dao-AILab/flash-attention) (**we support flash attention 2 now.**) for higher efficiency and lower memory usage. (**flash-attention is optional and the project can run normally without installing it**)
|
|
|
|
|
|
```bash
|
|
|
git clone https://github.com/Dao-AILab/flash-attention
|
|
|
cd flash-attention && pip install .
|
|
|
# Below are optional. Installing them might be slow.
|
|
|
# pip install csrc/layer_norm
|
|
|
# If the version of flash-attn is higher than 2.1.1, the following is not needed.
|
|
|
# pip install csrc/rotary
|
|
|
```
|
|
|
|
|
|
Now you can start with ModelScope or Transformers.
|
|
|
|
|
|
## 🤗 Transformers
|
|
|
|
|
|
To use Qwen-Chat for the inference, all you need to do is to input a few lines of codes as demonstrated below. Remember to pass in the correct model names or paths, such as "Qwen/Qwen-7B-Chat" and "Qwen/Qwen-14B-Chat". However, **please make sure that you are using the latest code.**
|
|
|
|
|
|
```python
|
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
from transformers.generation import GenerationConfig
|
|
|
|
|
|
# Model names: "Qwen/Qwen-7B-Chat", "Qwen/Qwen-14B-Chat"
|
|
|
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen-7B-Chat", trust_remote_code=True)
|
|
|
|
|
|
# use bf16
|
|
|
# model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen-7B-Chat", device_map="auto", trust_remote_code=True, bf16=True).eval()
|
|
|
# use fp16
|
|
|
# model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen-7B-Chat", device_map="auto", trust_remote_code=True, fp16=True).eval()
|
|
|
# use cpu only
|
|
|
# model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen-7B-Chat", device_map="cpu", trust_remote_code=True).eval()
|
|
|
# use auto mode, automatically select precision based on the device.
|
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
|
"Qwen/Qwen-7B-Chat",
|
|
|
device_map="auto",
|
|
|
trust_remote_code=True
|
|
|
).eval()
|
|
|
|
|
|
# Specify hyperparameters for generation. But if you use transformers>=4.32.0, there is no need to do this.
|
|
|
# model.generation_config = GenerationConfig.from_pretrained("Qwen/Qwen-7B-Chat", trust_remote_code=True)
|
|
|
|
|
|
# 1st dialogue turn
|
|
|
response, history = model.chat(tokenizer, "你好", history=None)
|
|
|
print(response)
|
|
|
# 你好！很高兴为你提供帮助。
|
|
|
|
|
|
# 2nd dialogue turn
|
|
|
response, history = model.chat(tokenizer, "给我讲一个年轻人奋斗创业最终取得成功的故事。", history=history)
|
|
|
print(response)
|
|
|
# 这是一个关于一个年轻人奋斗创业最终取得成功的故事。
|
|
|
# 故事的主人公叫李明，他来自一个普通的家庭，父母都是普通的工人。从小，李明就立下了一个目标：要成为一名成功的企业家。
|
|
|
# 为了实现这个目标，李明勤奋学习，考上了大学。在大学期间，他积极参加各种创业比赛，获得了不少奖项。他还利用课余时间去实习，积累了宝贵的经验。
|
|
|
# 毕业后，李明决定开始自己的创业之路。他开始寻找投资机会，但多次都被拒绝了。然而，他并没有放弃。他继续努力，不断改进自己的创业计划，并寻找新的投资机会。
|
|
|
# 最终，李明成功地获得了一笔投资，开始了自己的创业之路。他成立了一家科技公司，专注于开发新型软件。在他的领导下，公司迅速发展起来，成为了一家成功的科技企业。
|
|
|
# 李明的成功并不是偶然的。他勤奋、坚韧、勇于冒险，不断学习和改进自己。他的成功也证明了，只要努力奋斗，任何人都有可能取得成功。
|
|
|
|
|
|
# 3rd dialogue turn
|
|
|
response, history = model.chat(tokenizer, "给这个故事起一个标题", history=history)
|
|
|
print(response)
|
|
|
# 《奋斗创业：一个年轻人的成功之路》
|
|
|
```
|
|
|
|
|
|
Running Qwen, the base language model, is also simple.
|
|
|
|
|
|
<details>
|
|
|
<summary>Running Qwen</summary>
|
|
|
|
|
|
```python
|
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
from transformers.generation import GenerationConfig
|
|
|
|
|
|
# Model names: "Qwen/Qwen-7B", "Qwen/Qwen-14B"
|
|
|
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen-7B", trust_remote_code=True)
|
|
|
# use bf16
|
|
|
# model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen-7B", device_map="auto", trust_remote_code=True, bf16=True).eval()
|
|
|
# use fp16
|
|
|
# model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen-7B", device_map="auto", trust_remote_code=True, fp16=True).eval()
|
|
|
# use cpu only
|
|
|
# model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen-7B", device_map="cpu", trust_remote_code=True).eval()
|
|
|
# use auto mode, automatically select precision based on the device.
|
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
|
"Qwen/Qwen-7B",
|
|
|
device_map="auto",
|
|
|
trust_remote_code=True
|
|
|
).eval()
|
|
|
|
|
|
# Specify hyperparameters for generation. But if you use transformers>=4.32.0, there is no need to do this.
|
|
|
# model.generation_config = GenerationConfig.from_pretrained("Qwen/Qwen-7B", trust_remote_code=True)
|
|
|
|
|
|
inputs = tokenizer('蒙古国的首都是乌兰巴托（Ulaanbaatar）\n冰岛的首都是雷克雅未克（Reykjavik）\n埃塞俄比亚的首都是', return_tensors='pt')
|
|
|
inputs = inputs.to(model.device)
|
|
|
pred = model.generate(**inputs)
|
|
|
print(tokenizer.decode(pred.cpu()[0], skip_special_tokens=True))
|
|
|
# 蒙古国的首都是乌兰巴托（Ulaanbaatar）\n冰岛的首都是雷克雅未克（Reykjavik）\n埃塞俄比亚的首都是亚的斯亚贝巴（Addis Ababa）...
|
|
|
```
|
|
|
|
|
|
</details>
|
|
|
|
|
|
<p id="DownloadModel">
|
|
|
In the event of a network issue while attempting to download model checkpoints and codes from HuggingFace, an alternative approach is to initially fetch the checkpoint from ModelScope and then load it from the local directory as outlined below:
|
|
|
</p>
|
|
|
|
|
|
```python
|
|
|
from modelscope import snapshot_download
|
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
|
|
|
# Downloading model checkpoint to a local dir model_dir
|
|
|
# model_dir = snapshot_download('qwen/Qwen-7B')
|
|
|
# model_dir = snapshot_download('qwen/Qwen-7B-Chat')
|
|
|
# model_dir = snapshot_download('qwen/Qwen-14B')
|
|
|
model_dir = snapshot_download('qwen/Qwen-14B-Chat')
|
|
|
|
|
|
# Loading local checkpoints
|
|
|
# trust_remote_code is still set as True since we still load codes from local dir instead of transformers
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)
|
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
|
model_dir,
|
|
|
device_map="auto",
|
|
|
trust_remote_code=True
|
|
|
).eval()
|
|
|
```
|
|
|
|
|
|
## 🤖 ModelScope
|
|
|
|
|
|
ModelScope is an open-source platform for Model-as-a-Service (MaaS), which provides flexible and cost-effective model service to AI developers. Similarly, you can run the models with ModelScope as shown below:
|
|
|
|
|
|
```python
|
|
|
from modelscope import AutoModelForCausalLM, AutoTokenizer
|
|
|
from modelscope import GenerationConfig
|
|
|
|
|
|
# Model names: "qwen/Qwen-7B-Chat", "qwen/Qwen-14B-Chat"
|
|
|
tokenizer = AutoTokenizer.from_pretrained("qwen/Qwen-7B-Chat", trust_remote_code=True)
|
|
|
model = AutoModelForCausalLM.from_pretrained("qwen/Qwen-7B-Chat", device_map="auto", trust_remote_code=True, fp16=True).eval()
|
|
|
model.generation_config = GenerationConfig.from_pretrained("Qwen/Qwen-7B-Chat", trust_remote_code=True) # 可指定不同的生成长度、top_p等相关超参
|
|
|
|
|
|
response, history = model.chat(tokenizer, "你好", history=None)
|
|
|
print(response)
|
|
|
response, history = model.chat(tokenizer, "浙江的省会在哪里？", history=history)
|
|
|
print(response)
|
|
|
response, history = model.chat(tokenizer, "它有什么好玩的景点", history=history)
|
|
|
print(response)
|
|
|
```
|
|
|
|
|
|
## Batch Inference
|
|
|
Qwen supports batch inference. With flash attention enabled, using batch inference can bring a 40% speedup. The example code is shown below:
|
|
|
```python
|
|
|
import torch
|
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer
|
|
|
from transformers import GenerationConfig
|
|
|
from qwen_generation_utils import make_context, decode_tokens, get_stop_words_ids
|
|
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained(
|
|
|
'./',
|
|
|
pad_token='<|extra_0|>',
|
|
|
eos_token='<|endoftext|>',
|
|
|
padding_side='left',
|
|
|
trust_remote_code=True
|
|
|
)
|
|
|
model = AutoModelForCausalLM.from_pretrained(
|
|
|
'./',
|
|
|
pad_token_id=tokenizer.pad_token_id,
|
|
|
device_map="auto",
|
|
|
trust_remote_code=True
|
|
|
).eval()
|
|
|
model.generation_config = GenerationConfig.from_pretrained('./', pad_token_id=tokenizer.pad_token_id)
|
|
|
|
|
|
all_raw_text = ["我想听你说爱我。", "今天我想吃点啥，甜甜的，推荐下", "我马上迟到了，怎么做才能不迟到"]
|
|
|
batch_raw_text = []
|
|
|
for q in all_raw_text:
|
|
|
raw_text, _ = make_context(
|
|
|
tokenizer,
|
|
|
q,
|
|
|
system="You are a helpful assistant.",
|
|
|
max_window_size=model.generation_config.max_window_size,
|
|
|
chat_format=model.generation_config.chat_format,
|
|
|
)
|
|
|
batch_raw_text.append(raw_text)
|
|
|
|
|
|
batch_input_ids = tokenizer(batch_raw_text, padding='longest')
|
|
|
batch_input_ids = torch.LongTensor(batch_input_ids['input_ids']).to(model.device)
|
|
|
batch_out_ids = model.generate(
|
|
|
batch_input_ids,
|
|
|
return_dict_in_generate=False,
|
|
|
generation_config=model.generation_config
|
|
|
)
|
|
|
padding_lens = [batch_input_ids[i].eq(tokenizer.pad_token_id).sum().item() for i in range(batch_input_ids.size(0))]
|
|
|
|
|
|
batch_response = [
|
|
|
decode_tokens(
|
|
|
batch_out_ids[i][padding_lens[i]:],
|
|
|
tokenizer,
|
|
|
raw_text_len=len(batch_raw_text[i]),
|
|
|
context_length=(batch_input_ids[i].size(0)-padding_lens[i]),
|
|
|
chat_format="chatml",
|
|
|
verbose=False,
|
|
|
errors='replace'
|
|
|
) for i in range(len(all_raw_text))
|
|
|
]
|
|
|
print(batch_response)
|
|
|
|
|
|
response, _ = model.chat(tokenizer, "我想听你说爱我。", history=None)
|
|
|
print(response)
|
|
|
|
|
|
response, _ = model.chat(tokenizer, "今天我想吃点啥，甜甜的，推荐下", history=None)
|
|
|
print(response)
|
|
|
|
|
|
response, _ = model.chat(tokenizer, "我马上迟到了，怎么做才能不迟到", history=None)
|
|
|
print(response)
|
|
|
```
|
|
|
|
|
|
## CPU
|
|
|
|
|
|
To deploy our models on CPU, we strongly advise you to use [qwen.cpp](https://github.com/QwenLM/qwen.cpp), which is a pure C++ implementation of Qwen and tiktoken. Check the repo for more details!
|
|
|
|
|
|
Also, it is also simple to directly run the model on CPU, which requires your specification of device:
|
|
|
|
|
|
```python
|
|
|
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen-7B-Chat", device_map="cpu", trust_remote_code=True).eval()
|
|
|
```
|
|
|
|
|
|
However, it is likely that you suffer from extremely low inference efficiency.
|
|
|
|
|
|
## Multiple GPUs
|
|
|
|
|
|
If you suffer from lack of GPU memory and you would like to run the model on more than 1 GPU, you can directly use the default loading method, which is now supported by Transformers. The previous method based on `utils.py` is deprecated.
|
|
|
|
|
|
However, though this method is simple, the efficiency of the native pipeline parallelism is low. We advise you to use vLLM with FastChat and please read [the section](../vllm/README.md) for deployment. |