Llama2-7B-Chat大模型微调实战
2024-01-17 本文已影响0人
梅西爱骑车
Llama2-7B-Chat大模型微调实战
Llama2系列是Meta开发并公开的大型语言模型（LLMs），有7B、13B和70B三种不同参数大小的模型，每种参数大小分别对应一个预训练和一个微调的版本。
微调版本称为Llama2-Chat，使用了和 ChatGPT 相似的技术，针对对话进行了优化。相比于 Llama1，Llama2的训练数据多了 40%，上下文长度翻倍，并采用了分组查询注意力机制。特别地，Llama2的预训练模型在2万亿的token 上训练，精调的Llama2-Chat模型在100万人类标记数据上进行进一步训练得到。Llama-2-Chat模型在Meta多数基准上优于开源聊天模型，并且在Meta和安全性的人类评估中，与一些流行的闭源模型如ChatGPT和PaLM相当。
Llama2-7B-Chat是具有70亿参数的微调模型，本文将以Llama2-7B-Chat为例，为您介绍如何在PAI-DSW中微调Llama2大模型。
运行环境要求
Python环境3.9以上，GPU推荐使用V100（32GB）
准备工作
下载Llama2-7B-Chat
首先，下载模型文件。 您可以选择直接执行下面脚本下载，也可以选择从ModelScope下载模型（Modelscope模型主页）。 运行如下代码，会根据当前地域为您选择合适的下载地址，并将模型下载到当前目录。
wget后自解压的文件目录如下：
Llama-2-7b-chat目录结构
下载和安装环境
接着下载和安装所需要的环境。(记得安装完成之后重启notebook，不然这些包导入不进来发生报错)
- ColossalAI是大规模并行AI训练系统，在本例中我们使用该框架进行模型微调。
- transformers是基于transformers模型结构的预训练语言库。
-
gradio是一个快速构建机器学习Web展示页面的开源库。
依赖安装
下载json训练数据
下载训练所需的数据，这里我们提供的一份创意生成数据，包括发言稿生成等内容。
您也可以参考该格式，自行准备所需数据。
训练数据和测试数据格式
微调模型
! pwd
! sh ColossalAI/applications/Chat/examples/train_sft.sh
试用模型
模型训练完成后，下载我们提供的webUI demo，使用Gradio构建页面，使用pipeline进行推理，试玩微调完成的模型（注意模型地址替换为自己训练好的模型地址）。建议用英文进行提问。
import gradio as gr
import requests
import json
from transformers import AutoTokenizer, AutoModelForCausalLM
#模型地址替换为自己训练好的模型地址
tokenizer = AutoTokenizer.from_pretrained("llama2-7b",trust_remote_code=True)
#模型地址替换为自己训练好的模型地址
model = AutoModelForCausalLM.from_pretrained("llama2-7b",trust_remote_code=True).eval().half().cuda()
def inference(text):
from transformers import pipeline
pipe = pipeline("text-generation", model=model, tokenizer=tokenizer,device='cuda:0', max_new_tokens=400)
res=pipe(text)
return res[0]['generated_text'][len(text):]
demo = gr.Blocks()
with demo:
input_prompt = gr.Textbox(label="请输入需求", value="请以软件工程师的身份，写一篇入职的发言稿。", lines=6)
generated_txt = gr.Textbox(lines=6)
b1 = gr.Button("发送")
b1.click(inference, inputs=[input_prompt], outputs=generated_txt)
demo.launch(enable_queue=True, share=False)
完整的Jupyter文件llama2_notebook.ipynb如下：
{
"cells": [
{
"cell_type": "markdown",
"id": "d43602a6-233c-4281-b468-30fdd92b57b7",
"metadata": {},
"source": [
"# Llama2-7B-Chat大模型微调实战"
]
},
{
"cell_type": "markdown",
"id": "be1672a2-32a8-4def-807c-1423791f3ed7",
"metadata": {},
"source": [
"Llama2系列是Meta开发并公开的大型语言模型（LLMs），有7B、13B和70B三种不同参数大小的模型，每种参数大小分别对应一个预训练和一个微调的版本。\n",
"\n",
"微调版本称为Llama2-Chat，使用了和 ChatGPT 相似的技术，针对对话进行了优化。相比于 Llama1，Llama2的训练数据多了 40%，上下文长度翻倍，并采用了分组查询注意力机制。特别地，Llama2的预训练模型在2万亿的token 上训练，精调的Llama2-Chat模型在100万人类标记数据上进行进一步训练得到。Llama-2-Chat模型在Meta多数基准上优于开源聊天模型，并且在Meta和安全性的人类评估中，与一些流行的闭源模型如ChatGPT和PaLM相当。\n",
"\n",
"Llama2-7B-Chat是具有70亿参数的微调模型，本文将以Llama2-7B-Chat为例，为您介绍如何在PAI-DSW中微调Llama2大模型。\n"
]
},
{
"cell_type": "markdown",
"id": "eb728f1f-6b56-41dd-867f-1a133abf3fad",
"metadata": {
"jp-MarkdownHeadingCollapsed": true,
"tags": []
},
"source": [
"### 注意：使用此模型受Meta许可证的约束。在使用模型前，请确认已经前往[自定义可商用开源协议](https://ai.meta.com/resources/models-and-libraries/llama-downloads/)网站并完成申请。"
]
},
{
"cell_type": "markdown",
"id": "7aa624e9-5329-4547-9d79-720da66888de",
"metadata": {
"tags": []
},
"source": [
"## 运行环境要求"
]
},
{
"cell_type": "markdown",
"id": "99a786ce-0ea1-4295-94e2-8e34b26df9fe",
"metadata": {},
"source": [
"Python环境3.9以上，GPU推荐使用V100（32GB）"
]
},
{
"cell_type": "markdown",
"id": "6419088a-76b9-47bd-be1c-91829db2745e",
"metadata": {
"tags": []
},
"source": [
"## 准备工作"
]
},
{
"cell_type": "markdown",
"id": "516b74dd-77dd-4800-b79d-0741223891fc",
"metadata": {},
"source": [
"### 下载Llama2-7B-Chat"
]
},
{
"cell_type": "markdown",
"id": "da3c86c3-5c18-4067-8915-0d3e8b69b8eb",
"metadata": {
"tags": []
},
"source": [
"首先，下载模型文件。\n",
"您可以选择直接执行下面脚本下载，也可以选择从ModelScope下载模型（[Modelscope模型主页](https://modelscope.cn/models/modelscope/Llama-2-7b-chat-ms/summary)）。\n",
"运行如下代码，会根据当前地域为您选择合适的下载地址，并将模型下载到当前目录。"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e91caf0b-a89d-4230-9322-6e924b93d7e6",
"metadata": {
"ExecutionIndicator": {
"show": true
},
"execution": {
"iopub.execute_input": "2024-01-18T01:43:22.641118Z",
"iopub.status.busy": "2024-01-18T01:43:22.640823Z"
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"--2024-01-18 09:43:22-- https://atp-modelzoo.oss-cn-hangzhou-internal.aliyuncs.com/release/tutorials/llama2/llama2-7b.tar.gz\n",
"正在解析主机 atp-modelzoo.oss-cn-hangzhou-internal.aliyuncs.com (atp-modelzoo.oss-cn-hangzhou-internal.aliyuncs.com)... 100.118.28.44, 100.118.28.50, 100.118.28.49, ...\n",
"正在连接 atp-modelzoo.oss-cn-hangzhou-internal.aliyuncs.com (atp-modelzoo.oss-cn-hangzhou-internal.aliyuncs.com)|100.118.28.44|:443... 已连接。\n",
"已发出 HTTP 请求，正在等待回应... 200 OK\n",
"长度： 12621474981 (12G) [application/gzip]\n",
"正在保存至: ‘llama2-7b.tar.gz’\n",
"\n",
"llama2-7b.tar.gz 100%[===================>] 11.75G 72.2MB/s 用时 2m 40s \n",
"\n",
"2024-01-18 09:46:02 (75.2 MB/s) - 已保存 ‘llama2-7b.tar.gz’ [12621474981/12621474981])\n",
"\n",
"llama2-7b/\n",
"llama2-7b/generation_config.json\n",
"llama2-7b/gitattributes.txt\n",
"llama2-7b/config.json\n",
"llama2-7b/pytorch_model.bin.index.json\n",
"llama2-7b/USE_POLICY.md\n",
"llama2-7b/README.md\n",
"llama2-7b/LICENSE.txt\n",
"llama2-7b/pytorch_model-00003-of-00003.bin\n",
"llama2-7b/special_tokens_map.json\n",
"llama2-7b/Responsible-Use-Guide.pdf\n",
"llama2-7b/tokenizer.model\n",
"llama2-7b/pytorch_model-00001-of-00003.bin\n"
]
}
],
"source": [
"import os\n",
"dsw_region = os.environ.get(\"dsw_region\")\n",
"url_link = {\n",
" \"cn-shanghai\": \"https://atp-modelzoo-sh.oss-cn-shanghai-internal.aliyuncs.com/release/tutorials/llama2/llama2-7b.tar.gz\",\n",
" \"cn-hangzhou\": \"https://atp-modelzoo.oss-cn-hangzhou-internal.aliyuncs.com/release/tutorials/llama2/llama2-7b.tar.gz\",\n",
" \"cn-shenzhen\": \"https://atp-modelzoo-sz.oss-cn-shenzhen-internal.aliyuncs.com/release/tutorials/llama2/llama2-7b.tar.gz\",\n",
" \"cn-beijing\": \"https://atp-modelzoo-bj.oss-cn-beijing-internal.aliyuncs.com/release/tutorials/llama2/llama2-7b.tar.gz\", \n",
"}\n",
"\n",
"path = url_link[dsw_region] if dsw_region in url_link else \"https://atp-modelzoo.oss-cn-hangzhou.aliyuncs.com/release/tutorials/llama2/llama2-7b.tar.gz\"\n",
"os.environ['LINK_CHAT'] = path\n",
"!wget $LINK_CHAT\n",
"!tar -zxvf llama2-7b.tar.gz"
]
},
{
"cell_type": "markdown",
"id": "4b79abd9-bedb-42d3-8f79-04ab8635e4e8",
"metadata": {},
"source": [
"### 下载和安装环境\n"
]
},
{
"cell_type": "markdown",
"id": "e0358475-0163-495d-9434-eefc6f6afa1a",
"metadata": {},
"source": [
"接着下载和安装所需要的环境。(记得安装完成之后重启notebook，不然这些包导入不进来发生报错)\n",
"- ColossalAI是大规模并行AI训练系统，在本例中我们使用该框架进行模型微调。\n",
"- transformers是基于transformers模型结构的预训练语言库。\n",
"- gradio是一个快速构建机器学习Web展示页面的开源库。"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "808fd780-bc67-44eb-96bd-fd716d2b2bb3",
"metadata": {
"ExecutionIndicator": {
"show": false
},
"scrolled": true,
"tags": []
},
"outputs": [],
"source": [
"! wget https://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/release/tutorials/llama2/ColossalAI.tar.gz\n",
"! tar -zxvf ColossalAI.tar.gz\n",
"! pip install ColossalAI/.\n",
"! pip install ColossalAI/applications/Chat/.\n",
"! pip install transformers==4.30.0\n",
"! pip install gradio==4.7\n",
"! pip install tokenizers==0.13.3"
]
},
{
"cell_type": "markdown",
"id": "2e80d05a-6f40-438d-9a6a-927ad2279df4",
"metadata": {},
"source": [
"### 下载示例训练数据"
]
},
{
"cell_type": "markdown",
"id": "3f65757f-4275-48d6-a8b4-79d986af7ca0",
"metadata": {},
"source": [
"下载训练所需的数据，这里我们提供的一份创意生成数据，包括发言稿生成等内容。\n",
"\n",
"您也可以参考该格式，自行准备所需数据。"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ad6b5ae5-2e0c-42f6-b28e-73dc93d16e9f",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"! wget https://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/release/tutorials/llama2/llama_data.json\n",
"! wget https://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/release/tutorials/llama2/llama_test.json"
]
},
{
"cell_type": "markdown",
"id": "0313c446-ccf7-4129-89a7-c677043c99b9",
"metadata": {},
"source": [
"## 微调模型"
]
},
{
"cell_type": "markdown",
"id": "8220a63c-f752-47d1-8338-05c85a95329c",
"metadata": {},
"source": [
"您可以使用已经写好的训练脚本，进行模型训练，约占用31G显存。"
]
},
{
"cell_type": "code",
"execution_count": 22,
"id": "32e01f3b-e9ed-4584-9091-44b10c9eceb0",
"metadata": {
"ExecutionIndicator": {
"show": true
},
"execution": {
"iopub.execute_input": "2023-07-27T06:47:14.150932Z",
"iopub.status.busy": "2023-07-27T06:47:14.150575Z",
"iopub.status.idle": "2023-07-27T07:03:06.676378Z",
"shell.execute_reply": "2023-07-27T07:03:06.675729Z",
"shell.execute_reply.started": "2023-07-27T06:47:14.150908Z"
},
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"/mnt/workspace/demos/llama2_notebook\n",
"\u001b[2;36m[07/27/23 14:47:18]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
"\u001b[2;36m \u001b[0m \u001b[35m/home/pai/lib/python3.9/site-packages/colossalai/co\u001b[0m\n",
"\u001b[2;36m \u001b[0m \u001b[35mntext/\u001b[0m\u001b[95mparallel_context.py\u001b[0m:\u001b[1;36m522\u001b[0m set_device \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: process rank \u001b[1;36m0\u001b[0m is \n",
"\u001b[2;36m \u001b[0m bound to device \u001b[1;36m0\u001b[0m \n",
"dsw-73160-7f4959c7b5-m7p8x:2322:2322 [0] NCCL INFO Bootstrap : Using eth0:10.224.144.113<0>\n",
"dsw-73160-7f4959c7b5-m7p8x:2322:2322 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation\n",
"\n",
"dsw-73160-7f4959c7b5-m7p8x:2322:2322 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]\n",
"dsw-73160-7f4959c7b5-m7p8x:2322:2322 [0] NCCL INFO NET/Socket : Using [0]eth0:10.224.144.113<0>\n",
"dsw-73160-7f4959c7b5-m7p8x:2322:2322 [0] NCCL INFO Using network Socket\n",
"NCCL version 2.10.3+cuda11.3\n",
"dsw-73160-7f4959c7b5-m7p8x:2322:2398 [0] NCCL INFO NCCL_MAX_NCHANNELS set by environment to 2.\n",
"dsw-73160-7f4959c7b5-m7p8x:2322:2398 [0] NCCL INFO NCCL_MIN_NCHANNELS set by environment to 2.\n",
"dsw-73160-7f4959c7b5-m7p8x:2322:2398 [0] NCCL INFO Channel 00/02 : 0\n",
"dsw-73160-7f4959c7b5-m7p8x:2322:2398 [0] NCCL INFO Channel 01/02 : 0\n",
"dsw-73160-7f4959c7b5-m7p8x:2322:2398 [0] NCCL INFO Trees [0] -1/-1/-1->0->-1 [1] -1/-1/-1->0->-1\n",
"dsw-73160-7f4959c7b5-m7p8x:2322:2398 [0] NCCL INFO Connected all rings\n",
"dsw-73160-7f4959c7b5-m7p8x:2322:2398 [0] NCCL INFO Connected all trees\n",
"dsw-73160-7f4959c7b5-m7p8x:2322:2398 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer\n",
"dsw-73160-7f4959c7b5-m7p8x:2322:2398 [0] NCCL INFO comm 0x7f0824002fb0 rank 0 nranks 1 cudaDev 0 busId 80 - Init COMPLETE\n",
"\u001b[2;36m[07/27/23 14:47:19]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
"\u001b[2;36m \u001b[0m \u001b[35m/home/pai/lib/python3.9/site-packages/colossalai/co\u001b[0m\n",
"\u001b[2;36m \u001b[0m \u001b[35mntext/\u001b[0m\u001b[95mparallel_context.py\u001b[0m:\u001b[1;36m558\u001b[0m set_seed \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: initialized seed on\n",
"\u001b[2;36m \u001b[0m rank \u001b[1;36m0\u001b[0m, numpy: \u001b[1;36m42\u001b[0m, python random: \u001b[1;36m42\u001b[0m, \n",
"\u001b[2;36m \u001b[0m ParallelMode.DATA: \u001b[1;36m42\u001b[0m, ParallelMode.TENSOR: \u001b[1;36m42\u001b[0m,the \n",
"\u001b[2;36m \u001b[0m default parallel seed is ParallelMode.DATA. \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
"\u001b[2;36m \u001b[0m \u001b[35m/home/pai/lib/python3.9/site-packages/colossalai/\u001b[0m\u001b[95min\u001b[0m\n",
"\u001b[2;36m \u001b[0m \u001b[95mitialize.py\u001b[0m:\u001b[1;36m115\u001b[0m launch \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Distributed \n",
"\u001b[2;36m \u001b[0m environment is initialized, data parallel size: \u001b[1;36m1\u001b[0m, \n",
"\u001b[2;36m \u001b[0m pipeline parallel size: \u001b[1;36m1\u001b[0m, tensor parallel size: \u001b[1;36m1\u001b[0m \n",
"/home/pai/lib/python3.9/site-packages/colossalai/booster/booster.py:70: UserWarning: The plugin will control the accelerator, so the device argument will be ignored.\n",
" warnings.warn('The plugin will control the accelerator, so the device argument will be ignored.')\n",
"/home/pai/lib/python3.9/site-packages/colossalai/booster/booster.py:76: UserWarning: The plugin will control the precision, so the mixed_precision argument will be ignored.\n",
" warnings.warn('The plugin will control the precision, so the mixed_precision argument will be ignored.')\n",
"Loading checkpoint shards: 100%|██████████████████| 3/3 [00:13<00:00, 4.42s/it]\n",
"\u001b[2;36m[07/27/23 14:48:26]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
"\u001b[2;36m \u001b[0m \u001b[35m/mnt/workspace/demos/llama2_notebook/ColossalAI/app\u001b[0m\n",
"\u001b[2;36m \u001b[0m \u001b[35mlications/Chat/examples/\u001b[0m\u001b[95msft_dataset_diy.py\u001b[0m:\u001b[1;36m136\u001b[0m \n",
"\u001b[2;36m \u001b[0m __init__ \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Loading data\u001b[33m...\u001b[0m \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
"\u001b[2;36m \u001b[0m \u001b[35m/mnt/workspace/demos/llama2_notebook/ColossalAI/app\u001b[0m\n",
"\u001b[2;36m \u001b[0m \u001b[35mlications/Chat/examples/\u001b[0m\u001b[95msft_dataset_diy.py\u001b[0m:\u001b[1;36m138\u001b[0m \n",
"\u001b[2;36m \u001b[0m __init__ \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Loaded \u001b[1;36m250\u001b[0m \n",
"\u001b[2;36m \u001b[0m examples. \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
"\u001b[2;36m \u001b[0m \u001b[35m/mnt/workspace/demos/llama2_notebook/ColossalAI/app\u001b[0m\n",
"\u001b[2;36m \u001b[0m \u001b[35mlications/Chat/examples/\u001b[0m\u001b[95msft_dataset_diy.py\u001b[0m:\u001b[1;36m144\u001b[0m \n",
"\u001b[2;36m \u001b[0m __init__ \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Formatting \n",
"\u001b[2;36m \u001b[0m inputs\u001b[33m...\u001b[0m \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
"\u001b[2;36m \u001b[0m \u001b[35m/mnt/workspace/demos/llama2_notebook/ColossalAI/app\u001b[0m\n",
"\u001b[2;36m \u001b[0m \u001b[35mlications/Chat/examples/\u001b[0m\u001b[95msft_dataset_diy.py\u001b[0m:\u001b[1;36m152\u001b[0m \n",
"\u001b[2;36m \u001b[0m __init__ \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Tokenizing \n",
"\u001b[2;36m \u001b[0m inputs\u001b[33m...\u001b[0m This may take some time\u001b[33m...\u001b[0m \n",
"\u001b[2;36m[07/27/23 14:48:27]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
"\u001b[2;36m \u001b[0m \u001b[35m/mnt/workspace/demos/llama2_notebook/ColossalAI/app\u001b[0m\n",
"\u001b[2;36m \u001b[0m \u001b[35mlications/Chat/examples/\u001b[0m\u001b[95msft_dataset_diy.py\u001b[0m:\u001b[1;36m136\u001b[0m \n",
"\u001b[2;36m \u001b[0m __init__ \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Loading data\u001b[33m...\u001b[0m \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
"\u001b[2;36m \u001b[0m \u001b[35m/mnt/workspace/demos/llama2_notebook/ColossalAI/app\u001b[0m\n",
"\u001b[2;36m \u001b[0m \u001b[35mlications/Chat/examples/\u001b[0m\u001b[95msft_dataset_diy.py\u001b[0m:\u001b[1;36m138\u001b[0m \n",
"\u001b[2;36m \u001b[0m __init__ \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Loaded \u001b[1;36m6\u001b[0m examples. \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
"\u001b[2;36m \u001b[0m \u001b[35m/mnt/workspace/demos/llama2_notebook/ColossalAI/app\u001b[0m\n",
"\u001b[2;36m \u001b[0m \u001b[35mlications/Chat/examples/\u001b[0m\u001b[95msft_dataset_diy.py\u001b[0m:\u001b[1;36m144\u001b[0m \n",
"\u001b[2;36m \u001b[0m __init__ \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Formatting \n",
"\u001b[2;36m \u001b[0m inputs\u001b[33m...\u001b[0m \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
"\u001b[2;36m \u001b[0m \u001b[35m/mnt/workspace/demos/llama2_notebook/ColossalAI/app\u001b[0m\n",
"\u001b[2;36m \u001b[0m \u001b[35mlications/Chat/examples/\u001b[0m\u001b[95msft_dataset_diy.py\u001b[0m:\u001b[1;36m152\u001b[0m \n",
"\u001b[2;36m \u001b[0m __init__ \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Tokenizing \n",
"\u001b[2;36m \u001b[0m inputs\u001b[33m...\u001b[0m This may take some time\u001b[33m...\u001b[0m \n",
"steps: 0%| | 0/1240 [00:00<?, ?it/s]dsw-73160-7f4959c7b5-m7p8x:2322:2457 [0] NCCL INFO Channel 00/02 : 0\n",
"dsw-73160-7f4959c7b5-m7p8x:2322:2457 [0] NCCL INFO Channel 01/02 : 0\n",
"dsw-73160-7f4959c7b5-m7p8x:2322:2457 [0] NCCL INFO Trees [0] -1/-1/-1->0->-1 [1] -1/-1/-1->0->-1\n",
"dsw-73160-7f4959c7b5-m7p8x:2322:2457 [0] NCCL INFO Connected all rings\n",
"dsw-73160-7f4959c7b5-m7p8x:2322:2457 [0] NCCL INFO Connected all trees\n",
"dsw-73160-7f4959c7b5-m7p8x:2322:2457 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer\n",
"dsw-73160-7f4959c7b5-m7p8x:2322:2457 [0] NCCL INFO comm 0x7f009c002fb0 rank 0 nranks 1 cudaDev 0 busId 80 - Init COMPLETE\n",
"/home/pai/lib/python3.9/site-packages/torch/optim/lr_scheduler.py:131: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`. Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate\n",
" warnings.warn(\"Detected call of `lr_scheduler.step()` before `optimizer.step()`. \"\n",
"steps: 5%|█▋ | 62/1240 [00:36<11:27, 1.71it/s]\u001b[2;36m[07/27/23 14:49:04]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
"\u001b[2;36m \u001b[0m \u001b[35m/home/pai/lib/python3.9/site-packages/coati/trainer\u001b[0m\n",
"\u001b[2;36m \u001b[0m \u001b[35m/\u001b[0m\u001b[95msft.py\u001b[0m:\u001b[1;36m98\u001b[0m _eval \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Eval Epoch \u001b[1;36m0\u001b[0m/\u001b[1;36m20\u001b[0m \n",
"\u001b[2;36m \u001b[0m loss \u001b[1;36m0.5631510416666666\u001b[0m \n",
"steps: 10%|███▎ | 124/1240 [01:13<10:47, 1.72it/s]\u001b[2;36m[07/27/23 14:49:41]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
"\u001b[2;36m \u001b[0m \u001b[35m/home/pai/lib/python3.9/site-packages/coati/trainer\u001b[0m\n",
"\u001b[2;36m \u001b[0m \u001b[35m/\u001b[0m\u001b[95msft.py\u001b[0m:\u001b[1;36m98\u001b[0m _eval \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Eval Epoch \u001b[1;36m1\u001b[0m/\u001b[1;36m20\u001b[0m \n",
"\u001b[2;36m \u001b[0m loss \u001b[1;36m0.5260416666666666\u001b[0m \n",
"steps: 15%|████▉ | 186/1240 [01:49<10:06, 1.74it/s]\u001b[2;36m[07/27/23 14:50:17]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
"\u001b[2;36m \u001b[0m \u001b[35m/home/pai/lib/python3.9/site-packages/coati/trainer\u001b[0m\n",
"\u001b[2;36m \u001b[0m \u001b[35m/\u001b[0m\u001b[95msft.py\u001b[0m:\u001b[1;36m98\u001b[0m _eval \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Eval Epoch \u001b[1;36m2\u001b[0m/\u001b[1;36m20\u001b[0m \n",
"\u001b[2;36m \u001b[0m loss \u001b[1;36m0.5039876302083334\u001b[0m \n",
"steps: 20%|██████▌ | 248/1240 [02:26<09:38, 1.72it/s]\u001b[2;36m[07/27/23 14:50:54]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
"\u001b[2;36m \u001b[0m \u001b[35m/home/pai/lib/python3.9/site-packages/coati/trainer\u001b[0m\n",
"\u001b[2;36m \u001b[0m \u001b[35m/\u001b[0m\u001b[95msft.py\u001b[0m:\u001b[1;36m98\u001b[0m _eval \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Eval Epoch \u001b[1;36m3\u001b[0m/\u001b[1;36m20\u001b[0m \n",
"\u001b[2;36m \u001b[0m loss \u001b[1;36m0.4903971354166667\u001b[0m \n",
"steps: 25%|████████▎ | 310/1240 [03:02<09:06, 1.70it/s]\u001b[2;36m[07/27/23 14:51:30]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
"\u001b[2;36m \u001b[0m \u001b[35m/home/pai/lib/python3.9/site-packages/coati/trainer\u001b[0m\n",
"\u001b[2;36m \u001b[0m \u001b[35m/\u001b[0m\u001b[95msft.py\u001b[0m:\u001b[1;36m98\u001b[0m _eval \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Eval Epoch \u001b[1;36m4\u001b[0m/\u001b[1;36m20\u001b[0m \n",
"\u001b[2;36m \u001b[0m loss \u001b[1;36m0.4816080729166667\u001b[0m \n",
"steps: 30%|█████████▉ | 372/1240 [03:39<08:20, 1.74it/s]\u001b[2;36m[07/27/23 14:52:07]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
"\u001b[2;36m \u001b[0m \u001b[35m/home/pai/lib/python3.9/site-packages/coati/trainer\u001b[0m\n",
"\u001b[2;36m \u001b[0m \u001b[35m/\u001b[0m\u001b[95msft.py\u001b[0m:\u001b[1;36m98\u001b[0m _eval \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Eval Epoch \u001b[1;36m5\u001b[0m/\u001b[1;36m20\u001b[0m \n",
"\u001b[2;36m \u001b[0m loss \u001b[1;36m0.4752604166666667\u001b[0m \n",
"steps: 35%|███████████▌ | 434/1240 [04:16<07:51, 1.71it/s]\u001b[2;36m[07/27/23 14:52:43]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
"\u001b[2;36m \u001b[0m \u001b[35m/home/pai/lib/python3.9/site-packages/coati/trainer\u001b[0m\n",
"\u001b[2;36m \u001b[0m \u001b[35m/\u001b[0m\u001b[95msft.py\u001b[0m:\u001b[1;36m98\u001b[0m _eval \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Eval Epoch \u001b[1;36m6\u001b[0m/\u001b[1;36m20\u001b[0m \n",
"\u001b[2;36m \u001b[0m loss \u001b[1;36m0.4700520833333333\u001b[0m \n",
"steps: 40%|█████████████▏ | 496/1240 [04:52<07:18, 1.70it/s]\u001b[2;36m[07/27/23 14:53:20]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
"\u001b[2;36m \u001b[0m \u001b[35m/home/pai/lib/python3.9/site-packages/coati/trainer\u001b[0m\n",
"\u001b[2;36m \u001b[0m \u001b[35m/\u001b[0m\u001b[95msft.py\u001b[0m:\u001b[1;36m98\u001b[0m _eval \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Eval Epoch \u001b[1;36m7\u001b[0m/\u001b[1;36m20\u001b[0m \n",
"\u001b[2;36m \u001b[0m loss \u001b[1;36m0.46533203125\u001b[0m \n",
"steps: 45%|██████████████▊ | 558/1240 [05:29<06:37, 1.71it/s]\u001b[2;36m[07/27/23 14:53:57]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
"\u001b[2;36m \u001b[0m \u001b[35m/home/pai/lib/python3.9/site-packages/coati/trainer\u001b[0m\n",
"\u001b[2;36m \u001b[0m \u001b[35m/\u001b[0m\u001b[95msft.py\u001b[0m:\u001b[1;36m98\u001b[0m _eval \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Eval Epoch \u001b[1;36m8\u001b[0m/\u001b[1;36m20\u001b[0m \n",
"\u001b[2;36m \u001b[0m loss \u001b[1;36m0.4623209635416667\u001b[0m \n",
"steps: 50%|████████████████▌ | 620/1240 [06:05<05:53, 1.76it/s]\u001b[2;36m[07/27/23 14:54:33]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
"\u001b[2;36m \u001b[0m \u001b[35m/home/pai/lib/python3.9/site-packages/coati/trainer\u001b[0m\n",
"\u001b[2;36m \u001b[0m \u001b[35m/\u001b[0m\u001b[95msft.py\u001b[0m:\u001b[1;36m98\u001b[0m _eval \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Eval Epoch \u001b[1;36m9\u001b[0m/\u001b[1;36m20\u001b[0m \n",
"\u001b[2;36m \u001b[0m loss \u001b[1;36m0.4593912760416667\u001b[0m \n",
"steps: 55%|██████████████████▏ | 682/1240 [06:42<05:26, 1.71it/s]\u001b[2;36m[07/27/23 14:55:10]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
"\u001b[2;36m \u001b[0m \u001b[35m/home/pai/lib/python3.9/site-packages/coati/trainer\u001b[0m\n",
"\u001b[2;36m \u001b[0m \u001b[35m/\u001b[0m\u001b[95msft.py\u001b[0m:\u001b[1;36m98\u001b[0m _eval \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Eval Epoch \u001b[1;36m10\u001b[0m/\u001b[1;36m20\u001b[0m \n",
"\u001b[2;36m \u001b[0m loss \u001b[1;36m0.4574381510416667\u001b[0m \n",
"steps: 60%|███████████████████▊ | 744/1240 [07:18<04:50, 1.71it/s]\u001b[2;36m[07/27/23 14:55:46]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
"\u001b[2;36m \u001b[0m \u001b[35m/home/pai/lib/python3.9/site-packages/coati/trainer\u001b[0m\n",
"\u001b[2;36m \u001b[0m \u001b[35m/\u001b[0m\u001b[95msft.py\u001b[0m:\u001b[1;36m98\u001b[0m _eval \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Eval Epoch \u001b[1;36m11\u001b[0m/\u001b[1;36m20\u001b[0m \n",
"\u001b[2;36m \u001b[0m loss \u001b[1;36m0.454833984375\u001b[0m \n",
"steps: 65%|█████████████████████▍ | 806/1240 [07:55<04:14, 1.71it/s]\u001b[2;36m[07/27/23 14:56:23]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
"\u001b[2;36m \u001b[0m \u001b[35m/home/pai/lib/python3.9/site-packages/coati/trainer\u001b[0m\n",
"\u001b[2;36m \u001b[0m \u001b[35m/\u001b[0m\u001b[95msft.py\u001b[0m:\u001b[1;36m98\u001b[0m _eval \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Eval Epoch \u001b[1;36m12\u001b[0m/\u001b[1;36m20\u001b[0m \n",
"\u001b[2;36m \u001b[0m loss \u001b[1;36m0.45361328125\u001b[0m \n",
"steps: 70%|███████████████████████ | 868/1240 [08:32<03:37, 1.71it/s]\u001b[2;36m[07/27/23 14:56:59]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
"\u001b[2;36m \u001b[0m \u001b[35m/home/pai/lib/python3.9/site-packages/coati/trainer\u001b[0m\n",
"\u001b[2;36m \u001b[0m \u001b[35m/\u001b[0m\u001b[95msft.py\u001b[0m:\u001b[1;36m98\u001b[0m _eval \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Eval Epoch \u001b[1;36m13\u001b[0m/\u001b[1;36m20\u001b[0m \n",
"\u001b[2;36m \u001b[0m loss \u001b[1;36m0.4524739583333333\u001b[0m \n",
"steps: 75%|████████████████████████▊ | 930/1240 [09:08<03:01, 1.71it/s]\u001b[2;36m[07/27/23 14:57:36]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
"\u001b[2;36m \u001b[0m \u001b[35m/home/pai/lib/python3.9/site-packages/coati/trainer\u001b[0m\n",
"\u001b[2;36m \u001b[0m \u001b[35m/\u001b[0m\u001b[95msft.py\u001b[0m:\u001b[1;36m98\u001b[0m _eval \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Eval Epoch \u001b[1;36m14\u001b[0m/\u001b[1;36m20\u001b[0m \n",
"\u001b[2;36m \u001b[0m loss \u001b[1;36m0.451416015625\u001b[0m \n",
"steps: 80%|██████████████████████████▍ | 992/1240 [09:45<02:24, 1.72it/s]\u001b[2;36m[07/27/23 14:58:13]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
"\u001b[2;36m \u001b[0m \u001b[35m/home/pai/lib/python3.9/site-packages/coati/trainer\u001b[0m\n",
"\u001b[2;36m \u001b[0m \u001b[35m/\u001b[0m\u001b[95msft.py\u001b[0m:\u001b[1;36m98\u001b[0m _eval \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Eval Epoch \u001b[1;36m15\u001b[0m/\u001b[1;36m20\u001b[0m \n",
"\u001b[2;36m \u001b[0m loss \u001b[1;36m0.4507649739583333\u001b[0m \n",
"steps: 85%|███████████████████████████▏ | 1054/1240 [10:21<01:48, 1.71it/s]\u001b[2;36m[07/27/23 14:58:49]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
"\u001b[2;36m \u001b[0m \u001b[35m/home/pai/lib/python3.9/site-packages/coati/trainer\u001b[0m\n",
"\u001b[2;36m \u001b[0m \u001b[35m/\u001b[0m\u001b[95msft.py\u001b[0m:\u001b[1;36m98\u001b[0m _eval \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Eval Epoch \u001b[1;36m16\u001b[0m/\u001b[1;36m20\u001b[0m \n",
"\u001b[2;36m \u001b[0m loss \u001b[1;36m0.4506022135416667\u001b[0m \n",
"steps: 90%|████████████████████████████▊ | 1116/1240 [10:58<01:11, 1.73it/s]\u001b[2;36m[07/27/23 14:59:26]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
"\u001b[2;36m \u001b[0m \u001b[35m/home/pai/lib/python3.9/site-packages/coati/trainer\u001b[0m\n",
"\u001b[2;36m \u001b[0m \u001b[35m/\u001b[0m\u001b[95msft.py\u001b[0m:\u001b[1;36m98\u001b[0m _eval \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Eval Epoch \u001b[1;36m17\u001b[0m/\u001b[1;36m20\u001b[0m \n",
"\u001b[2;36m \u001b[0m loss \u001b[1;36m0.4505208333333333\u001b[0m \n",
"steps: 95%|██████████████████████████████▍ | 1178/1240 [11:35<00:36, 1.71it/s]\u001b[2;36m[07/27/23 15:00:02]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
"\u001b[2;36m \u001b[0m \u001b[35m/home/pai/lib/python3.9/site-packages/coati/trainer\u001b[0m\n",
"\u001b[2;36m \u001b[0m \u001b[35m/\u001b[0m\u001b[95msft.py\u001b[0m:\u001b[1;36m98\u001b[0m _eval \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Eval Epoch \u001b[1;36m18\u001b[0m/\u001b[1;36m20\u001b[0m \n",
"\u001b[2;36m \u001b[0m loss \u001b[1;36m0.4501953125\u001b[0m \n",
"steps: 100%|████████████████████████████████| 1240/1240 [12:11<00:00, 1.75it/s]\u001b[2;36m[07/27/23 15:00:39]\u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: \n",
"\u001b[2;36m \u001b[0m \u001b[35m/home/pai/lib/python3.9/site-packages/coati/trainer\u001b[0m\n",
"\u001b[2;36m \u001b[0m \u001b[35m/\u001b[0m\u001b[95msft.py\u001b[0m:\u001b[1;36m98\u001b[0m _eval \n",
"\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0m\u001b[34mINFO \u001b[0m colossalai - colossalai - INFO: Eval Epoch \u001b[1;36m19\u001b[0m/\u001b[1;36m20\u001b[0m \n",
"\u001b[2;36m \u001b[0m loss \u001b[1;36m0.4501953125\u001b[0m \n",
"steps: 100%|████████████████████████████████| 1240/1240 [14:36<00:00, 1.41it/s]\n"
]
}
],
"source": [
"! pwd\n",
"! sh ColossalAI/applications/Chat/examples/train_sft.sh"
]
},
{
"cell_type": "markdown",
"id": "96366c97-720e-49c3-9727-9be9e2552c28",
"metadata": {},
"source": [
"## 试玩模型"
]
},
{
"cell_type": "markdown",
"id": "4d062161-e56f-45d5-9cd0-487476832fa9",
"metadata": {},
"source": [
"模型训练完成后，下载我们提供的webUI demo，试玩微调完成的模型（注意模型地址替换为自己训练好的模型地址）。建议用英文进行提问。"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "bbd6d240-b903-4a0d-afbe-be07e64d5e52",
"metadata": {
"ExecutionIndicator": {
"show": true
},
"tags": []
},
"outputs": [],
"source": [
"import gradio as gr\n",
"import requests\n",
"import json\n",
"from transformers import AutoTokenizer, AutoModelForCausalLM\n",
"\n",
"#模型地址替换为自己训练好的模型地址\n",
"tokenizer = AutoTokenizer.from_pretrained(\"sft_llama2-7b\",trust_remote_code=True)\n",
"#模型地址替换为自己训练好的模型地址\n",
"model = AutoModelForCausalLM.from_pretrained(\"sft_llama2-7b\",trust_remote_code=True).eval().half().cuda()\n",
"\n",
"def inference(text):\n",
" from transformers import pipeline\n",
"\n",
" pipe = pipeline(\"text-generation\", model=model, tokenizer=tokenizer,device='cuda:0', max_new_tokens=400)\n",
" res=pipe(text)\n",
" return res[0]['generated_text'][len(text):]\n",
" \n",
"\n",
"demo = gr.Blocks()\n",
"with demo:\n",
" input_prompt = gr.Textbox(label=\"请输入需求\", value=\"请以软件工程师的身份，写一篇入职的发言稿。\", lines=6)\n",
" generated_txt = gr.Textbox(lines=6)\n",
" b1 = gr.Button(\"发送\")\n",
" b1.click(inference, inputs=[input_prompt], outputs=generated_txt) \n",
"\n",
"demo.launch(enable_queue=True, share=False)"
]
},
{
"cell_type": "markdown",
"id": "09a1875f-db2b-48ac-9582-e11f36417d8e",
"metadata": {},
"source": [
"## 模型上传至OSS并在线部署"
]
},
{
"cell_type": "markdown",
"id": "bd80d943-cb08-4546-9e5e-e517d6cd26f2",
"metadata": {},
"source": [
"如果希望将上述模型部署至PAI-EAS，您需要将前面训练完成的模型上传至OSS。\n",
"\n",
"下列OSS客户端参数配置请您根据自己的信息填写后再执行。"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "14d809e0-de6a-4230-9086-f6ebdff87ae2",
"metadata": {
"ExecutionIndicator": {
"show": false
},
"execution": {
"iopub.execute_input": "2023-07-21T03:47:15.243065Z",
"iopub.status.busy": "2023-07-21T03:47:15.242706Z",
"iopub.status.idle": "2023-07-21T03:47:16.220501Z",
"shell.execute_reply": "2023-07-21T03:47:16.219607Z",
"shell.execute_reply.started": "2023-07-21T03:47:15.243045Z"
},
"tags": []
},
"outputs": [],
"source": [
"# encoding=utf-8\n",
"import oss2\n",
"import os\n",
"\n",
"AK='yourAccessKeyId'\n",
"SK='yourAccessKeySecret'\n",
"endpoint = 'yourEndpoint'\n",
"dir='your model output dir'\n",
"auth = oss2.Auth(AK, SK)\n",
"bucket = oss2.Bucket(auth, endpoint, 'examplebucket')\n",
"for filename in os.listdir(dir):\n",
" current_file_path = dir+filename\n",
" file_path = '需要上传地址'\n",
" bucket.put_object_from_file(file_path, current_file_path)"
]
},
{
"cell_type": "markdown",
"id": "ef00f95b-cfef-445e-8fae-5ebc0de26eb3",
"metadata": {},
"source": [
"最后进行部署工作，步骤请参考[EAS部署最佳实践](https://help.aliyun.com/document_detail/2392575.html)。"
]
}
],
"metadata": {
"dsw_sample": {
"buildId": "648",
"pipeline": "pai-dsw-examples-master"
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.13"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
