大模型推理服务如何统计token

背景

最近在做一个大模型推理服务,需要统计每个请求的token数量以便计费

简单示例

非stream

请求示例

curl -H "Authorization: Bearer yourapikey"  https://yourdomain/v1/chat/completions -H "Content-Type: application/json" -d '{    "model": "","messages":[{"role":"user","content":"如何学习java"}],"stream":true,"stream_options":{"include_usage":true,"continuous_usage_stats":true}}'

响应示例

image.png

stream

请求示例

curl -H "Authorization: Bearer yourapikey"  https://yourdomain/v1/chat/completions -H "Content-Type: application/json" -d '{    "model": "","messages":[{"role":"user","content":"如何学习java"}],"stream":true,"stream_options":{"include_usage":true,"continuous_usage_stats":true}}'

响应示例

image.png