goreplay(gor) golang 流量拷贝工具试用
1. 项目地址
https://github.com/buger/goreplay
2. 安装
wget https://github.com/buger/goreplay/releases/download/v0.16.1/gor_0.16.1_x64.tar.gz
tar xvf gor_0.16.1_x64.tar.gz
cp goreplay /usr/bin
3. 简单使用
// 测试环境使用的是nginx 有连个机器一台进行反向代理，具体的安装配置比较简单
a.进行流量拷贝并输出到标准输出
goreplay --input-raw:80 --output-stdout
response message
GET/HTTP/1.1
Host:XXXXXXXXX
User-Agent:Mozilla/5.0(WindowsNT6.1;Win64;x64)AppleWebKit/537.36(KHTML,likeGecko)Chrome/62.0.3202.75Safari/537.36
Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: zh-CN,zh;q=0.9,en;q=0.8
Cache-Control: max-age=0
If-Modified-Since: Sat, 04 Nov 2017 13:52:14 GMT
If-None-Match: W/"59fdc60e-232"
Proxy-Connection: keep-alive
Upgrade-Insecure-Requests: 1
X-Forwarded-For: XXXXXXXXX
X-Lantern-Version: 4.4.0
b. 流量拷贝其他环境
goreplay --input-raw :80 --output-http="http://xxxxx:port"
c. 流量拷贝存储为本地文件（也可以使用s3存储，开源的有minio）
copy:
goreplay --input-raw :80 --output-file=requests.gor
replay:
goreplay --input-file requests.gor --output-http="http://XXXXXXXX:port"
4. 可用的插件
a.input
--input-raw-usedtocaptureHTTPtraffic,youshouldspecifyIPaddressor
interfaceandapplicationport.MoreaboutCapturingandreplayingtraffic.
--input-file-acceptsfilewhichpreviouslywasrecordedusing--output-file.
MoreaboutSavingandReplayingfromfile
--input-tcp-usedbyGoraggregationinstanceifyoudecidedforwardtraffic
frommultipleforwarderGorinstancestoit.
b.output
--output-http-replayHTTPtraffictogivenendpoint,acceptsbaseurl
--output-file-recordsincomingtraffictothefile.MoreaboutSavingandReplayingfromfile
--output-tcp-forwardincomingdatatoanotherGorinstance,usedinconjunctionwith--input-tcp.ReadmoreaboutAggregator-forwardersetup.
--output-stdout-usedfordebugging,outputsalldatatostdout.
5. 扩展
Rate limiting
a. Limiting replay using absolute number
# staging.server will not get more than ten requests per second
gor--input-tcp:28020--output-http"http://staging.com|10"
b. Limiting listener using percent age based limiter
# replay server will not get more than 10% of requests
# useful for high-load environments
gor--input-raw:80--output-tcp"replay.local:28020|10%"
c.Consistent limiting based on Heade ror URL paramvalue
# Limit based on header value
gor--input-raw:80--output-tcp"replay.local:28020|10%"--http-header-limiter"X-API-KEY: 10%"
# Limit based on header value
gor--input-raw:80--output-tcp"replay.local:28020|10%"--http-param-limiter"api_key: 10%"
Request filtering
a.Allow url regexp
# only forward requests being sent to the /api endpoint
gor--input-raw:8080--output-httpstaging.com--http-allow-url/api
b.Disallow url regexp
# only forward requests NOT being sent to the /api... endpoint
gor--input-raw:8080--output-httpstaging.com--http-disallow-url/api
c.Filter based on regexp of header
# only forward requests with an api version of 1.0x
gor--input-raw:8080--output-httpstaging.com--http-allow-headerapi-version:^1\.0\d
# only forward requests NOT containing User-Agent header value "Replayed by Gor"
gor--input-raw:8080--output-httpstaging.com--http-disallow-header"User-Agent: Replayed by Gor"
d.Filter based on HTTP method
gor--input-raw:80--output-http"http://staging.server"\
--http-allow-methodGET\
--http-allow-methodOPTIONS
6. 类似工具
nginx mirror
openresty 通过lua
tcpcopy
7. 参考资料
// wiki
https://github.com/buger/goreplay/wiki
https://github.com/buger/goreplay
// kafka
https://github.com/buger/goreplay/wiki/Streaming-from-and-to-Apache-Kafka
// Distributed-configuration
https://github.com/buger/goreplay/wiki/Distributed-configuration
