使用Debezium+Kafka 为SqlServer 表建立S
2021-08-27 本文已影响0人
Codmowa
本文不涉及 zookeeper kafaka debezium的原理, 也不使用 docker 等.
requirement
- 一个已经部署好的或者正在运行的SQLServer 实例
- 一个Windows10 主机可以运行Wsl2 , 或者一台Linux主机(或者虚拟机, docker container 都可以)必须能连接 SQL Server
安装java , 博主用的是 WSL2 debian buster(10)
// 默认安装的是 openjdk-11 可以用 java --version 检查.
sudo apt install default-jdk
安装 kafaka
mkdir /kafaka
cd /kafaka
wget https://archive.apache.org/dist/kafka/2.8.0/kafka_2.13-2.8.0.tgz
tar -xzf ./kafka_2.13-2.8.0.tgz
安装 Debezium (这里用1.6 正式版.
mkdir /kafaka/kafka_2.12-2.8.0/connect
cd /kafaka/kafka_2.12-2.8.0/connect
wget https://repo1.maven.org/maven2/io/debezium/debezium-connector-sqlserver/1.6.1.Final/debezium-connector-sqlserver-1.6.1.Final-plugin.tar.gz
tar -xzf debezium-connector-sqlserver-1.6.1.Final-plugin.tar.gz
修改 kfaka 配置
code /kafka/kafka_2.12-2.8.0/config/connect-distributed.properties
# 修改 kfaka connect-distributed plugin.path 配置, 把它指向放 Debezium的文件夹
plugin.path=/kafka/kafka_2.12-2.8.0/connect
# 修改一下 listener,
code /kafka/kafka_2.12-2.8.0/config/server.properties
listeners=PLAINTEXT://:9092
# 绑定到 wsl2 局域网ip
advertised.listeners=PLAINTEXT://192.168.200.199:9092
依次启动 zookeeper kafka kafka connect
# 这里可以多开几个terminal 用来查看日志
cd /kafka/kafka_2.12-2.8.0/bin
./zookeeper-server-start.sh ../config/zookeeper.properties
./kafka-server-start.sh ../config/server.properties
./connect-distributed.sh ../config/connect-distributed.properties
sql server 开启 cdc
-- 数据库开启cdc
USE AdventureWorks2012;
GO
EXECUTE sys.sp_cdc_enable_db;
GO
-- 开启追踪
EXECUTE sys.sp_cdc_enable_table
@source_schema = N'dbo'
, @source_name = N'Customer'
, @role_name = N'public';
GO
向 kafka connect 添加 connector
curl --location --request POST 'http://localhost:8083/connectors/' \
--header 'Content-Type: application/json' \
--data-raw '{
"name": "sqlserver-yourdb-connector",
"config": {
"connector.class": "io.debezium.connector.sqlserver.SqlServerConnector",
"database.hostname": "localhost",
"database.port": "1433",
"database.user": "sa",
"database.password": "password",
"database.dbname": "yourdb",
"table.include.list":"dbo.Customer",
"database.server.name": "",
"database.history.kafka.bootstrap.servers": "localhost:9092",
"database.history.kafka.topic": "dbhistory.yourdb"
}
}'
完成
这个时候用kafka tools 或者其他的工具, 连接到zookeeper 应该就能看到数据了
image.png
但是如果你是从一个很大的表上面建立了connector 那么你可能要等待kafka 建立快照几个小时.
