后台开发实战

Zookeeper和Kafka环境搭建总结

2019-03-19  本文已影响0人  蛮三刀酱
题图

前言

由于项目需要涉及到zookeeper和Kafka的使用，快速做了一篇笔记，方便小伙伴们搭建环境。

zookeeper

官方定义

What is ZooKeeper?

ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications. Each time they are implemented there is a lot of work that goes into fixing the bugs and race conditions that are inevitable. Because of the difficulty of implementing these kinds of services, applications initially usually skimp on them, which make them brittle in the presence of change and difficult to manage. Even when done correctly, different implementations of these services lead to management complexity when the applications are deployed.

ZooKeeper是一个分布式的，开放源码的分布式应用程序协调服务，是Google的Chubby一个开源的实现，是Hadoop和Hbase的重要组件。它是一个为分布式应用提供一致性服务的软件，提供的功能包括：配置维护、域名服务、分布式同步、组服务等。
ZooKeeper的目标就是封装好复杂易出错的关键服务，将简单易用的接口和性能高效、功能稳定的系统提供给用户。

zookeeper常用端口号：2181

环境搭建

  1. 下载zookeeper：

    下载地址：http://mirrors.hust.edu.cn/apache/zookeeper/

  1. 在zookeeper-3.4.10目录下，新建文件夹，并命名，例如data。(路径为：D:\zookeeper-3.4.10\conf\data)

  2. 复制“zoo_sample.cfg”副本à并将副本重命名为“zoo.cfg，在任意文本编辑器（eg：记事本）中打开zoo.cfg，找到并编辑dataDir=D:\\zookeeper-3.4.10\\data

  3. windows的话，在cmd下运行：D:\zookeeper-3.4.13\bin>zkServer.cmd

在这里插入图片描述 
D:\zookeeper-3.4.13\bin>zkServer.cmd

D:\zookeeper-3.4.13\bin>call "C:\Program Files\Java\jdk1.8.0_181"\bin\java "-Dzookeeper.log.dir=D:\zookeeper-3.4.13\bin\.." "-Dzookeeper.root.logger=INFO,CONSOLE" -cp "D:\zookeeper-3.4.13\bin\..\build\classes;D:\zookeeper-3.4.13\bin\..\build\lib\*;D:\zookeeper-3.4.13\bin\..\*;D:\zookeeper-3.4.13\bin\..\lib\*;D:\zookeeper-3.4.13\bin\..\conf" org.apache.zookeeper.server.quorum.QuorumPeerMain "D:\zookeeper-3.4.13\bin\..\conf\zoo.cfg"
2019-03-18 22:07:46,267 [myid:] - INFO  [main:QuorumPeerConfig@136] - Reading configuration from: D:\zookeeper-3.4.13\bin\..\conf\zoo.cfg
2019-03-18 22:07:46,276 [myid:] - INFO  [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2019-03-18 22:07:46,276 [myid:] - INFO  [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2019-03-18 22:07:46,277 [myid:] - INFO  [main:DatadirCleanupManager@101] - Purge task is not scheduled.
2019-03-18 22:07:46,280 [myid:] - WARN  [main:QuorumPeerMain@116] - Either no config or no quorum defined in config, running  in standalone mode
2019-03-18 22:07:46,397 [myid:] - INFO  [main:QuorumPeerConfig@136] - Reading configuration from: D:\zookeeper-3.4.13\bin\..\conf\zoo.cfg
2019-03-18 22:07:46,398 [myid:] - INFO  [main:ZooKeeperServerMain@98] - Starting server
2019-03-18 22:07:46,422 [myid:] - INFO  [main:Environment@100] - Server environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 04:05 GMT
2019-03-18 22:07:46,422 [myid:] - INFO  [main:Environment@100] - Server environment:host.name=DESKTOP-D5MA1P9
2019-03-18 22:07:46,422 [myid:] - INFO  [main:Environment@100] - Server environment:java.version=1.8.0_181
2019-03-18 22:07:46,423 [myid:] - INFO  [main:Environment@100] - Server environment:java.vendor=Oracle Corporation
2019-03-18 22:07:46,423 [myid:] - INFO  [main:Environment@100] - Server environment:java.home=C:\Program Files\Java\jdk1.8.0_181\jre
2019-03-18 22:07:46,423 [myid:] - INFO  [main:Environment@100] - Server environment:java.class.path=D:\zookeeper-3.4.13\bin\..\build\classes;D:\zookeeper-3.4.13\bin\..\build\lib\*;D:\zookeeper-3.4.13\bin\..\zookeeper-3.4.13.jar;D:\zookeeper-3.4.13\bin\..\lib\audience-annotations-0.5.0.jar;D:\zookeeper-3.4.13\bin\..\lib\jline-0.9.94.jar;D:\zookeeper-3.4.13\bin\..\lib\log4j-1.2.17.jar;D:\zookeeper-3.4.13\bin\..\lib\netty-3.10.6.Final.jar;D:\zookeeper-3.4.13\bin\..\lib\slf4j-api-1.7.25.jar;D:\zookeeper-3.4.13\bin\..\lib\slf4j-log4j12-1.7.25.jar;D:\zookeeper-3.4.13\bin\..\conf
2019-03-18 22:07:46,424 [myid:] - INFO  [main:Environment@100] - Server environment:java.library.path=C:\Program Files\Java\jdk1.8.0_181\bin;C:\WINDOWS\Sun\Java\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\nodejs\;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\Git\cmd;C:\Program Files\Java\jre1.8.0_191\bin;C:\Program Files\Java\jdk1.8.0_181\bin;C:\Users\yzd\AppData\Local\Microsoft\WindowsApps;C:\Users\yzd\AppData\Roaming\npm;.
2019-03-18 22:07:46,425 [myid:] - INFO  [main:Environment@100] - Server environment:java.io.tmpdir=C:\Users\yzd\AppData\Local\Temp\
2019-03-18 22:07:46,425 [myid:] - INFO  [main:Environment@100] - Server environment:java.compiler=<NA>
2019-03-18 22:07:46,427 [myid:] - INFO  [main:Environment@100] - Server environment:os.name=Windows 10
2019-03-18 22:07:46,427 [myid:] - INFO  [main:Environment@100] - Server environment:os.arch=amd64
2019-03-18 22:07:46,427 [myid:] - INFO  [main:Environment@100] - Server environment:os.version=10.0
2019-03-18 22:07:46,428 [myid:] - INFO  [main:Environment@100] - Server environment:user.name=yzd
2019-03-18 22:07:46,428 [myid:] - INFO  [main:Environment@100] - Server environment:user.home=C:\Users\yzd
2019-03-18 22:07:46,428 [myid:] - INFO  [main:Environment@100] - Server environment:user.dir=D:\zookeeper-3.4.13\bin
2019-03-18 22:07:46,438 [myid:] - INFO  [main:ZooKeeperServer@836] - tickTime set to 2000
2019-03-18 22:07:46,438 [myid:] - INFO  [main:ZooKeeperServer@845] - minSessionTimeout set to -1
2019-03-18 22:07:46,439 [myid:] - INFO  [main:ZooKeeperServer@854] - maxSessionTimeout set to -1
2019-03-18 22:07:46,647 [myid:] - INFO  [main:ServerCnxnFactory@117] - Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory
2019-03-18 22:07:46,651 [myid:] - INFO  [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:2181

搭建成功

zookeeper和eureka比较

ZooKeeper、Eureka对比：

https://www.cnblogs.com/jieqing/p/8394001.html

CAP定理（C-数据一致性；A-服务可用性；P-服务对网络分区故障的容错性，这三个特性在任何分布式系统中不能同时满足，最多同时满足两个）

ZooKeeper基于CP，不保证高可用，如果zookeeper正在选主，或者Zookeeper集群中半数以上机器不可用，那么将无法获得数据。Eureka基于AP，能保证高可用，即使所有机器都挂了，也能拿到本地缓存的数据。作为注册中心，其实配置是不经常变动的，只有发版和机器出故障时会变。对于不经常变动的配置来说，CP是不合适的，而AP在遇到问题时可以用牺牲一致性来保证可用性，既返回旧数据，缓存数据。

所以理论上Eureka是更适合作注册中心。而现实环境中大部分项目可能会使用ZooKeeper，那是因为集群不够大，并且基本不会遇到用做注册中心的机器一半以上都挂了的情况。所以实际上也没什么大问题。

kfaka

在这里插入图片描述

Kafka® is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies.

Kafka是由Apache软件基金会开发的一个开源流处理平台，由Scala和Java编写。Kafka是一种高吞吐量的分布式发布订阅消息系统，它可以处理消费者规模的网站中的所有动作流数据。 这种动作（网页浏览，搜索和其他用户的行动）是在现代网络上的许多社会功能的一个关键因素。 这些数据通常是由于吞吐量的要求而通过处理日志和日志聚合来解决。 对于像Hadoop一样的日志数据和离线分析系统，但又要求实时处理的限制，这是一个可行的解决方案。Kafka的目的是通过Hadoop的并行加载机制来统一线上和离线的消息处理，也是为了通过集群来提供实时的消息。

kafka包内自带整合好的zookeeper，所以如果要搭建kafka可以省去单独下载zookeeper的步骤。想起了欲练武功必先自宫的笑话，哈哈哈。

环境搭建

  1. 下载地址：http://kafka.apache.org/downloads.html
  2. 解压缩，建立一个空文件夹 logs. eg: D:\kafka_2.11-1.0.0\logs
  3. 进入config目录，编辑 server.properties文件(eg: 用“写字板”打开)。找到并编辑log.dirs= D:\kafka_2.11-1.0.0\logs
  4. 请确保在启动Kafka服务器前，Zookeeper实例已经准备好并开始运行。
  5. 在此处打开Powershell窗口，运行：.\bin\windows\kafka-server-start.bat .\config\server.properties
在这里插入图片描述 
PS D:\kafka_2.12-2.1.1> .\bin\windows\kafka-server-start.bat .\config\server.properties
[2019-03-18 22:15:34,866] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2019-03-18 22:15:35,503] INFO starting (kafka.server.KafkaServer)
[2019-03-18 22:15:35,505] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2019-03-18 22:15:35,530] INFO [ZooKeeperClient] Initializing a new session to localhost:2181. (kafka.zookeeper.ZooKeeperClient)
[2019-03-18 22:15:35,536] INFO Client environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,536] INFO Client environment:host.name=DESKTOP-D5MA1P9 (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,536] INFO Client environment:java.version=1.8.0_181 (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,537] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,537] INFO Client environment:java.home=C:\Program Files\Java\jdk1.8.0_181\jre (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,537] INFO Client environment:java.class.path=D:\kafka_2.12-2.1.1\libs\activation-1.1.1.jar;D:\kafka_2.12-2.1.1\libs\aopalliance-repackaged-2.5.0-b42.jar;D:\kafka_2.12-2.1.1\libs\argparse4j-0.7.0.jar;D:\kafka_2.12-2.1.1\libs\audience-annotations-0.5.0.jar;D:\kafka_2.12-2.1.1\libs\commons-lang3-3.8.1.jar;D:\kafka_2.12-2.1.1\libs\connect-api-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\connect-basic-auth-extension-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\connect-file-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\connect-json-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\connect-runtime-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\connect-transforms-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\guava-20.0.jar;D:\kafka_2.12-2.1.1\libs\hk2-api-2.5.0-b42.jar;D:\kafka_2.12-2.1.1\libs\hk2-locator-2.5.0-b42.jar;D:\kafka_2.12-2.1.1\libs\hk2-utils-2.5.0-b42.jar;D:\kafka_2.12-2.1.1\libs\jackson-annotations-2.9.8.jar;D:\kafka_2.12-2.1.1\libs\jackson-core-2.9.8.jar;D:\kafka_2.12-2.1.1\libs\jackson-databind-2.9.8.jar;D:\kafka_2.12-2.1.1\libs\jackson-jaxrs-base-2.9.8.jar;D:\kafka_2.12-2.1.1\libs\jackson-jaxrs-json-provider-2.9.8.jar;D:\kafka_2.12-2.1.1\libs\jackson-module-jaxb-annotations-2.9.8.jar;D:\kafka_2.12-2.1.1\libs\javassist-3.22.0-CR2.jar;D:\kafka_2.12-2.1.1\libs\javax.annotation-api-1.2.jar;D:\kafka_2.12-2.1.1\libs\javax.inject-1.jar;D:\kafka_2.12-2.1.1\libs\javax.inject-2.5.0-b42.jar;D:\kafka_2.12-2.1.1\libs\javax.servlet-api-3.1.0.jar;D:\kafka_2.12-2.1.1\libs\javax.ws.rs-api-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\javax.ws.rs-api-2.1.jar;D:\kafka_2.12-2.1.1\libs\jaxb-api-2.3.0.jar;D:\kafka_2.12-2.1.1\libs\jersey-client-2.27.jar;D:\kafka_2.12-2.1.1\libs\jersey-common-2.27.jar;D:\kafka_2.12-2.1.1\libs\jersey-container-servlet-2.27.jar;D:\kafka_2.12-2.1.1\libs\jersey-container-servlet-core-2.27.jar;D:\kafka_2.12-2.1.1\libs\jersey-hk2-2.27.jar;D:\kafka_2.12-2.1.1\libs\jersey-media-jaxb-2.27.jar;D:\kafka_2.12-2.1.1\libs\jersey-server-2.27.jar;D:\kafka_2.12-2.1.1\libs\jetty-client-9.4.12.v20180830.jar;D:\kafka_2.12-2.1.1\libs\jetty-continuation-9.4.12.v20180830.jar;D:\kafka_2.12-2.1.1\libs\jetty-http-9.4.12.v20180830.jar;D:\kafka_2.12-2.1.1\libs\jetty-io-9.4.12.v20180830.jar;D:\kafka_2.12-2.1.1\libs\jetty-security-9.4.12.v20180830.jar;D:\kafka_2.12-2.1.1\libs\jetty-server-9.4.12.v20180830.jar;D:\kafka_2.12-2.1.1\libs\jetty-servlet-9.4.12.v20180830.jar;D:\kafka_2.12-2.1.1\libs\jetty-servlets-9.4.12.v20180830.jar;D:\kafka_2.12-2.1.1\libs\jetty-util-9.4.12.v20180830.jar;D:\kafka_2.12-2.1.1\libs\jopt-simple-5.0.4.jar;D:\kafka_2.12-2.1.1\libs\kafka-clients-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\kafka-log4j-appender-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\kafka-streams-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\kafka-streams-examples-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\kafka-streams-scala_2.12-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\kafka-streams-test-utils-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\kafka-tools-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\kafka_2.12-2.1.1-javadoc.jar;D:\kafka_2.12-2.1.1\libs\kafka_2.12-2.1.1-javadoc.jar.asc;D:\kafka_2.12-2.1.1\libs\kafka_2.12-2.1.1-scaladoc.jar;D:\kafka_2.12-2.1.1\libs\kafka_2.12-2.1.1-scaladoc.jar.asc;D:\kafka_2.12-2.1.1\libs\kafka_2.12-2.1.1-sources.jar;D:\kafka_2.12-2.1.1\libs\kafka_2.12-2.1.1-sources.jar.asc;D:\kafka_2.12-2.1.1\libs\kafka_2.12-2.1.1-test-sources.jar;D:\kafka_2.12-2.1.1\libs\kafka_2.12-2.1.1-test-sources.jar.asc;D:\kafka_2.12-2.1.1\libs\kafka_2.12-2.1.1-test.jar;D:\kafka_2.12-2.1.1\libs\kafka_2.12-2.1.1-test.jar.asc;D:\kafka_2.12-2.1.1\libs\kafka_2.12-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\kafka_2.12-2.1.1.jar.asc;D:\kafka_2.12-2.1.1\libs\log4j-1.2.17.jar;D:\kafka_2.12-2.1.1\libs\lz4-java-1.5.0.jar;D:\kafka_2.12-2.1.1\libs\maven-artifact-3.6.0.jar;D:\kafka_2.12-2.1.1\libs\metrics-core-2.2.0.jar;D:\kafka_2.12-2.1.1\libs\osgi-resource-locator-1.0.1.jar;D:\kafka_2.12-2.1.1\libs\plexus-utils-3.1.0.jar;D:\kafka_2.12-2.1.1\libs\reflections-0.9.11.jar;D:\kafka_2.12-2.1.1\libs\rocksdbjni-5.14.2.jar;D:\kafka_2.12-2.1.1\libs\scala-library-2.12.7.jar;D:\kafka_2.12-2.1.1\libs\scala-logging_2.12-3.9.0.jar;D:\kafka_2.12-2.1.1\libs\scala-reflect-2.12.7.jar;D:\kafka_2.12-2.1.1\libs\slf4j-api-1.7.25.jar;D:\kafka_2.12-2.1.1\libs\slf4j-log4j12-1.7.25.jar;D:\kafka_2.12-2.1.1\libs\snappy-java-1.1.7.2.jar;D:\kafka_2.12-2.1.1\libs\validation-api-1.1.0.Final.jar;D:\kafka_2.12-2.1.1\libs\zkclient-0.11.jar;D:\kafka_2.12-2.1.1\libs\zookeeper-3.4.13.jar;D:\kafka_2.12-2.1.1\libs\zstd-jni-1.3.7-1.jar (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,538] INFO Client environment:java.library.path=C:\Program Files\Java\jdk1.8.0_181\bin;C:\WINDOWS\Sun\Java\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\nodejs\;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\Git\cmd;C:\Program Files\Java\jre1.8.0_191\bin;C:\Program Files\Java\jdk1.8.0_181\bin;C:\Users\yzd\AppData\Local\Microsoft\WindowsApps;C:\Users\yzd\AppData\Roaming\npm;. (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,539] INFO Client environment:java.io.tmpdir=C:\Users\yzd\AppData\Local\Temp\ (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,542] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,542] INFO Client environment:os.name=Windows 10 (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,543] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,543] INFO Client environment:os.version=10.0 (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,547] INFO Client environment:user.name=yzd (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,547] INFO Client environment:user.home=C:\Users\yzd (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,547] INFO Client environment:user.dir=D:\kafka_2.12-2.1.1 (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,549] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=6000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@3e2e18f2 (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,570] INFO [ZooKeeperClient] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
[2019-03-18 22:15:35,570] INFO Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2019-03-18 22:15:35,574] INFO Socket connection established to localhost/0:0:0:0:0:0:0:1:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2019-03-18 22:15:35,598] INFO Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x1000ae3e3690000, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2019-03-18 22:15:35,602] INFO [ZooKeeperClient] Connected. (kafka.zookeeper.ZooKeeperClient)
[2019-03-18 22:15:35,974] INFO Cluster ID = brvhJ2K3R4qZnGJ6CQhghA (kafka.server.KafkaServer)
[2019-03-18 22:15:35,980] WARN No meta.properties file under dir D:\kafka_2.12-2.1.1\logs\meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2019-03-18 22:15:36,049] INFO KafkaConfig values:
        advertised.host.name = null
        ...(省略若干)
        zookeeper.sync.time.ms = 2000
 (kafka.server.KafkaConfig)
[2019-03-18 22:15:36,092] INFO [ThrottledChannelReaper-Fetch]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2019-03-18 22:15:36,092] INFO [ThrottledChannelReaper-Produce]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2019-03-18 22:15:36,094] INFO [ThrottledChannelReaper-Request]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2019-03-18 22:15:36,132] INFO Loading logs. (kafka.log.LogManager)
[2019-03-18 22:15:36,143] INFO Logs loading complete in 11 ms. (kafka.log.LogManager)
[2019-03-18 22:15:36,161] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
[2019-03-18 22:15:36,166] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2019-03-18 22:15:36,486] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
[2019-03-18 22:15:36,534] INFO [SocketServer brokerId=0] Started 1 acceptor threads (kafka.network.SocketServer)
[2019-03-18 22:15:36,562] INFO [ExpirationReaper-0-Produce]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-03-18 22:15:36,563] INFO [ExpirationReaper-0-DeleteRecords]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-03-18 22:15:36,563] INFO [ExpirationReaper-0-Fetch]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-03-18 22:15:36,581] INFO [LogDirFailureHandler]: Starting (kafka.server.ReplicaManager$LogDirFailureHandler)
[2019-03-18 22:15:36,607] INFO Creating /brokers/ids/0 (is it secure? false) (kafka.zk.KafkaZkClient)
[2019-03-18 22:15:36,612] INFO Result of znode creation at /brokers/ids/0 is: OK (kafka.zk.KafkaZkClient)
[2019-03-18 22:15:36,613] INFO Registered broker 0 at path /brokers/ids/0 with addresses: ArrayBuffer(EndPoint(DESKTOP-D5MA1P9,9092,ListenerName(PLAINTEXT),PLAINTEXT)) (kafka.zk.KafkaZkClient)
[2019-03-18 22:15:36,615] WARN No meta.properties file under dir D:\kafka_2.12-2.1.1\logs\meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2019-03-18 22:15:36,682] INFO [ExpirationReaper-0-topic]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-03-18 22:15:36,684] INFO [ExpirationReaper-0-Heartbeat]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-03-18 22:15:36,685] INFO [ExpirationReaper-0-Rebalance]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-03-18 22:15:36,689] INFO Successfully created /controller_epoch with initial epoch 0 (kafka.zk.KafkaZkClient)
[2019-03-18 22:15:36,714] INFO [GroupCoordinator 0]: Starting up. (kafka.coordinator.group.GroupCoordinator)
[2019-03-18 22:15:36,716] INFO [GroupCoordinator 0]: Startup complete. (kafka.coordinator.group.GroupCoordinator)
[2019-03-18 22:15:36,731] INFO [GroupMetadataManager brokerId=0] Removed 0 expired offsets in 15 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2019-03-18 22:15:36,748] INFO [ProducerId Manager 0]: Acquired new producerId block (brokerId:0,blockStartProducerId:0,blockEndProducerId:999) by writing to Zk with path version 1 (kafka.coordinator.transaction.ProducerIdManager)
[2019-03-18 22:15:36,780] INFO [TransactionCoordinator id=0] Starting up. (kafka.coordinator.transaction.TransactionCoordinator)
[2019-03-18 22:15:36,782] INFO [TransactionCoordinator id=0] Startup complete. (kafka.coordinator.transaction.TransactionCoordinator)
[2019-03-18 22:15:36,791] INFO [Transaction Marker Channel Manager 0]: Starting (kafka.coordinator.transaction.TransactionMarkerChannelManager)
[2019-03-18 22:15:36,875] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
[2019-03-18 22:15:36,892] INFO [SocketServer brokerId=0] Started processors for 1 acceptors (kafka.network.SocketServer)
[2019-03-18 22:15:36,901] INFO Kafka version : 2.1.1 (org.apache.kafka.common.utils.AppInfoParser)
[2019-03-18 22:15:36,919] INFO Kafka commitId : 21234bee31165527 (org.apache.kafka.common.utils.AppInfoParser)
[2019-03-18 22:15:36,922] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)

搭建成功。

测试kafka生产者消费者

  1. 开启zookeeper和kafka服务

  2. 创建topic：

> bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

看一眼刚才创建的主题

> bin/kafka-topics.sh --list --zookeeper localhost:2181
test
  1. 发消息：接收者
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
test
在这里插入图片描述
  1. 收消息：消费者
> bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
This is a message
This is another message
在这里插入图片描述
上一篇下一篇

猜你喜欢

热点阅读