安装,配置kafka，安装配置kafka

安装,配置kafka，安装配置kafka

1、安装jdk,kafka的使用要用到jdk 

node2:/root#which java
/usr/java/jdk1.8.0_20/bin/java
node2:/root#java -version
java version "1.8.0_20"
Java(TM) SE Runtime Environment (build 1.8.0_20-b26)
Java HotSpot(TM) 64-Bit Server VM (build 25.20-b23, mixed mode)
You have new mail in /var/spool/mail/root


2、安装zookeeper 


cd /usr/local/kafka 
wget https://archive.apache.org/dist/zookeeper/zookeeper-3.3.6/zookeeper-3.3.6.tar.gz 
tar -zxvf zookeeper-3.3.6.tar.gz 

export zookeeper_home=/usr/local/kafka/zookeeper-3.3.6

测试zookeeper 是否按照成功:


node2:/usr/local/kafka/zookeeper-3.3.6/bin#./zkServer.sh start
JMX enabled by default
Using config: /usr/local/kafka/zookeeper-3.3.6/bin/../conf/zoo.cfg
grep: /usr/local/kafka/zookeeper-3.3.6/bin/../conf/zoo.cfg: No such file or directory
Starting zookeeper ... STARTED
node2:/usr/local/kafka/zookeeper-3.3.6/bin#

说明启动成功

3、安装kafka 
cd /usr/local/kafka 
wget https://archive.apache.org/dist/kafka/0.8.0/kafka_2.8.0-0.8.0.tar.gz 
tar -zxvf kafka_2.8.0-0.8.0.tar.gz 

配置kafka,修改文件server.properties 
cd /usr/local/kafka/kafka_2.8.0-0.8.0/config


host.name=192.168.137.3
log.dirs=/usr/local/kafka/log/kafka


zookeeper.connect=localhost:2181



默认为2181端口

配置kafka下zookeeper 

mkdir /usr/local/kafka/zookeeper #创建zookeeper目录 
mkdir /usr/local/kafka/log/zookeeper #创建zookeeper日志目录 
cd /usr/local/kafka/kafka_2.8.0-0.8.0/config 
vim zookeeper.properties


dataDir=/usr/local/kafka/zookeeper
dataLogDir=/usr/local/kafka/log/zookeeper
# the port at which the clients will connect
clientPort=2181
# disable the per-ip limit on the number of connections since this is a non-production config
maxClientCnxns=100
tickTime=2000
initLimit=10


启动kafka:
cd /usr/local/kafka/kafka_2.8.0-0.8.0/bin

node2:/usr/local/kafka/kafka_2.8.0-0.8.0/bin#./zookeeper-server-start.sh /usr/local/kafka/kafka_2.8.0-0.8.0/config/zookeeper.properties &


测试kafka:

node2:/usr/local/kafka/kafka_2.8.0-0.8.0/bin#./kafka-create-topic.sh -partition 1 -replica 1 -zookeeper localhost:2181 -topic test
[2017-09-28 20:52:35,942] INFO Accepted socket connection from /127.0.0.1:35639 (org.apache.zookeeper.server.NIOServerCnxn)
[2017-09-28 20:52:35,966] INFO Client attempting to establish new session at /127.0.0.1:35639 (org.apache.zookeeper.server.NIOServerCnxn)
[2017-09-28 20:52:35,969] INFO Established session 0x15ec87bb0b60001 with negotiated timeout 30000 for client /127.0.0.1:35639 (org.apache.zookeeper.server.NIOServerCnxn)
[2017-09-28 20:52:36,128] INFO Got user-level KeeperException when processing sessionid:0x15ec87bb0b60001 type:create cxid:0x2 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/brokers/topics Error:KeeperErrorCode = NoNode for /brokers/topics (org.apache.zookeeper.server.PrepRequestProcessor)
creation succeeded!
[2017-09-28 20:52:36,191] INFO Processed session termination for sessionid: 0x15ec87bb0b60001 (org.apache.zookeeper.server.PrepRequestProcessor)
[2017-09-28 20:52:36,201] INFO Closed socket connection for client /127.0.0.1:35639 which had sessionid 0x15ec87bb0b60001 (org.apache.zookeeper.server.NIOServerCnxn)
[2017-09-28 20:52:36,281] INFO Got user-level KeeperException when processing sessionid:0x15ec87bb0b60000 type:create cxid:0x26 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/brokers/topics/test/partitions/0 Error:KeeperErrorCode = NoNode for /brokers/topics/test/partitions/0 (org.apache.zookeeper.server.PrepRequestProcessor)
[2017-09-28 20:52:36,283] INFO Got user-level KeeperException when processing sessionid:0x15ec87bb0b60000 type:create cxid:0x27 zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:/brokers/topics/test/partitions Error:KeeperErrorCode = NoNode for /brokers/topics/test/partitions (org.apache.zookeeper.server.PrepRequestProcessor)
[2017-09-28 20:52:36,372] INFO [Replica Manager on Broker 0]: Handling LeaderAndIsr request Name:LeaderAndIsrRequest;Version:0;Controller:0;ControllerEpoch:1;CorrelationId:6;ClientId:id_0-host_192.168.137.3-port_9092;PartitionState:(test,0) -> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:0);Leaders:id:0,host:192.168.137.3,port:9092 (kafka.server.ReplicaManager)
[2017-09-28 20:52:36,396] INFO [ReplicaFetcherManager on broker 0] Removing fetcher for partition [test,0] (kafka.server.ReplicaFetcherManager)
[2017-09-28 20:52:36,442] INFO [Kafka Log on Broker 0], Completed load of log test-0 with log end offset 0 (kafka.log.Log)
[2017-09-28 20:52:36,446] INFO [Log Manager on Broker 0] Created log for partition [test,0] in /usr/local/kafka/log/kafka. (kafka.log.LogManager)
[2017-09-28 20:52:36,452] WARN No highwatermark file is found. Returning 0 as the highwatermark for partition [test,0] (kafka.server.HighwaterMarkCheckpoint)
[2017-09-28 20:52:36,469] INFO [Replica Manager on Broker 0]: Handled leader and isr request Name:LeaderAndIsrRequest;Version:0;Controller:0;ControllerEpoch:1;CorrelationId:6;ClientId:id_0-host_192.168.137.3-port_9092;PartitionState:(test,0) -> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:0);Leaders:id:0,host:192.168.137.3,port:9092 (kafka.server.ReplicaManager)

检查主题是否创建成功 
./kafka-list-topic.sh --zookeeper localhost:2181



node2:/usr/local/kafka/kafka_2.8.0-0.8.0/bin#./kafka-list-topic.sh  -zookeeper localhost:2181
[2017-09-28 20:53:09,275] INFO Accepted socket connection from /127.0.0.1:35640 (org.apache.zookeeper.server.NIOServerCnxn)
[2017-09-28 20:53:09,286] INFO Client attempting to establish new session at /127.0.0.1:35640 (org.apache.zookeeper.server.NIOServerCnxn)
[2017-09-28 20:53:09,289] INFO Established session 0x15ec87bb0b60002 with negotiated timeout 30000 for client /127.0.0.1:35640 (org.apache.zookeeper.server.NIOServerCnxn)
topic: test	partition: 0	leader: 0	replicas: 0	isr: 0
[2017-09-28 20:53:09,813] INFO Processed session termination for sessionid: 0x15ec87bb0b60002 (org.apache.zookeeper.server.PrepRequestProcessor)
[2017-09-28 20:53:09,815] INFO Closed socket connection for client /127.0.0.1:35640 which had sessionid 0x15ec87bb0b60002 (org.apache.zookeeper.server.NIOServerCnxn)
node2:/usr/local/kafka/kafka_2.8.0-0.8.0/bin#


（2）启动producer 
./kafka-console-producer.sh -broker-list 192.168.137.3:9092 -topic test 
（192.168.18.229是本地服务器IP地址，读者根据情况填写



查看kafka消息:
##查看topic分布情况kafka-list-topic.sh
bin/kafka-list-topic.sh --zookeeper 192.168.197.170:2181,192.168.197.171:2181 （列出所有topic的分区情况）
bin/kafka-list-topic.sh --zookeeper  localhost:2181 --topic test （查看test的分区情况）


kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group **  --topic ***  --zookeeper *:2181,*:2181,*:2181/kafka

kafka-list-topic.sh --zookeeper 192.168.137.3:2181

查看所有topic列表：

node2:/usr/local/kafka/kafka_2.8.0-0.8.0/bin#./kafka-list-topic.sh --zookeeper localhost:2181 
topic: test	partition: 0	leader: 0	replicas: 0	isr: 0
node2:/usr/local/kafka/kafka_2.8.0-0.8.0/bin#

查看指定topic信息
./kafka-list-topic.sh  --zookeeper localhost:2181 --describe --topic test


 (1) 启动kafka:
node2:/root/sbin#cat startkafka.sh 
cd /usr/local/kafka/kafka_2.8.0-0.8.0/bin
./zookeeper-server-start.sh /usr/local/kafka/kafka_2.8.0-0.8.0/config/zookeeper.properties &
sleep 3

（2）启动producer 
./kafka-console-producer.sh -broker-list 192.168.137.3:9092 -topic test 
（192.168.18.229是本地服务器IP地址，读者根据情况填写

（3）启动consumer 
./kafka-console-consumer.sh --zookeeper localhost:2181 --topic test

node2:/usr/local/kafka/kafka_2.8.0-0.8.0/bin#./kafka-console-consumer.sh --zookeeper localhost:2181 --topic test
ddddddddddddd
ffffffffffffff