Kafka — The Legend
When we deal with real-time data analytics, it is always a herculian task to setup a data pipeline and handle without any data loss. In those situation, we have Kafka to rescue.
In this tutorial, you are going to see how quickly, you can setup Kakfa up and running. — Single node multibroker
Step 1 — Download and install Java (Click here)
Step 2 — Download Kafka Binaries (Click here)
Step 3 — Untar it
tar -xvzf kafka_2.11-1.0.0.tgz
Step 4 — Start ZooKeeper
bin/zookeeper-server-start.sh config/zookeeper.properties
Step 5 — Let’s setup 3 brokers to do that, you need to change the server.properties, so copy file and change the filename accordingly.
cp config/server.properties config/server.1.properties
cp config/server.properties config/server.2.properties
cp config/server.properties config/server.3.properties
server.1.properties
broker.id=1
listeners=PLAINTEXT://:9093
log.dirs=/tmp/logs1
server.2.properties
broker.id=2
listeners=PLAINTEXT://:9094
log.dirs=/tmp/logs2
server.3.properties
broker.id=3
listeners=PLAINTEXT://:9095
log.dirs=/tmp/logs3
Step 6 — Create those log directory
mkdir /tmp/logs1
mkdir /tmp/logs2
mkdir /tmp/logs3
Step 7 — Now, all set and ready to run the broker instances, using below commands
bin/kafka-server-start.sh config/server.1.properties
bin/kafka-server-start.sh config/server.2.properties
bin/kafka-server-start.sh config/server.3.properties
Step 8 — To test, all working fine, let’s create a Topic
bin/kafka-topics.sh --create --topic mytopicname --zookeeper localhost:2181 --partitions 3 --replication-factor 2
Step 9 — Open another terminal and let’s produce a message using a producer
bin/kafka-console-producer.sh --broker-list localhost:9093,localhost:9094,localhost:9095 --topic mytopicname
Step 10 — Open another terminal and consume this message by using below command
bin/kafka-console-consumer.sh --bootstrap-server localhost:9093 --topic mytopicname --from-beginning