Kafka — The Legend

When we deal with real-time data analytics, it is always a herculian task to setup a data pipeline and handle without any data loss. In those situation, we have Kafka to rescue.

In this tutorial, you are going to see how quickly, you can setup Kakfa up and running. — Single node multibroker

Step 1 — Download and install Java (Click here)

Step 2 — Download Kafka Binaries (Click here)

Step 3 — Untar it

tar -xvzf kafka_2.11-1.0.0.tgz

Step 4 — Start ZooKeeper

bin/zookeeper-server-start.sh config/zookeeper.properties

Step 5 — Let’s setup 3 brokers to do that, you need to change the server.properties, so copy file and change the filename accordingly.

cp config/server.properties config/server.1.properties
cp config/server.properties config/server.2.properties
cp config/server.properties config/server.3.properties

server.1.properties

broker.id=1
listeners=PLAINTEXT://:9093
log.dirs=/tmp/logs1

server.2.properties

broker.id=2
listeners=PLAINTEXT://:9094
log.dirs=/tmp/logs2

server.3.properties

broker.id=3
listeners=PLAINTEXT://:9095
log.dirs=/tmp/logs3

Step 6 — Create those log directory

mkdir /tmp/logs1
mkdir /tmp/logs2
mkdir /tmp/logs3

Step 7 — Now, all set and ready to run the broker instances, using below commands

bin/kafka-server-start.sh config/server.1.properties
bin/kafka-server-start.sh config/server.2.properties
bin/kafka-server-start.sh config/server.3.properties

Step 8 — To test, all working fine, let’s create a Topic

bin/kafka-topics.sh --create --topic mytopicname --zookeeper localhost:2181 --partitions 3 --replication-factor 2

Step 9 — Open another terminal and let’s produce a message using a producer

bin/kafka-console-producer.sh --broker-list localhost:9093,localhost:9094,localhost:9095 --topic mytopicname

Step 10 — Open another terminal and consume this message by using below command

bin/kafka-console-consumer.sh --bootstrap-server localhost:9093 --topic mytopicname --from-beginning