ELK Stack · Part 1 of 5

Kafka on Linux

Complete installation & configuration guide for Apache Kafka 3.6.1 with Zookeeper on Ubuntu/Debian — including broker setup, topic management, and log verification.

Complete Logging Infrastructure

Understand the full log flow architecture before you begin. Each component plays a distinct role in the pipeline.

Log event flowing through the pipeline Stage-to-stage stream

Figure 1 — Live log flow. Events stream from the Node.js producer through Kafka's buffer, get filtered by Logstash, land in Elasticsearch, and surface in Kibana.

Architecture Benefits

Reliability: Kafka acts as a buffer, preventing log loss during bulk operations
Scalability: Each component can be scaled independently based on load
Data Quality: Logstash filters ensure clean, structured data in Elasticsearch
Visibility: Kibana provides real-time insights into your application logs

01 Install Java

Kafka requires Java 8+. OpenJDK 17 is recommended for production.

bash

sudo apt update
sudo apt install openjdk-17-jdk -y   # Ubuntu/Debian

Verify Installation

bash

java -version

02 Download Kafka

bash

cd /opt
sudo wget https://archive.apache.org/dist/kafka/3.6.1/kafka_2.13-3.6.1.tgz

03 Extract Kafka

bash

sudo tar -xvf kafka_2.13-3.6.1.tgz
sudo mv kafka_2.13-3.6.1 kafka

04 Configure Zookeeper

bash

sudo nano /opt/kafka/config/zookeeper.properties

Default Zookeeper Configuration

properties

# Directory where the snapshot is stored
dataDir=/tmp/zookeeper
# Port at which clients connect
clientPort=2181
# Disable per-ip connection limit (non-production)
maxClientCnxns=0
# Disable admin server to avoid port conflicts
admin.enableServer=false
# admin.serverPort=8080

05 Start Zookeeper

bash

cd /opt/kafka
bin/zookeeper-server-start.sh config/zookeeper.properties

06 Configure Kafka Broker

The main configuration file is server.properties — key settings below.

bash

sudo nano /opt/kafka/config/server.properties

Key Configuration Settings

properties

# Broker ID
broker.id=0

# Log storage
log.dirs=/var/lib/kafka-logs

# Network listeners
listeners=PLAINTEXT://0.0.0.0:9092
# Advertised address for clients (applications, Logstash)
advertised.listeners=PLAINTEXT://<BROKER_IP>:9092

# Zookeeper connection
zookeeper.connect=localhost:2181

# Topic retention
log.retention.hours=168
log.segment.bytes=1073741824

# Optional: enable auto topic creation
auto.create.topics.enable=true

Key Configuration Points

listeners — Kafka port for producers (apps) and consumers (Logstash)
advertised.listeners — The address clients use to reach Kafka from outside
log.dirs — Where Kafka stores messages on disk
zookeeper.connect — Link Kafka to Zookeeper for cluster coordination

07 Start Kafka Broker

bash

cd /opt/kafka
bin/kafka-server-start.sh config/server.properties

Kafka broker is now running on port 9092

08 Check Broker Configuration

Verify broker config at runtime via Kafka scripts:

bash

/opt/kafka/bin/kafka-configs.sh \
  --bootstrap-server 192.168.20.208:9092 \
  --entity-type brokers \
  --describe

Shows broker-specific configs like num.network.threads, log.dirs, etc.

09 Create a Topic

Simple Topic Creation

bash

bin/kafka-topics.sh --create --topic test --bootstrap-server localhost:9092

Create Topic with Partitions & Replication

bash

/opt/kafka/bin/kafka-topics.sh --create \
  --topic application_logs \
  --bootstrap-server localhost:9092 \
  --partitions 3 \
  --replication-factor 1

Configuration Parameters

partitions — For scaling and parallel processing (sensitive)
replication-factor — For fault tolerance — set ≥ 1 (sensitive)
topic — Storage name used to differentiate log types

10 List Topics & Partition Info

bash

/opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server 192.168.20.208:9092 \
  --describe

11 Verify Logs in Kafka

bash

/opt/kafka/bin/kafka-console-consumer.sh \
  --bootstrap-server 192.168.20.208:9092 \
  --topic application_logs \
  --from-beginning

What This Command Does

Reads messages from the topic application_logs
Shows messages sent before (--from-beginning) or after the command runs

Confirms Kafka is running
Confirms the topic exists
Confirms producers are sending logs
Confirms consumers can read logs