Kafka - Topics, Partitions & Offset

KAFKA - EVENT PROCESSING SYSTEM

• No need to wait for response
• Fire and Forget
• Real time processing (Streams)
• High throughput & Low latency

 

Topics 

    - Particular stream of data

    - Can be identified by name

        e.g. Tables in a database

    - Support all type of messages

    - The sequence of message is called, data stream

    - You cannot query topics, instead use kafka producers to send data and kafka consumers to read the data

    - Kafka topics are immutable, Once data is written to a partition, it cannot be changed

    - Data is kept for a limited time (default is one week - configurable)

Partitions

    - Topics are split into partitions

    - Messages within each partitions are ordered

Offset

    - Each message within a partition gets an incremental id, called offset

Producers

    - Write data to topics

    - Producers know to which partition to write

Kafka Connect

    -Getting data in and out of kafka

Step-by-Step to Start Kafka

• Step 1: Start ZooKeeper
• This will keep running in the terminal. In a new terminal window
• Step 2: Start Kafka Server (Broker)
• Step 3: Create a Kafka Topic
• Step 4: Start Producer
• Type messages here to send to Kafka.
• Step 5: Start Consumer (in a new terminal)
• You will see the messages you type in the producer appear here.

Architecture