KAFKA - EVENT PROCESSING SYSTEM
- No need to wait for response
- Fire and Forget
- Real time processing (Streams)
- High throughput & Low latency
Topics
- Particular stream of data
- Can be identified by name
e.g. Tables in a database
- Support all type of messages
- The sequence of message is called, data stream
- You cannot query topics, instead use kafka producers to send data and kafka consumers to read the data
- Kafka topics are immutable, Once data is written to a partition, it cannot be changed
- Data is kept for a limited time (default is one week - configurable)
Partitions
- Topics are split into partitions
- Messages within each partitions are ordered
Offset
- Each message within a partition gets an incremental id, called offset
Producers
- Write data to topics
- Producers know to which partition to write
Kafka Connect
-Getting data in and out of kafka
Step-by-Step to Start Kafka
- Step 1: Start ZooKeeper
- This will keep running in the terminal. In a new terminal window
- Step 2: Start Kafka Server (Broker)
- Step 3: Create a Kafka Topic
- Step 4: Start Producer
- Type messages here to send to Kafka.
- Step 5: Start Consumer (in a new terminal)
- You will see the messages you type in the producer appear here.
