Mini Kafka — Web Dashboard

An educational, from-scratch simulation of the core ideas behind Apache Kafka, built with Spring Boot (backend) and React + Vite + Tailwind (frontend).

No Apache Kafka dependency is used anywhere in this project. Topics, partitions, offsets, producers, consumers and consumer groups are all implemented using plain Java collections, ExecutorService, and thread-safe data structures.

---

1. Project Overview

Mini Kafka simulates four independent "microservices" — OrderService, PaymentService, NotificationService, and UserService — that never call each other directly. Instead, they publish and subscribe to topics through a broker that behaves like a simplified Kafka. A React dashboard visualizes everything happening inside the broker in real time over WebSocket.

2. What is Apache Kafka?

Apache Kafka is a distributed event-streaming platform. Producers publish records to named topics; each topic is split into partitions for parallelism and ordering; consumers read records from partitions, tracking their position with an offset; and consumer groups let multiple consumers coordinate reading a topic. Kafka is durable, replicated, and horizontally scalable across a cluster of brokers.

3. What is Mini Kafka?

Mini Kafka reproduces the same concepts — topics, partitions, offsets, producers, consumers, consumer groups — but as a single in-process, in-memory Java application. There's no replication, no disk persistence, and no cluster; it's meant purely to make the concepts tangible and observable through a live dashboard.

4. Architecture

OrderService
      |
      v
   Producer
      |
      v
   MINI KAFKA (BrokerService)
      |
   +--+----------------+
   |                   |
   v                   v
Payment Consumer   Notification Consumer
   |                   |
   v                   v
PaymentService    NotificationService

• OrderService publishes to the orders topic.
• PaymentService and NotificationService each independently consume orders — neither knows the other exists.
• PaymentService publishes to payments; NotificationService also consumes payments.
• UserService publishes to users.

All communication happens only through BrokerService.

5. Producer

A producer is anything that calls ProducerService.publish(topic, content), which delegates to BrokerService.publishMessage(). This:

1. Auto-creates the topic if it doesn't exist (matching real Kafka's default).
2. Picks a partition via round-robin.
3. Appends the message and assigns it the next offset.
4. Broadcasts the new message and updated stats over WebSocket.

6. Broker

BrokerService is the single source of truth. It owns every Topic, every ConsumerGroup, and every Consumer, using ConcurrentHashMap for thread safety since multiple producer/consumer threads (including the simulation's ScheduledExecutorService tasks) hit it concurrently.

7. Topic

A Topic is a named stream backed by one or more Partitions. Partition selection for new messages is round-robin.

8. Partition

A Partition is an ordered, append-only log (CopyOnWriteArrayList) with a monotonically increasing offset counter (AtomicLong). Once written, a message is never modified.

9. Consumer

Each Consumer tracks its own offset per partition in a ConcurrentHashMap<Integer, Long>. This is what allows two different consumers — say, payment-service-consumer and notification-service-orders-consumer — to both read the orders topic from the beginning without stepping on each other's progress.

10. Consumer Groups

A ConsumerGroup is a named collection of consumers (e.g. payment-group, notification-group). Groups exist to organize consumers logically; in this simplified model, offset tracking still lives on the individual Consumer, which is enough to demonstrate that groups have independent progress from one another.

11. Offset

An offset is the position of a message within a partition, starting at 0 and incrementing by 1 for every message appended to that partition. Consumers advance their own offset as they read; producers never touch offsets directly.

12. WebSocket

The backend uses STOMP over WebSocket (spring-boot-starter-websocket). Whenever BrokerService.publishMessage() runs, it broadcasts:

• The new message to /topic/messages
• Updated dashboard stats to /topic/stats

The frontend connects via SockJS + @stomp/stompjs (see src/hooks/useWebSocket.js) and updates the UI without polling for the live stream (a 5s polling fallback still refreshes topic/consumer lists for consistency).

13. Backend Setup

Requirements: Java 17+ and Maven 3.8+.

cd backend
mvn clean install

14. Frontend Setup

Requirements: Node.js 18+.

cd frontend
npm install

15. How to Run

Terminal 1 — backend (starts on port 8080):

cd backend
mvn spring-boot:run

Terminal 2 — frontend (starts on port 5173, proxies /api and /ws to the backend — see vite.config.js):

cd frontend
npm run dev

Open http://localhost:5173 in your browser. Click ▶ Start Simulation on the dashboard to see live traffic, or use the "Publish Message" form to send messages manually.

Note: this project was generated in a sandboxed environment without internet access, so the Maven and npm dependency downloads above have not been executed or verified end-to-end. Please run mvn clean install and npm install yourself and fix any dependency-version issues that surface (e.g. bumping spring-boot-starter-parent or npm package versions) before relying on this in production.

16. API Documentation

Topics

Method	Path	Description
POST	/api/topics	Create a topic. Body: { "name": "orders", "partitions": 3 }
GET	/api/topics	List all topics with summary stats
GET	/api/topics/{topicName}	Get topic detail including all partitions
DELETE	/api/topics/{topicName}	Delete a topic
GET	/api/topics/{topicName}/messages	All messages in a topic, across partitions

Producer

Method	Path	Description
POST	/api/messages	Publish a message. Body: { "topic": "orders", "message": "Order #101 placed" }

Consumers

Method	Path	Description
POST	/api/consumers/subscribe	Subscribe. Body: { "consumerId": "...", "groupId": "...", "topic": "orders" }
GET	/api/consumers/{consumerId}/messages	Fetch new messages for a consumer (advances its offset)
GET	/api/consumers	List all consumers
GET	/api/consumers/{consumerId}	Get one consumer's summary

Dashboard

Method	Path	Description
GET	/api/dashboard/stats	Live counts: topics, messages, producers, consumers, msgs/sec

Simulation

Method	Path	Description
POST	/api/simulation/start	Start the demo services
POST	/api/simulation/stop	Stop the demo services
GET	/api/simulation/status	{ "running": true/false }

WebSocket

Destination	Description
/ws (SockJS)	Connect endpoint
/topic/messages	New message broadcasts
/topic/stats	Updated dashboard stats

17. Screenshots

Add screenshots of the Dashboard, Topics, and Consumers pages here once you've run the app locally.

18. Future Improvements

• Database persistence (the backend is already layered so a repository implementation could swap in behind BrokerService without touching controllers)
• Authentication / authorization
• Multiple brokers + replication
• Message retention / compaction policies
• Consumer groups with multiple consumers splitting partitions
• Docker / Kubernetes deployment
• Metrics and observability (e.g. Micrometer + Prometheus)

---

Folder Structure

mini-kafka/
├── backend/
│   ├── pom.xml
│   └── src/main/
│       ├── java/com/minikafka/
│       │   ├── MiniKafkaApplication.java
│       │   ├── controller/
│       │   │   ├── TopicController.java
│       │   │   ├── ProducerController.java
│       │   │   ├── ConsumerController.java
│       │   │   ├── DashboardController.java
│       │   │   └── SimulationController.java
│       │   ├── service/
│       │   │   ├── BrokerService.java
│       │   │   ├── TopicService.java
│       │   │   ├── ProducerService.java
│       │   │   ├── ConsumerService.java
│       │   │   └── SimulationService.java
│       │   ├── model/
│       │   │   ├── Message.java
│       │   │   ├── Topic.java
│       │   │   ├── Partition.java
│       │   │   ├── Consumer.java
│       │   │   └── ConsumerGroup.java
│       │   ├── dto/
│       │   │   ├── PublishMessageRequest.java
│       │   │   ├── CreateTopicRequest.java
│       │   │   ├── SubscribeRequest.java
│       │   │   └── MessageResponse.java
│       │   ├── websocket/
│       │   │   ├── WebSocketConfig.java
│       │   │   └── MessageBroadcaster.java
│       │   └── exception/
│       │       ├── GlobalExceptionHandler.java
│       │       ├── TopicNotFoundException.java
│       │       ├── TopicAlreadyExistsException.java
│       │       ├── InvalidMessageException.java
│       │       └── ConsumerNotFoundException.java
│       └── resources/application.properties
└── frontend/
    ├── package.json, vite.config.js, tailwind.config.js, postcss.config.js
    ├── index.html
    └── src/
        ├── main.jsx, App.jsx, index.css
        ├── api/client.js
        ├── hooks/useWebSocket.js
        ├── components/
        │   ├── Sidebar.jsx, Header.jsx, StatsCards.jsx
        │   ├── TopicList.jsx, MessageStream.jsx, PublishMessage.jsx
        │   ├── ConsumerPanel.jsx, PartitionView.jsx
        └── pages/
            ├── Dashboard.jsx, Topics.jsx, Consumers.jsx

Message Flow, End to End

1. User fills in the Publish Message form and clicks Publish (or the simulation's OrderService fires on its own schedule).
2. Frontend calls POST /api/messages.
3. ProducerController → ProducerService → BrokerService.publishMessage().
4. The broker auto-creates the topic if needed, picks a partition via round-robin, and appends the message with the next sequential offset.
5. The broker broadcasts the new message over /topic/messages and fresh stats over /topic/stats.
6. Every connected dashboard receives both broadcasts instantly via STOMP/WebSocket and updates the Live Message Stream and Overview cards with no page refresh.
7. Independently, PaymentService and NotificationService (running as scheduled tasks) poll their own subscriptions via ConsumerService.consume(), which fetches only the messages after each consumer's own last-read offset per partition, then advances that offset — completely decoupled from any other consumer's progress.