Streaming with Java

A project for comparing blocking versus reactive Java when processing streaming ADS-B (Automatic Dependent Surveillance-Broadcast) data from Kafka. The applications consume messages from a Kafka topic, enrich them with aircraft registration data from MongoDB, and write the results to MongoDB.

Project Structure

Directory	Description
streamer/	Go application that streams historical Stratux ADS-B messages to Kafka at a configurable rate
blocking_java/	Blocking Java implementation using Kafka consumer and MongoDB sync driver
reactive_java/	Reactive Java implementation using Reactor Kafka and MongoDB reactive streams driver
run_benchmark.sh	Script to run a coordinated benchmark of both Java applications

Prerequisites

• Java 21+ – for the Java applications
• Maven – to build the Java applications
• MongoDB – running locally with:
  • stratux_raw collection (source data for the streamer)
  • registered_aircraft collection (lookup: n_number, manufacturer, model)
  • Sample ADS-B and aircraft registration data is available in streamer/collection_data/ – see streamer/README.md for import instructions and details.
• Docker – for Kafka (started automatically by the streamer)

Quick Start

1. Streamer: Build from source (cd streamer && go build -o stratux-simulator .) or use a pre-built executable from streamer/distributions/ (see streamer/README.md).

2. Sample data (optional): Import sample ADS-B and aircraft registration data from streamer/collection_data/ – see streamer/README.md for details.

3. Ensure MongoDB is running with the required collections populated.

4. Run the benchmark script (builds Java apps, starts streamer, runs blocking then reactive for 60 seconds each):

   ./run_benchmark.sh 60

5. Or run components manually:

   • Start the streamer (after building or using a pre-built binary): cd streamer && ./stratux-simulator --rate 1000
   • Run blocking app: java -jar blocking_java/target/blocking-java-adsb-processor-1.0.0.jar
   • Run reactive app: java -jar reactive_java/target/reactive-java-adsb-processor-1.0.0.jar

Processing Pipeline

Both Java applications implement the same pipeline:

1. Consume JSON messages from the Kafka stratux topic
2. Discard messages without a reg (registration) field
3. Look up registered_aircraft by n_number (matching reg, with optional N prefix stripped)
4. Add manufacturer and model to the message when a match is found
5. Write enriched messages to a configurable MongoDB collection

Configuration

Each application loads configuration from a .env file in its directory. Key variables:

Variable	Description	Default
MONGO_URI	MongoDB connection string	mongodb://localhost:27017
MONGO_DATABASE	MongoDB database	Stratux
KAFKA_BROKER	Kafka bootstrap servers	localhost:9092
KAFKA_TOPIC	Kafka topic name	stratux
OUTPUT_COLLECTION	MongoDB collection for enriched output	enriched_adsb

Statistics

On shutdown (Ctrl+C or SIGTERM), both applications print:

• Messages received from Kafka
• Messages written to MongoDB
• Messages rejected (missing reg)

License

This project is licensed under the Apache License, Version 2.0. See LICENSE for the full text.