Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

Apache Spark Connect Client for Rust

Sparglim✨ makes PySpark App Configurable and Deploy Spark Connect Server Easier!

Apache Spark Connect for Rust

velib-v2: An ETL pipeline that employs batch and streaming jobs using Spark, Kafka, Airflow, and other tools, all orchestrated with Docker Compose.

Extension to {sparklyr} that allows you to interact with Spark & Databricks Connect

A reverse proxy server which allows secure connectivity to a Spark Connect server

An Apache Spark Connect Client for C++.

Docker Compose environment for big data research and machine learning development

Collection of articles, using the Literate Programming style, about Data Engineering and Software Tooling in general

Interactive metadata visualizer for Apache Iceberg. Trace snapshots, manifests, and file lineages through a real-time, graph-based UI.

The Internals of Spark Connect

Spark Connect client library in Rust

Real-time insights on bike availability using the JCDECAUX API. Data flows through Kafka, processed by Spark, and visualized in a Streamlit app. Deployed on Kubernetes.

User-land broadcast-variable workaround for PySpark on Spark Connect / Databricks Serverless

TypeScript client for Apache Spark Connect