Skip to content

[SPARK-52428] Add comprehensive CI workflow with unit and integration tests#40

Open
rafafrdz wants to merge 1 commit intoapache:masterfrom
rafafrdz:add-ci-workflow
Open

[SPARK-52428] Add comprehensive CI workflow with unit and integration tests#40
rafafrdz wants to merge 1 commit intoapache:masterfrom
rafafrdz:add-ci-workflow

Conversation

@rafafrdz
Copy link
Copy Markdown

Summary

Add a new ci.yml workflow that improves on the existing build.yml with:

Jobs

Job What it does Needs Spark?
lint cargo fmt --check + cargo clippy -D warnings (default + all features) No
unit-tests Build + test across feature matrix (4 combinations) No
docs cargo doc with -D warnings No
integration-tests Full test suite against Docker Spark Connect server Yes

Improvements over existing build.yml

  • Feature matrix testing — tests default, polars, datafusion, and combined features separately
  • Smart unit test detection — distinguishes transport errors (expected without Spark) from real failures
  • Server readiness check — waits up to 60s for Spark Connect with nc probe before running integration tests
  • Failure diagnostics — dumps docker compose logs on integration test failure
  • Updated actions — uses arduino/setup-protoc@v3 (not v2), dtolnay/rust-toolchain@stable
  • Triggers on dev branch — supports the fork's integration branch

Test plan

  • Workflow YAML is valid
  • Covers all feature combinations
  • Integration tests wait for server readiness
  • Failure diagnostics included

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant