Skip to content

[SPARK-53317] Add SparkSession::create_dataframe_from_polars()#38

Open
rafafrdz wants to merge 1 commit intoapache:masterfrom
rafafrdz:spark-53317-polars-interop
Open

[SPARK-53317] Add SparkSession::create_dataframe_from_polars()#38
rafafrdz wants to merge 1 commit intoapache:masterfrom
rafafrdz:spark-53317-polars-interop

Conversation

@rafafrdz
Copy link
Copy Markdown

Summary

  • Add SparkSession::create_dataframe_from_polars() behind #[cfg(feature = "polars")]
  • Converts Polars DataFrame to Arrow RecordBatch using polars-arrow arrow_rs interop
  • Delegates to existing create_dataframe(&RecordBatch)
  • No new dependencies needed — uses existing polars-arrow with arrow_rs feature

Test plan

  • cargo build --features polars passes
  • cargo fmt -- --check passes
  • Integration test creating Spark DF from Polars DF

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant