Add native Polars DataFrame support

[stewjb]

Closes #98

Summary

• Export: export_data() now accepts polars.DataFrame and polars.Series directly, mapping Polars dtypes to SBDF types without any Pandas intermediary. Supported types: Boolean, Int8/16/32, Int64, Float32/64, Utf8/String, Date, Datetime, Duration, Time, Binary, Decimal, Categorical.
• Import: import_data() gains an output_format parameter (default "pandas" for backwards compatibility). When output_format="polars", a polars.DataFrame is built directly from the raw numpy arrays — no Pandas DataFrame is created at any point.
• Dependency: Polars is added as an optional dependency (spotfire[polars]), following the same pattern as spotfire[geo] and spotfire[plot].

Performance benefit

The previous workaround required polars_df.to_pandas() before export, which doubles peak memory usage and adds 2–5 seconds of conversion time at 10M rows. The native path eliminates this entirely for export.

Spotfire data function context

When running inside a Spotfire data function, SBDF import and export happen automatically via data_function.py — users never call import_data or export_data directly. This has two implications:

Export (output variables): Full benefit. A user can build a polars.DataFrame in their script and return it as an output variable — export_data() handles it natively with no conversion.

Import (input variables): No benefit from import_data(output_format="polars"). Input data is always loaded by the framework via sbdf.import_data(self._file) (no output_format argument), so input variables always arrive in the script as pd.DataFrame. Users who want Polars for processing would still need to call pl.from_pandas(input_df) themselves. Fixing this properly would require changes to data_function.py and a mechanism for users to declare their preference — out of scope for this PR.

In short: the output_format parameter on import_data is primarily useful outside the Spotfire data function context (e.g. standalone scripts using the spotfire package directly). Inside a data function, only the export side benefits.

Test plan

• test_write_polars_basic — export DataFrame with common types, re-import as Pandas and verify data
• test_write_polars_nulls — null values are preserved through the roundtrip
• test_write_polars_series — Polars Series export works
• test_import_as_polars — import with output_format="polars" returns a native polars.DataFrame
• test_polars_roundtrip — full Polars → SBDF → Polars roundtrip
• All 72 existing tests continue to pass (1 pre-existing skip unrelated to this change)
• pylint 10.00/10, mypy clean, cython-lint clean

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds native Polars support to Spotfire’s SBDF import/export layer to avoid Pandas conversions (improving memory usage and performance for large datasets), and wires it up as an optional extra.

Changes:

• Add polars as an optional dependency (spotfire[polars]) and enable it in dev/test setups.
• Extend sbdf.export_data() to accept polars.DataFrame / polars.Series directly, with dtype→SBDF mapping.
• Extend sbdf.import_data() with output_format to optionally construct a native polars.DataFrame without creating a Pandas DataFrame.

Reviewed changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
spotfire/sbdf.pyx	Implements Polars import/export paths and dtype mappings; adds output_format to import_data().
spotfire/sbdf.pyi	Updates type stub for import_data() to include output_format.
spotfire/test/test_sbdf.py	Adds Polars-focused unit tests for export/import/roundtrip + warnings.
pyproject.toml	Adds polars extra and includes it in dev extra.
test_requirements_default.txt	Installs Polars for test runs.
README.md	Documents spotfire[polars] and the new import/export behavior.
.gitignore	Ignores .venv, uv.lock, and .claude.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

In the Polars export path, invalids are derived from series.is_null(), which does not mark floating-point NaN values as invalid. In the existing Pandas path pd.isnull() treats NaN as missing, so exporting a Polars float column containing NaN will write NaNs as real values instead of SBDF invalids (behavior mismatch vs Pandas and likely incorrect for Spotfire missing-values semantics). Consider treating NaN as invalid for Float32/Float64 columns (e.g., combine is_null() with is_nan() when applicable).

[stewjb]

added series.is_null() and series.is_nan() for floats to handle like pandas does.

[Copilot]

_export_polars_series_to_numpy converts to an object ndarray when na_value is None. For Polars Datetime / Duration series, to_numpy() already produces datetime64 / timedelta64 arrays that the existing SBDF exporters can handle, so forcing dtype=object will box scalars and create an unnecessary copy (hurting the performance goal of this PR). Consider special-casing datetime/timespan to keep the native NumPy dtype (ideally normalized to the SBDF-supported resolution) instead of casting to object.

Suggested change

	else:
	else:
	# For Datetime/Duration, keep native NumPy datetime64/timedelta64 dtypes instead of boxing to object.
	if dtype_name in ("Datetime", "Duration"):
	return series.to_numpy(allow_copy=True)

[stewjb]

datetime and duration go to numpy early now.

[stewjb]

@vrane-tibco @bbassett-tibco @mpanke-tibco thanks for considering this PR. Let me know if you all have thoughts.

[vrane-tibco]

@stewjb Appreciate the work here — the dtype mapping is thorough and the export mechanics are solid. But I have two concerns, one architectural and one about the performance framing.

On metadata -

This package isn't just a data serializer - it's specifically the bridge that carries Spotfire's interpretation of that data back and forth. The three things Spotfire cares about beyond raw values are spotfire_table_metadata, spotfire_column_metadata, and spotfire_type. All three are attached to Pandas objects and the entire surface API (copy_metadata, get_spotfire_types, set_spotfire_types, set_geocoding_table, all exported from init.py) is built on top of those Pandas extension points.
Polars DataFrames don't support any of this. So right now:

• On import: In sbdf.pyx _import_build_polars_dataframe (line 875) only receives column_names and importer_contexts. The table_metadata and column_metadata that were correctly parsed from the SBDF file a few lines above simply never make it into the returned DataFrame. Silently dropped

• On export: In sbdf.pyx _export_obj_polars_dataframe hardcodes {} for both table and column metadata (lines 1248, 1251). The Pandas path reads these from the object; the Polars path has nowhere to read them from.

• In the public API: copy_metadata(), get_spotfire_types(), and set_spotfire_types() all raise TypeError on non-Pandas objects (lines 32–35, 68–69, 86–87). So a user who gets a pl.DataFrame back from import_data(output_format="polars") and immediately tries to use any of those three functions hits a confusing error.

On performance -

The export side is genuinely better than the to_pandas() workaround.

The import side though is a different story, Performance gain get for Numeric columns (Boolean, Integer, LongInteger, Float, Real): The Polars path does pl.Series(values=numpy_array) - Polars can reference the NumPy buffer directly for these types. This is the genuine zero-copy , but String or Datetime I doubt there will be performance gain.

Take strings as a concrete example. In sbdf.pyx _import_vt_string (line 534–536) runs a Python loop that creates a Python str object for every row and stores them in a NPY_OBJECT array — that's unavoidable because SBDF stores strings as C char pointers, not Arrow buffers. Then _import_build_polars_dataframe calls .tolist() on that object array (line 726) to produce a Python list, because Polars can't consume a NPY_OBJECT array directly. Then pl.Series(values=list) re-encodes all those strings into Polars' Arrow buffer. So at peak memory you have the NPY_OBJECT array, the Python list, and the Arrow buffer all alive simultaneously - three representations of the same data. The Pandas import path just wraps the NPY_OBJECT array in a Series header and stops there: two representations, same str objects shared by reference.

So for import: numeric columns genuinely benefit, strings and datetimes are actually worse than the Pandas path in both memory and time.

[stewjb]

@vrane-tibco this is good feedback. Thank you for taking the time to consider this. If you all feel the metadata is non-negotiable for this package, I understand that. We can fork off this package if we deem this important. The biggest benefit is not having doubling of peak memory when using polars, but we can increase our memory to get around this.

The metadata loss is a limitation of polars, which you've mentioned.

Polars has an outstanding issue to address this (pola-rs/polars#5117), but it's been open since 2022 which isn't promising.

My suggestion would be to log a warning to the user on the limitation of Polars on import/export. That would handle your point on dropping the metadata silently. I don't think there is a clean way to do that hand off natively into polars. The alternative I can see is to return a list with the dataframe and metadata unpacked. That seems clunky and confusing. A possible compromise is another function strictly for polars that returns the metadata unpacked.

Your comment on the confusing error makes sense and I can address it.

On performance, the export benefit is the main driver for me of this PR.

As of now, I don't know how the user would specify within spotfire that they want a polars dataframe returned instead of pandas. Personally, i just use df = spotfire_data_table within a data function, and I believe the only way to specify a polars output would be to hack into the internals of this package.

One proposal would be to drop the import functionality totally given you can't reach it as intended currently. I included it because it helps for testing and if polars adoption becomes more widespread in the future.

I think we can address the concerns on datetimes and strings. I definitely think we can drop it to to have only 2 copies of the same data, which is still a net improvement over having to use .to_pandas(), which is the current default and holds 2 copies of the whole df in that process.

[stewjb]

@vrane-tibco addressing your point on import copy performance — several optimizations have
landed since your review that change the picture for both strings and datetimes.

String import

Your analysis was correct at the time of review: the NPY_OBJECT array, the Python list,
and the Arrow buffer were all live simultaneously — three representations. Two changes
since then:

1. The NPY_OBJECT array is now explicitly freed (del values) before pl.Series() is
   called, so peak memory is two representations (list + Arrow), not three. This matches
   the Pandas path (NPY_OBJECT array + pd.Series, which share the same str objects).

2. .tolist() copies object references, not string bytes. The Python str objects
   created in _import_vt_string are the same objects in both the NumPy array and the
   list — no string data is duplicated at that step. So there are two copies of string
   content across the whole pipeline: once into Python str objects, once into the
   Arrow Utf8 buffer. The Pandas path also has two (Python str objects + the NPY_OBJECT
   array header), so the two paths are now equivalent in both peak memory and copy count.

Datetime import

Your concern was that datetimes would not see a performance gain. The current
implementation avoids Python object boxing entirely:

• _import_vts_numpy reads raw int64 ms directly from the SBDF C buffer into a NumPy
  int64 array — no Python datetime objects are created.
• The epoch offset is subtracted in-place on the NumPy array (no allocation).
• pl.Series(int64).cast(pl.Datetime('ms')) is a zero-copy metadata change — Datetime
  and Int64 share the same Arrow int64 backing store.

The Pandas path (_import_vt_datetime) runs a Python loop boxing each value into a
pd.Timestamp, which is significantly more expensive.

Full import picture

Type	Copies (Polars path)
Bool / Int / Float	1 (file → NumPy; Polars may share buffer)
String / Binary	2 (SBDF bytes → str objects; str objects → Arrow Utf8); peak 2 live
Datetime	1 (file → int64 NumPy; epoch subtract in-place; zero-copy cast to Datetime)
Date	1 (file → int32 days NumPy; zero-copy cast to Date)
Duration	1 (file → int64 NumPy; zero-copy cast to Duration)
Time	1 (file → int64 ns NumPy; zero-copy cast to Time)

String is now equivalent to the Pandas path. All temporal types are strictly better —
the Pandas path boxes every value into a Python object; the Polars path works entirely
at the int64 level and lets Polars handle the reinterpretation.

[stewjb]

This is getting large for one PR. Here are my thoughts on splitting it up.

PR A — Export

export_data() accepts pl.DataFrame and pl.Series directly. This is the part you
explicitly praised — no new public API surface, no metadata questions, just an overload
on an existing function.

Covers:

• dtype → SBDF type mapping for all Polars types
• Zero-copy temporal export (_export_vt_polars_*, _export_polars_setup_arrays)
• Edge cases: Null dtype, Categorical, UInt64 overflow, tz-aware Datetime, NaN as missing
• Metadata warning on export
• All test_write_polars_* tests
• pyproject.toml (spotfire[polars] extra, test requirements)
• README (spotfire[polars] installation and export_data() usage)

PR B — Import

import_data() gains output_format=OutputFormat.POLARS returning a native
pl.DataFrame. Keeping this separate means the import discussion doesn't block the
export merge.

Covers:

• OutputFormat enum and stub changes
• _import_build_polars_dataframe and all C-level import functions
• Zero-copy temporal import (datetime epoch shift, date int32, timespan, time ns)
• Null sentinel fix for pl.Time
• Metadata warning on import; descriptive TypeError on copy_metadata /
  get_spotfire_types / set_spotfire_types when passed a Polars object
• All import, roundtrip, and metadata error tests
• README (import_data() usage with OutputFormat.POLARS)

PR A could land quickly given it's the uncontested part, and PR B can absorb the metadata/import discussion on its own timeline.

[stewjb]

Benchmark Results

Measured on Python 3.14.3 / Polars 1.39.3 / Pandas 2.3.3 / NumPy 2.4.3, 7 reps (first excluded as warmup).
Time = mean of 7 reps. Δ RSS = peak increase in resident set size during the call (captures Arrow/Rust/C allocations).
"Old Polars workaround" = .to_pandas() before export / pl.from_pandas() after import.

Export time — 100,000 rows

Profile	pandas	polars (old: via pandas)	polars (this PR)	Speedup vs old
Numeric, no nulls	46 ms	54 ms	27 ms	2.0×
Numeric, ~10% nulls	25 ms	24 ms	16 ms	1.5×
String, no nulls	139 ms	139 ms	54 ms	2.6×
String, ~10% nulls	257 ms	209 ms	96 ms	2.2×
Temporal (datetime/date/duration/time), no nulls	1809 ms	1758 ms	35 ms	50×
Temporal (datetime/date/duration/time), ~10% nulls	1711 ms	1459 ms	54 ms	27×
Binary, no nulls	149 ms	199 ms	105 ms	1.9×
Binary, ~10% nulls	134 ms	172 ms	101 ms	1.7×

Export memory (Δ RSS) — 100,000 rows

Profile	pandas	polars (old: via pandas)	polars (this PR)
Numeric, no nulls	0 MB	0 MB	0 MB
Numeric, ~10% nulls	0 MB	0 MB	0 MB
String, no nulls	0.3 MB	0.9 MB	1.3 MB
String, ~10% nulls	0.1 MB	0.9 MB	0.9 MB
Temporal (datetime/date/duration/time), no nulls	4.0 MB	4.0 MB	0 MB
Temporal (datetime/date/duration/time), ~10% nulls	2.0 MB	2.0 MB	0.1 MB
Binary, no nulls	10.0 MB	12.2 MB	10.6 MB
Binary, ~10% nulls	8.4 MB	13.1 MB	10.4 MB

Import time — 100,000 rows

Profile	→ pandas	→ polars (old: via pandas)	→ polars (this PR)	Speedup vs old
Numeric, no nulls	18 ms	33 ms	7 ms	4.7×
Numeric, ~10% nulls	11 ms	14 ms	13 ms	1.1×
String, no nulls	75 ms	90 ms	100 ms	~even
String, ~10% nulls	65 ms	93 ms	70 ms	1.3×
Temporal (datetime/date/duration/time), no nulls	136 ms	247 ms	56 ms	4.4×
Temporal (datetime/date/duration/time), ~10% nulls	145 ms	245 ms	54 ms	4.5×
Binary, no nulls	91 ms	111 ms	84 ms	1.3×
Binary, ~10% nulls	76 ms	102 ms	80 ms	1.3×

Import memory (Δ RSS) — 100,000 rows

Profile	→ pandas	→ polars (old: via pandas)	→ polars (this PR)
Numeric, no nulls	0.2 MB	0 MB	0 MB
Numeric, ~10% nulls	0 MB	0 MB	0.4 MB
String, no nulls	0.8 MB	1.6 MB	3.0 MB
String, ~10% nulls	3.9 MB	1.5 MB	0.8 MB
Temporal (datetime/date/duration/time), no nulls	1.5 MB	1.5 MB	5.4 MB
Temporal (datetime/date/duration/time), ~10% nulls	1.6 MB	5.7 MB	5.5 MB
Binary, no nulls	9.8 MB	10.6 MB	0 MB
Binary, ~10% nulls	0.1 MB	10.4 MB	10.6 MB

Notes

• Temporal export shows the largest gain (~27–50×): the old path boxes every value into a Python datetime object when converting through pandas; the new path casts directly to int64 in C with no Python allocation. The 4
  MB RSS spike in the old paths disappears entirely.
• String export (~2–2.6×): the new path reads raw UTF-8 bytes directly from Polars' Arrow LargeUtf8 buffer in C, bypassing str object creation and re-encoding. The small RSS increase reflects the SBDF output buffer,
  not intermediate Python objects.
• String import is roughly flat on time — building an Arrow buffer directly is comparable work to converting through pandas.
• Binary (~1.7–1.9× export, ~1.3× import): benefits from skipping the Polars → pandas conversion chain; the memory profile is similar across paths since the raw byte data dominates.