Summary
The output from nanoarrow's to_arrow_ipc() DuckDB extension does not include the Arrow IPC end-of-stream (EOS) marker (0xFFFFFFFF 0x00000000). This causes consumers that expect a well-terminated IPC stream to fail.
Reproduction
When using to_arrow_ipc() from the nanoarrow DuckDB extension and concatenating the resulting chunks, the stream lacks the 8-byte EOS sentinel defined by the Arrow IPC spec:
The end of a stream is indicated by a 0 length and 0 metadata length.
For example, feeding the output to DuckDB-WASM's insertArrowFromIPCStream fails with:
Header-type of flatbuffer-encoded Message is not RecordBatch
Workaround
Manually appending the EOS marker after concatenating chunks resolves the issue:
chunks.push(Buffer.from([0xFF, 0xFF, 0xFF, 0xFF, 0x00, 0x00, 0x00, 0x00]));
return Buffer.concat(chunks);
Expected behavior
to_arrow_ipc() should produce a complete, spec-compliant IPC stream that includes the EOS marker, so consumers don't need to append it manually.
Context
Discussion originated in duckdb/duckdb-node-neo#45. The Mosaic project also uses this to_arrow_ipc() + nanoarrow pattern.
Summary
The output from nanoarrow's
to_arrow_ipc()DuckDB extension does not include the Arrow IPC end-of-stream (EOS) marker (0xFFFFFFFF 0x00000000). This causes consumers that expect a well-terminated IPC stream to fail.Reproduction
When using
to_arrow_ipc()from the nanoarrow DuckDB extension and concatenating the resulting chunks, the stream lacks the 8-byte EOS sentinel defined by the Arrow IPC spec:For example, feeding the output to DuckDB-WASM's
insertArrowFromIPCStreamfails with:Workaround
Manually appending the EOS marker after concatenating chunks resolves the issue:
Expected behavior
to_arrow_ipc()should produce a complete, spec-compliant IPC stream that includes the EOS marker, so consumers don't need to append it manually.Context
Discussion originated in duckdb/duckdb-node-neo#45. The Mosaic project also uses this
to_arrow_ipc()+ nanoarrow pattern.