Skip to content

to_arrow_ipc() output missing Arrow IPC end-of-stream (EOS) marker #867

@jatorre

Description

@jatorre

Summary

The output from nanoarrow's to_arrow_ipc() DuckDB extension does not include the Arrow IPC end-of-stream (EOS) marker (0xFFFFFFFF 0x00000000). This causes consumers that expect a well-terminated IPC stream to fail.

Reproduction

When using to_arrow_ipc() from the nanoarrow DuckDB extension and concatenating the resulting chunks, the stream lacks the 8-byte EOS sentinel defined by the Arrow IPC spec:

The end of a stream is indicated by a 0 length and 0 metadata length.

For example, feeding the output to DuckDB-WASM's insertArrowFromIPCStream fails with:

Header-type of flatbuffer-encoded Message is not RecordBatch

Workaround

Manually appending the EOS marker after concatenating chunks resolves the issue:

chunks.push(Buffer.from([0xFF, 0xFF, 0xFF, 0xFF, 0x00, 0x00, 0x00, 0x00]));
return Buffer.concat(chunks);

Expected behavior

to_arrow_ipc() should produce a complete, spec-compliant IPC stream that includes the EOS marker, so consumers don't need to append it manually.

Context

Discussion originated in duckdb/duckdb-node-neo#45. The Mosaic project also uses this to_arrow_ipc() + nanoarrow pattern.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions