[Go] How to approach to implement Parquet-Go file format

[hkpeaks]

Describe the usage question you have. Please include as many useful details as possible.

Based on current testing my Golang dataframe project which is achieving outstanding performance when data source is csv file. Next step I want to implement parquet file format when it becomes very popular. After a basic reserach in this topic, I am still not clear to approach how to implement Parquet using your golang library as there are little code sample I can find in the community.

I have built a in-memory and streaming engine which is using simple data structure bytearray similar to csv, I have developed a set of algorithms to support byte-to-byte conversion directly for ETL functons such as Read/Write file, Distinct, GroupBy, JoinTable and Select column/row. Also solve JoinTable for billion of row using 32GB memory.

My requirement to support Parquet format is simple. I need to know how to read bytearray of particular columns of Parquet format. After I can read, of course need to know how to write to disk.

Today I have published the pre-release 64-bit runtime for Windows/Linux of my project @ https://github.com/hkpeaks/peaks-consolidation/releases

Component(s)

Go

[mapleFU]

Parquet follows the format described in https://github.com/apache/parquet-format

If you're using arrow, you can use pqarrow under https://github.com/apache/arrow/tree/main/go/parquet

[hkpeaks]

I have tested arrow before. My project use bytearray as it consumes less CPU and memory resources. In fact, I have a long video that shows how my app processes large CSV files faster than Polars which implements arrow to process parquet files. https://youtu.be/1Kn665ADSck
So I want to know what processing speed I can get if my app support read data from Parquet file particular columns directly.

[mapleFU]

Then just follow the parquet standard :-) https://github.com/apache/parquet-format

[zeroshade]

@hkpeaks You can use the parquet package (https://pkg.go.dev/github.com/apache/arrow/go/v12/parquet) to get direct access and reading of columns and data from the parquet file without going through the arrow conversions if you don't need to do so. The primary interface for interacting with a file directly would be the file package which provides readers for row groups and individual columns.

I did a lot of work trying to get it as performant as possible when i originally wrote it, but as with many things in software I'm sure there are still areas that could be improved and I'll happily review any contributions you make or ideas you have for improving the performance further! Feel free to reach out if you need more help understanding how to use the package.

[hkpeaks]

@zeroshade

1. go get -u github.com/apache/arrow/go/v12/parquet
2. go install github.com/apache/arrow/go/v12/parquet/cmd/parquet_reader@latest
3. go install github.com/apache/arrow/go/v12/parquet/cmd/parquet_schema@latest

I have downloaded for each, why the file in 2 and 3 are identical, 1 and 2 also have some duplications

Where I can find hello world code example to use this libraries?

[hkpeaks]

Firstly, I need to know where to find the start and end address for each row block. Secondly, need to know where to find the start and end address of each column and column page contained within a row block. For a particular query which requires 5 columns, I can use Goruntine and mmap to read given set of row blocks for selected columns in parallel for each batch. e.g.

Row block batch 1: use 20+ threads to read 5 columns of 1~20 blocks

Row block batch 2: use 20+ threads to read 5 columns of 21~40 blocks

Row block batch 3: use 20+ threads to read 5 columns of 41~60 blocks

To determine how much row block I shall read for each stream, I need to know how many columns and row blocks of a file.

I have implemented streaming for csv in a similar way and want to extend it to cover parquet and json file format. The parquet file format offers two advantages: 1) it allows reading select columns directly and 2) it offers compression.

If the Apache library offer API to answer the about question, it can offer great help of my project to support a hyper performance of parquet format.

Data Structure my current project: https://github.com/hkpeaks/peaks-consolidation/blob/main/PeaksFramework/data_structure.go

Runtime for Windows/Linux you can find https://github.com/hkpeaks/peaks-consolidation/releases
You can have a test to see whehter its read-query-write csv is faster than other software read-query-write parquet. It supports JoinTable for a billion rows using only 32GB RAM. Input 67.2GB csv output 90 GB csv.

[mapleFU]

Oopts, seems parquet-go in arrow doesn't provide some examples

I guess: https://github.com/apache/arrow/blob/65520b361941e9abad386614999dbc250295959e/go/parquet/file/file_reader.go#L79 can be a good start point. You can pay attention to the ReaderProperties

Once you open the file, you can get metadata and know the file schema and rowgroups from metadata interface.

Then, you can open rowgroups via https://github.com/apache/arrow/blob/65520b361941e9abad386614999dbc250295959e/go/parquet/file/file_reader.go#L305 . And open column chunk in row group

[hkpeaks]

I have implemented mmap for csv that can work in Windows.
For try the parquet, I see this error "error opening parquet file: mmap not implemented on windows"

C:\Users\85292\go\pkg\mod\github.com\apache\arrow\go\v13@v13.0.0-20230520140400-65520b361941\parquet\cmd\parquet_schema>parquet_schema 0.1MillionRows.parquet
required group field_id=-1 root {
optional int64 field_id=-1 Date;
optional byte_array field_id=-1 Ledger (String);
optional int64 field_id=-1 Account;
optional byte_array field_id=-1 PartNo (String);
optional byte_array field_id=-1 Project (String);
optional byte_array field_id=-1 Contact (String);
optional byte_array field_id=-1 Unit Code (String);
optional int64 field_id=-1 Quantity;
optional double field_id=-1 Unit Price;
optional byte_array field_id=-1 D/C (String);
optional byte_array field_id=-1 Currency (String);
optional double field_id=-1 Exchange Rate;
optional double field_id=-1 Original Amount;
optional double field_id=-1 Base Amount;
}

C:\Users\85292\go\pkg\mod\github.com\apache\arrow\go\v13@v13.0.0-20230520140400-65520b361941\parquet\cmd\parquet_schema>parquet_reader 0.1Million_reader
error opening parquet file: mmap not implemented on windows

C:\Users\85292\go\pkg\mod\github.com\apache\arrow\go\v13@v13.0.0-20230520140400-65520b361941\parquet\cmd\parquet_schema>

[mapleFU]

https://github.com/apache/arrow/blob/main/go/parquet/file/file_reader_mmap_windows.go

I guess go file reader can only use mmap in other os. You can disable mmap on windows( by set memoryMap = false)

[hkpeaks]

My app need to support billion rows with little memory, so using mmap is essential.
My csv extractor success in using mmap for billion rows JoinTable.

[mapleFU]

I guess you can just comparing them under a linux machine, or wsl. Seems that using mmap needs extra development in this library. You can try on wsl2 or linux machines, develop mmap yourself if you want.

By the way, I think JoinTable should use spill instead of mmap.

[hkpeaks]

I try linux before, but fail to complie it
mmap is extensive use througout my system
See my source code how to use mmap to read csv file partition to support streaming https://github.com/hkpeaks/peaks-consolidation/blob/main/PeaksFramework/read_file.go

[zeroshade]

@hkpeaks To answer your questions:

1. go get -u github.com/apache/arrow/go/v12/parquet

This will just get the whole package and add it to your module. The others are commands in that package so of course there'll be duplications the two commands both rely on this package.

2. go install github.com/apache/arrow/go/v12/parquet/cmd/parquet_reader@latest

This is just a simple mainprog which will dump metadata, info and the data from a parquet file.

3. go install github.com/apache/arrow/go/v12/parquet/cmd/parquet_schema@latest

This mainprog is just a further stripped down command which only dumps a schema from a parquet file, it will not read data.

Firstly, I need to know where to find the start and end address for each row block. Secondly, need to know where to find the start and end address of each column and column page contained within a row block. For a particular query which requires 5 columns, I can use Goruntine and mmap to read given set of row blocks for selected columns in parallel for each batch. e.g.

Row block batch 1: use 20+ threads to read 5 columns of 1~20 blocks

Row block batch 2: use 20+ threads to read 5 columns of 21~40 blocks

Row block batch 3: use 20+ threads to read 5 columns of 41~60 blocks

To determine how much row block I shall read for each stream, I need to know how many columns and row blocks of a file.

Once you open the file, (either by OpenParquetFile or NewParquetReader), you can retrieve the information you're looking for:

• Reader.NumRowGroups() gives you the total number of row groups in the file.
• Reader.Metadata().RowGroup(i int) will get you the Row Group metadata for a specific row group. (RowGroupMetaData)
  • You can also use Reader.RowGroup to get a sub reader specifically for that row group. This reader can report the total byte size for the row group and provide the metadata for that row group directly. It can also get you page readers for any given column via GetColumnPageReader
    • It should be pretty easy to leverage the Column method to get column readers for reading columns in parallel. In fact, we already do this for the pqarrow package when reading a whole row group into a single record, or a whole file into a table: see here: https://github.com/apache/arrow/blob/go/v12.0.0/go/parquet/pqarrow/file_reader.go#L300
  • RowGroupMetaData.FileOffset() is the location in the file where the data for this row group begins
  • You can use the ColumnChunk method to get the metadata for a specific column, which contains the offsets for that particular column along with the rest of the column metadata.

I hope that the above answers your questions concerning the API (along with pointing you at the documentation for any other methods you might need). But let me know if there's any other methods/functions you need that you can't find (or don't exist yet).

See my source code how to use mmap to read csv file partition to support streaming https://github.com/hkpeaks/peaks-consolidation/blob/main/PeaksFramework/read_file.go

I looked at the file there I don't see any usage of mmap at all there. You're just using os.Open and using the ReadAt method on the file. (If I'm wrong, can you please provide the line number / direct link to the mmap?) If you follow @mapleFU's suggestion of passing memoryMap = false to OpenParquetFile then that's what will be used, it will just use os.Open and ReadAt to read from the file.

[zeroshade]

@hkpeaks All of this being said, it would be amazing if you could make some contributions to the CSV reader in the Arrow package as there's a lot of work there that needs to be done to optimize it. Right now it's just a naive reader that doesn't do any parallelism.