Skip to content

fix: parquet serializer#187

Merged
runejo merged 6 commits intomainfrom
511-parquet-serializer
Apr 22, 2026
Merged

fix: parquet serializer#187
runejo merged 6 commits intomainfrom
511-parquet-serializer

Conversation

@trygu
Copy link
Copy Markdown
Member

@trygu trygu commented Feb 25, 2026

  • Fixes issues related to too many rows when you have more than one content. Tests have been updated to reflect that.
  • Fixes issues with wrong time in the Parquet-file when the VALUES order differs from the TIMEVAL order.

Fixes PxTools/PxWebApi#511

@trygu
Copy link
Copy Markdown
Member Author

trygu commented Feb 25, 2026

There is still an issue related to Px-files without CODE for Time.. since .TimeValue is missing (or wrong), which makes the datetime creation fail.

Copy link
Copy Markdown
Contributor

@likp likp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test fail, have to be fixed, connected to @trygu comment above.

// the data which follows VALUES(...).
row[dataFieldIndices[variable.Name]] = value; // Original time-value (from VALUES)
row[dataFieldIndices["timestamp"]] = ParseTimeScale(value, variable.TimeScale); // Parsed timestamp
value = variable.Values[index[i]].TimeValue;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@trygu why did you add this line again?
value = variable.Values[index[i]].TimeValue;

If I remove it the 14216.parquet works

{"tettsted":"0801","år":"2025","timestamp":1735689600000,"ContentsCode_Areal":275.87,"ContentsCode_Areal_symbol":null,"ContentsCode_Bosatte":1110887,"ContentsCode_Bosatte_symbol":null}
{"tettsted":"0801","år":"2024","timestamp":1704067200000,"ContentsCode_Areal":276.3,"ContentsCode_Areal_symbol":null,"ContentsCode_Bosatte":1098061,"ContentsCode_Bosatte_symbol":null}
Image

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I get it, I didn't know that TimeValue looked like 20071 and the Code is 2007K1

Copy link
Copy Markdown
Member

@runejo runejo Apr 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can make it work in the ParquetSerilizer using Code and Value, but the real problem seems to be in Paxiom. TimeValue is clearly wrong. Can we discuss it tomorrow @likp ?

image

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, we discussed it yesterday in our weekly PxWebApi meeting, and we agreed to make it work without looking at TimeValue. I'll commit the fix on this branch now.

@likp
Copy link
Copy Markdown
Contributor

likp commented Apr 21, 2026

Connected to PxTools/PxWebApi#550

@runejo runejo changed the title bugfix: 511 parquet serializer fix: parquet serializer Apr 22, 2026
@sonarqubecloud
Copy link
Copy Markdown

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@runejo runejo merged commit 79f59f8 into main Apr 22, 2026
10 checks passed
@runejo runejo deleted the 511-parquet-serializer branch April 22, 2026 06:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Parquet serializer

3 participants