Skip to content

feat: upgrade msgpackr to v2 and use structon for random-access struct encoding#457

Draft
kriszyp wants to merge 4 commits intomainfrom
feat/structon-random-access
Draft

feat: upgrade msgpackr to v2 and use structon for random-access struct encoding#457
kriszyp wants to merge 4 commits intomainfrom
feat/structon-random-access

Conversation

@kriszyp
Copy link
Copy Markdown
Member

@kriszyp kriszyp commented May 2, 2026

Summary

  • Upgrades msgpackr from 1.11.9 to ^2.0.1
  • Adopts the structon package (the intended replacement for msgpackr v1's randomAccessStructure feature)
  • RecordEncoder now extends createStructon(Encoder) instead of plain Encoder
  • Removes this.randomAccessStructure = true from OpenDBIObject (the option was removed in msgpackr v2)

Purpose

msgpackr v1's built-in randomAccessStructure option was removed in v2. The author extracted this functionality into the separate structon package. This change adopts structon to maintain random-access struct encoding (which enables efficient in-place field reads without full deserialization).

A bug in structon's onLoadedStructures was also fixed (in the structon repo): it previously only reloaded typed structures when loading shared structures from the DB, but not named structures. This caused isCompatible to always fail (named structure count mismatch), triggering an infinite encode retry loop. The fix reloads named structures into this.structures, unfreezes array elements (needed because freezeData: true froze them, but msgpackr lazily adds a read property), and clears transitions to force msgpackr to rebuild and update lastNamedStructuresLength.

Test plan

  • npm run test:unit:resources — 336 passing, 5 pre-existing failures (blob test + 4 vector index tests, pre-existing before this change)
  • Integration tests

Attention areas

  • The TypeScript cast createStructon(Encoder) as typeof Encoder is needed because createStructon's return type needs to be an exact typeof Encoder for subclassing to work cleanly in TypeScript. The runtime behavior is correct — structon does return a subclass of the input.
  • The structon dependency uses file:../../structon (local sibling directory) for now. Once structon 1.0.1 is published to npm (with the onLoadedStructures fix), this should be updated to ^1.0.1.
  • The structon fix is in a separate PR in the structon repo (feat/onloaded-named-structures-fix branch).

🤖 Generated with Claude Code

…t encoding

msgpackr v1's randomAccessStructure option was removed in v2. This adopts the
structon package (the intended replacement) which provides the same capability
as a mixin via createStructon(Encoder). RecordEncoder now extends the structon-
enhanced encoder class instead of plain Encoder, and the removed
randomAccessStructure option is dropped from OpenDBIObject.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@kriszyp kriszyp requested a review from a team as a code owner May 2, 2026 19:00
@socket-security
Copy link
Copy Markdown

socket-security Bot commented May 2, 2026

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addedmsgpackr@​2.0.19810010094100

View full report

Comment thread package.json Outdated
"mqtt-packet": "~9.0.1",
"msgpackr": "1.11.9",
"msgpackr": "^2.0.1",
"structon": "file:../../structon",
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

file: path dependency — not installable from npm.

"structon": "file:../../structon" will break npm install for anyone who doesn't have a sibling structon/ directory (CI, other contributors, downstream consumers of the published package). The PR description acknowledges this is intentional while waiting for structon 1.0.1 to be published — this line needs to be updated to "^1.0.1" (or whichever published semver range is correct) before merging to main.

Also: structon is a new runtime dependency but there's no entry for it in dependencies.md, which documents the justification for every runtime dep. A short entry covering size, security track record, and removal plan should be added alongside this change.

@claude
Copy link
Copy Markdown
Contributor

claude Bot commented May 2, 2026

2 blockers.

1. structon not documented in dependencies.md (unchanged from prior run)

File: dependencies.md
What: structon ^1.0.3 is a runtime dependency in package.json but has no entry in dependencies.md.
Why it matters: Repo policy requires every runtime dependency to have a justified section covering size, security, memory cost, overlap with existing packages, and a removal plan.
Suggested fix: Add a ## structon section before merging.

2. Lock file stale — npm ci breaks in lint CI

File: package.json:211 / package-lock.json
What: package.json now declares "structon": "^1.0.3" but package-lock.json still records file:../../structon (local symlink, version 1.0.0). The three "Update structon" commits each edited only package.json; the lock was never regenerated. lint-code.yaml runs npm ci, which exits with "lock file out of sync" when packages[""].dependencies.structon in the lock disagrees with package.json.
Suggested fix: Run npm install to regenerate the lock from the npm registry and commit the updated package-lock.json.

Comment thread package.json Outdated
"mqtt-packet": "~9.0.1",
"msgpackr": "1.11.9",
"msgpackr": "^2.0.1",
"structon": "^1.0.1",
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The file: path is now fixed to ^1.0.1 — thanks. The dependencies.md entry for structon is still missing (repo policy requires every runtime dep to have a justified section there covering size, security, and removal plan). Please add one before merging.

@kriszyp kriszyp marked this pull request as draft May 2, 2026 21:50
Comment thread package.json
"mqtt-packet": "~9.0.1",
"msgpackr": "1.11.9",
"msgpackr": "^2.0.1",
"structon": "^1.0.3",
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lock file is stale — npm ci will fail in lint CI.

package.json says ^1.0.3 but package-lock.json still records "structon": "file:../../structon" (version 1.0.0, resolved as a local symlink). The three "Update structon" commits each bumped only package.json; the lock was never regenerated after the original commit.

lint-code.yaml uses npm ci, which compares packages[""].dependencies.structon in the lock against package.json and exits with "lock file out of sync" when they disagree.

Fix: run npm install (to regenerate the lock from the npm registry) and commit the updated package-lock.json.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant