diff --git a/docs/DECISIONS/2026-05-12-flowpulse-observation-identity.md b/docs/DECISIONS/2026-05-12-flowpulse-observation-identity.md index 8fb58d9b..0fe30a63 100644 --- a/docs/DECISIONS/2026-05-12-flowpulse-observation-identity.md +++ b/docs/DECISIONS/2026-05-12-flowpulse-observation-identity.md @@ -4,7 +4,7 @@ Date: 2026-05-12 ## Status -Accepted for the MVP foundation. +Accepted for the V0 local indexer/verifier package and MVP foundation. ## Context @@ -14,10 +14,11 @@ The indexer needs a canonical identity for an observed FlowPulse log after recei ## Decision -FlowMemory will use three separate identifiers in the indexer/verifier MVP: +FlowMemory V0 uses separate identifiers: - `pulseId`: emitted by the contract inside the FlowPulse event. - `observationId`: derived by the indexer from receipt/log metadata after execution. +- `cursorId`: derived by the indexer for deterministic fixture scan progress. - `reportId`: derived by the verifier from canonical JSON report content. `pulseId` is protocol data, not canonical observation identity. The canonical indexer identity is `observationId`. @@ -38,53 +39,92 @@ The `topic0` hash is: ## Observation ID -The indexer derives `observationId` from: +The indexer derives `observationId` from the crypto V0 type: + +```text +FlowPulseObservationV0(uint256 chainId,address emittingContract,uint64 blockNumber,bytes32 blockHash,bytes32 txHash,uint32 transactionIndex,uint32 logIndex,bytes32 eventSignature,bytes32 pulseId,bytes32 rootfieldId) +``` + +Fields: -- Domain: `flowmemory.flowpulse.observation.v0` - `chainId` - `emittingContract` -- `eventSignature` - `blockNumber` - `blockHash` - `txHash` - `transactionIndex` - `logIndex` +- `eventSignature` +- `pulseId` +- `rootfieldId` Preimage: ```text keccak256(abi.encode( - "flowmemory.flowpulse.observation.v0", + keccak256("FlowPulseObservationV0(uint256 chainId,address emittingContract,uint64 blockNumber,bytes32 blockHash,bytes32 txHash,uint32 transactionIndex,uint32 logIndex,bytes32 eventSignature,bytes32 pulseId,bytes32 rootfieldId)"), chainId, emittingContract, - eventSignature, blockNumber, blockHash, txHash, transactionIndex, - logIndex + logIndex, + eventSignature, + pulseId, + rootfieldId )) ``` -`blockHash` and `blockNumber` are included so a reorged log occurrence and a later re-mined occurrence can be represented as distinct observations. `eventSignature` is included so the identity is explicitly scoped to FlowPulse v0 logs, not arbitrary logs at the same receipt location. +`blockHash` and `blockNumber` are included so a reorged log occurrence and a later re-mined occurrence can be represented as distinct observations. `eventSignature` scopes the identity to FlowPulse v0 logs, not arbitrary logs at the same receipt location. + +Decoded fields such as `actor`, `pulseType`, `subject`, `commitment`, `parentPulseId`, `sequence`, `occurredAt`, and `uri` are stored with the observation but are not part of the identity preimage. If the same `observationId` is observed with different decoded fields, that is an indexer integrity failure. + +## Cursor ID + +The indexer derives `cursorId` from: + +- Domain: `flowmemory.indexer.cursor.v0` +- `chainId` +- `sourceSetId` +- `blockNumber` +- `blockHash` +- `transactionIndex` +- `logIndex` + +Preimage: + +```text +keccak256(abi.encode( + "flowmemory.indexer.cursor.v0", + chainId, + sourceSetId, + blockNumber, + blockHash, + transactionIndex, + logIndex +)) +``` -Decoded fields such as `pulseId`, `rootfieldId`, `actor`, `pulseType`, `subject`, `commitment`, `parentPulseId`, `sequence`, `occurredAt`, and `uri` are stored with the observation but are not part of the identity preimage. If the same `observationId` is observed with different decoded fields, that is an indexer integrity failure. +`sourceSetId` is a deterministic hash of the chain id and normalized emitting-contract set. Cursor identity includes block hash so scan progress can detect a changed canonical chain. ## Lifecycle Names -- `pending`: candidate scan work or pre-receipt/pre-finality context. -- `mined`: successful receipt contains a decodable FlowPulse log and `observationId` exists. -- `finalized`: mined observation remains canonical after the configured finality policy. -- `reorged`: observation block/log is no longer canonical or was marked removed. +- `observed`: successful receipt contains a decodable FlowPulse log and no finality policy is applied. +- `pending`: decoded observation is above the configured finality threshold. +- `finalized`: decoded observation is at or below the configured finality threshold. +- `removed`: provider marks the log removed. +- `superseded`: an older observation for the same `pulseId` was replaced. +- `reorged`: canonical block-hash check shows the indexed block is no longer canonical. -The MVP does not require mempool indexing. A canonical `observationId` begins at `mined`. +The V0 package models these states with fixtures. It does not implement production reorg handling. ## Duplicate Names -- Exact duplicate: same `observationId` and same decoded content; idempotent replay. -- Conflicting duplicate: same `observationId` but changed decoded content or metadata; indexer integrity failure. -- Pulse duplicate: same contract-emitted `pulseId` but different `observationId`; separate observations that require verifier/operator policy. -- Reorg replacement: different `observationId` caused by changed `blockHash`, block position, transaction position, or log position. +- Exact duplicate: same `observationId` and same canonical observation JSON; idempotent replay. +- Conflicting duplicate: same `observationId` but changed canonical content; indexer integrity failure. +- Pulse duplicate: same contract-emitted `pulseId` but different `observationId`; preserve both observations. +- Reorg replacement: same `pulseId` at a changed block/log location. ## Report ID @@ -94,29 +134,32 @@ The verifier derives `reportId` from the canonical report body: keccak256(canonical_json(reportCore)) ``` -The `reportCore` includes `schema = flowmemory.verifier.report.v0`, `observationId`, observed receipt/log metadata, decoded FlowPulse fields, status, reason codes, resolver policy id, and verifier spec version. The report id does not include signatures, wall-clock generation timestamps, local file paths, or operator notes. +The `reportCore` includes `schema = flowmemory.verifier.report.v0`, `observationId`, observed receipt/log metadata, decoded FlowPulse fields, status, reason codes, evidence refs, resolver policy id, and verifier spec version. The report id does not include signatures, wall-clock generation timestamps, local file paths, or operator notes. ## Consequences +- Contracts remain unaware of receipt-only metadata. - Indexers can distinguish contract-emitted pulse identity from observed-log identity. - Reorged observations remain addressable for audits without being treated as current canonical facts. +- Cursor progress can include block hash without changing observation identity. - Verifiers can produce deterministic reports bound to receipt/log facts. - Dashboards and explorers can distinguish observed, verified, unresolved, unsupported, failed, reorged, stale, disputed, and superseded outcomes later without changing the observation identity. ## Out Of Scope - Production indexer runtime. -- Live RPC integration. -- Database schema. +- Production live RPC deployment. +- Production database schema. - Verifier economics. - Proof network. - Artifact canonicalization format. -- Resolver policy. +- Resolver policy beyond local fixtures. - Report signing or verifier attestation implementation. ## Follow-Ups - Define artifact commitment canonicalization. -- Define resolver policy v0. +- Define resolver policy beyond local fixtures. - Define Base finality policy. +- Define durable persistence schema. - Decide whether future report attestations should use EIP-712 or another signature format. diff --git a/docs/INDEXER_VERIFIER_MVP.md b/docs/INDEXER_VERIFIER_MVP.md new file mode 100644 index 00000000..46ac5761 --- /dev/null +++ b/docs/INDEXER_VERIFIER_MVP.md @@ -0,0 +1,271 @@ +# Indexer Verifier MVP + +This document describes the runnable FlowMemory Indexer + Verifier V0 local package. It advances issues #13, #14, #43, #44, #45, #46, #47, #54, and #55 by proving the off-chain path with fixtures, pure functions, local JSON persistence, CLIs, and tests. + +V0 is non-production. It does not include tokenomics, a verifier network, production RPC deployment, a production database, proof infrastructure, or chain/L1 implementation. + +## Packages + +- `services/shared`: FlowPulse constants, narrow ABI helpers, Keccak-256, canonical JSON, observation/cursor/report identity helpers, fixture parser, and tests. +- `services/indexer`: fixture receipt ingestion, FlowPulse log decoding, observation identity, cursor identity, duplicate/reorg state modeling, JSON persistence, CLI, and tests. +- `services/verifier`: fixture artifact resolver, commitment checks, report schema, deterministic report IDs, JSON persistence, CLI, and tests. + +Run from the repository root: + +```powershell +npm test +npm run index:fixtures +npm run verify:fixtures +npm run e2e +``` + +The commands require no secrets and no live RPC. + +## FlowPulse Input + +Contracts emit `FlowPulse` events. V0 reads those events only after receipts/logs exist. + +Event signature: + +```text +FlowPulse(bytes32,bytes32,address,uint8,bytes32,bytes32,bytes32,uint64,uint64,string) +``` + +`topic0`: + +```text +0x5d07190b9ae441b4d7b16259a48424acd451492b12f5f99a29f5bfd992c13e43 +``` + +Indexed topics: + +- `pulseId` +- `rootfieldId` +- `actor` + +Data fields: + +- `pulseType` +- `subject` +- `commitment` +- `parentPulseId` +- `sequence` +- `occurredAt` +- `uri` + +Receipt/log metadata: + +- `chainId` +- `blockNumber` +- `blockHash` +- `transactionHash` / `txHash` +- `transactionIndex` +- `logIndex` +- `address` / `emittingContract` +- `status` +- provider `removed` flag when present + +Contracts do not know `txHash`, `transactionIndex`, `logIndex`, or final block metadata during execution. The indexer derives those values from receipts/logs. + +## Indexer Pipeline + +1. Load receipt fixtures from `services/indexer/fixtures/flowpulse-receipts.json`. +2. Drop reverted receipts into `rejectedLogs` with `receipt.reverted`. +3. Decode only logs whose `topic0` matches FlowPulse v0. +4. Normalize hex, addresses, hashes, and integer-like values. +5. Derive `observationId` from receipt/log location. +6. Derive `cursorId` from source-set and block/log ordering metadata. +7. Assign lifecycle state from fixture finality and reorg inputs. +8. Detect exact duplicates, conflicting duplicates, pulse duplicates, and reorg replacements. +9. Persist deterministic JSON to `services/indexer/out/indexer-state.json`. + +Malformed logs are rejected with deterministic reason codes and do not become verifier inputs. + +## Identity Model + +`pulseId` is contract-emitted protocol data. It is not the canonical observed-log identity. + +`observationId` is indexer-derived: + +```text +keccak256(abi.encode( + "flowmemory.flowpulse.observation.v0", + chainId, + sourceContract, + txHash, + logIndex +)) +``` + +`sourceContract` is the normalized emitting contract address. `txHash` and `logIndex` come from the receipt/log. `blockHash`, `blockNumber`, `transactionIndex`, `eventSignature`, and decoded FlowPulse fields are stored with the observation but are not part of the V0 `observationId` preimage. + +`cursorId` is indexer-derived scan progress identity: + +```text +keccak256(abi.encode( + "flowmemory.indexer.cursor.v0", + chainId, + sourceSetId, + blockNumber, + blockHash, + transactionIndex, + logIndex +)) +``` + +`sourceSetId` is a deterministic hash of the chain id and normalized emitting-contract set: + +```text +keccak256("flowmemory.indexer.source_set.v0||") +``` + +`reportId` is verifier-derived: + +```text +keccak256(canonical_json(reportCore)) +``` + +The report digest excludes wall-clock timestamps, local paths, signatures, and operator notes. + +## Indexer States + +V0 observation lifecycle states: + +- `observed`: decoded from a successful receipt/log with no finality policy applied. +- `pending`: decoded but above the configured fixture finality threshold. +- `finalized`: at or below the configured fixture finality threshold. +- `removed`: provider fixture marks the log as removed. +- `superseded`: older observation for the same `pulseId` was replaced by another observation. +- `reorged`: fixture canonical block-hash check says the observation's block is no longer canonical. + +This is a fixture state model, not production reorg handling. + +## Verifier Pipeline + +1. Read persisted indexer state from `services/indexer/out/indexer-state.json`, or build it from fixtures if missing. +2. Load fixture artifacts from `services/verifier/fixtures/artifacts.json`. +3. Use resolver policy `flowmemory.resolver.policy.v0.fixture`. +4. Refuse arbitrary HTTP/IPFS fetching in V0; fixture URIs are lookup hints only. +5. Apply supported commitment rules. +6. Generate canonical report cores and `reportId` digests. +7. Persist deterministic JSON to `services/verifier/out/reports.json`. + +## Verifier Statuses + +V0 report statuses: + +- `valid`: supported checks passed against allowed fixture evidence. +- `invalid`: supported checks ran and at least one required check failed. +- `unresolved`: required fixture evidence is missing or policy-rejected. +- `unsupported`: pulse type or artifact semantics are outside V0 rules. +- `reorged`: observation is removed or reorged and should not be treated as current canonical evidence. + +Earlier planning terms map as follows: `verified` becomes `valid`, `failed` becomes `invalid`, and `observed`, `stale`, `disputed`, and `superseded` remain future report or lifecycle concepts outside the V0 report result enum. + +## Commitment Checks + +For `ROOTFIELD_REGISTERED` (`pulseType = 1`): + +- `subject` must equal `rootfieldId`. +- `commitment` must equal `keccak256(abi.encode(schemaHash, metadataHash))`. +- `schemaHash` and `metadataHash` come from allowed fixture evidence. + +For `ROOT_COMMITTED` (`pulseType = 2`): + +- `subject` must equal `root`. +- `commitment` must equal `keccak256(abi.encode(root, artifactCommitment))`. +- `root` and `artifactCommitment` come from allowed fixture evidence. + +Unknown pulse types return `unsupported`, not `valid`. +Missing evidence returns `unresolved`, not `invalid`. +Bad commitments or subject mismatches return `invalid`. +Removed or reorged observations return `reorged`. + +## Persistence + +Indexer persistence schema: + +```text +flowmemory.indexer.persistence.v0 +``` + +Indexer state schema: + +```text +flowmemory.indexer.state.v0 +``` + +Verifier persistence schema: + +```text +flowmemory.verifier.persistence.v0 +``` + +Verifier report schema: + +```text +flowmemory.verifier.report.v0 +``` + +The JSON writer uses canonical key ordering, no wall-clock timestamps, no secrets, and stable fixture ordering. + +## Off-Chain Boundary + +These stay off-chain in V0: + +- Raw artifacts and evidence bundles. +- Resolver caches. +- Full verifier reports unless a later storage policy intentionally persists them elsewhere. +- AI memory, model data, embeddings, media, and heavy artifacts. +- Private keys, RPC keys, API keys, seed phrases, webhook URLs, and local operator notes. + +Future on-chain candidates: + +- Compact report digests. +- Report-root commitments. +- Verifier attestations or signatures. +- Dispute state. +- Proof results. + +Those are future protocol decisions, not part of this local package. + +## Handoff Outputs + +- Dashboard-friendly indexer state: `services/indexer/out/indexer-state.json` +- Chain/devnet-friendly verifier report fixture: `services/verifier/out/reports.json` +- Indexer state JSON schema: `services/indexer/fixtures/indexer-state.schema.json` +- Verifier report JSON schema: `services/verifier/fixtures/verification-report.schema.json` +- Receipt fixtures: `services/indexer/fixtures/flowpulse-receipts.json` +- Artifact fixtures: `services/verifier/fixtures/artifacts.json` + +## Open Questions + +- What exact artifact canonicalization format should produce `artifactCommitment`? +- What finality depth should a future Base RPC indexer use? +- Should live RPC indexing persist cursors before or after report generation? +- Should future attestations use EIP-712, raw digest signatures, or another envelope? +- How should dashboards display pulse duplicates versus exact duplicate observations? + +## PR-Ready Summary + +What changed: + +- Added a runnable fixture-first indexer/verifier package with CLIs, persistence, schemas, and tests. +- Defined contract `pulseId`, indexer `observationId`, indexer `cursorId`, and verifier `reportId`. +- Defined V0 lifecycle states, duplicate behavior, resolver policy boundaries, and report statuses. + +Why it changed: + +- The service layer needs a deterministic off-chain path before live RPC, production storage, verifier networking, or on-chain attestations. + +Checks: + +- `npm test` +- `npm run index:fixtures` +- `npm run verify:fixtures` +- `npm run e2e` + +Risks and follow-ups: + +- V0 fixtures are synthetic and do not claim production reorg handling. +- Live RPC, durable database storage, artifact canonicalization, report signing, and attestations need separate scoped issues. diff --git a/package-lock.json b/package-lock.json new file mode 100644 index 00000000..1f3fbdf4 --- /dev/null +++ b/package-lock.json @@ -0,0 +1,36 @@ +{ + "name": "flowmemory-services-v0", + "lockfileVersion": 3, + "requires": true, + "packages": { + "": { + "name": "flowmemory-services-v0", + "workspaces": [ + "services/shared", + "services/indexer", + "services/verifier" + ] + }, + "node_modules/@flowmemory/indexer-v0": { + "resolved": "services/indexer", + "link": true + }, + "node_modules/@flowmemory/indexer-verifier-foundation": { + "resolved": "services/shared", + "link": true + }, + "node_modules/@flowmemory/verifier-v0": { + "resolved": "services/verifier", + "link": true + }, + "services/indexer": { + "name": "@flowmemory/indexer-v0" + }, + "services/shared": { + "name": "@flowmemory/indexer-verifier-foundation" + }, + "services/verifier": { + "name": "@flowmemory/verifier-v0" + } + } +} diff --git a/package.json b/package.json new file mode 100644 index 00000000..ea644bec --- /dev/null +++ b/package.json @@ -0,0 +1,18 @@ +{ + "name": "flowmemory-services-v0", + "private": true, + "type": "module", + "workspaces": [ + "services/shared", + "services/indexer", + "services/verifier" + ], + "scripts": { + "test": "npm test --prefix services/shared && npm test --prefix services/indexer && npm test --prefix services/verifier", + "index:fixtures": "npm run index:fixtures --prefix services/indexer", + "verify:fixtures": "npm run verify:fixtures --prefix services/verifier", + "e2e": "npm run index:fixtures && npm run verify:fixtures", + "demo:indexer": "npm run demo --prefix services/indexer", + "demo:verifier": "npm run demo --prefix services/verifier" + } +} diff --git a/services/indexer/OBSERVATION_IDENTITY.md b/services/indexer/OBSERVATION_IDENTITY.md new file mode 100644 index 00000000..52b54aa0 --- /dev/null +++ b/services/indexer/OBSERVATION_IDENTITY.md @@ -0,0 +1,137 @@ +# FlowPulse Observation Identity + +This is the indexer-facing identity model for FlowPulse V0. + +## Identity Layers + +- `pulseId`: emitted by the FlowPulse contract. +- `observationId`: derived by the indexer from receipt/log location after execution. +- `cursorId`: derived by the indexer for source-set scan progress. +- `reportId`: derived by the verifier from canonical report content. + +## Required Fields + +Decoded from FlowPulse: + +- `pulseId` +- `rootfieldId` +- `actor` +- `pulseType` +- `subject` +- `commitment` +- `parentPulseId` +- `sequence` +- `occurredAt` +- `uri` + +Attached from receipt/log context: + +- `chainId` +- `emittingContract` +- `eventSignature` +- `blockNumber` +- `blockHash` +- `txHash` +- `transactionIndex` +- `logIndex` +- `receiptStatus` + +Receipt/log fields are derived after execution. Contracts and hooks do not know them. + +## Observation ID + +`observationId` identifies a specific observed FlowPulse log location: + +```text +keccak256(abi.encode( + "flowmemory.flowpulse.observation.v0", + chainId, + sourceContract, + txHash, + logIndex +)) +``` + +Canonicalization before hashing: + +- `sourceContract` is the normalized emitting contract address. +- `txHash` is a normalized 32-byte transaction hash. +- `logIndex` is a nonnegative integer. +- `chainId` is serialized as a uint256 ABI value. +- Hash function is EVM Keccak-256. + +`blockHash`, `blockNumber`, `transactionIndex`, `eventSignature`, and decoded FlowPulse fields are stored with the observation for audit and reorg handling, but they are not part of the V0 observation-id preimage. + +## Cursor ID + +`cursorId` identifies scan progress for a configured source set: + +```text +keccak256(abi.encode( + "flowmemory.indexer.cursor.v0", + chainId, + sourceSetId, + blockNumber, + blockHash, + transactionIndex, + logIndex +)) +``` + +`sourceSetId` is: + +```text +keccak256("flowmemory.indexer.source_set.v0||") +``` + +Cursor identity includes `blockHash` because scan progress must be able to detect replays over a changed canonical chain. + +## State Model + +- `observed`: decoded from a successful receipt/log with no finality policy applied. +- `pending`: decoded but above the configured fixture finality threshold. +- `finalized`: at or below the configured fixture finality threshold. +- `removed`: provider fixture marks the log removed. +- `superseded`: older observation for the same `pulseId` is replaced by another observation. +- `reorged`: canonical block-hash check says the indexed block is no longer canonical. + +This model is enough for fixtures and tests. It is not production reorg handling. + +## Duplicate Rules + +Exact duplicate: + +- Same `observationId`. +- Same canonical observation JSON. +- Safe to treat as idempotent replay. + +Conflicting duplicate: + +- Same `observationId`. +- Required metadata or decoded fields differ. +- Treat as an indexer integrity failure. + +Pulse duplicate: + +- Same contract-emitted `pulseId`. +- Different `observationId`. +- Preserve both observations for verifier/operator policy. + +Reorg replacement: + +- Same `pulseId`. +- Different block/log location. +- Previous observation may become `superseded` or `reorged` depending on canonicality evidence. + +Unsupported observation: + +- FlowPulse is decoded but current verifier rules do not support the pulse type or artifact semantics. +- Preserve the observation and let verifier status become `unsupported`. + +## Non-Goals + +- No production live RPC service. +- No production database schema. +- No production reorg worker. +- No tokenomics or verifier economics. +- No secrets in env files. diff --git a/services/indexer/README.md b/services/indexer/README.md new file mode 100644 index 00000000..88ec31fa --- /dev/null +++ b/services/indexer/README.md @@ -0,0 +1,167 @@ +# FlowMemory Indexer V0 + +This package is a local, fixture-first FlowPulse indexer. It decodes sample receipts/logs, derives deterministic observation and cursor identities, models basic lifecycle states, and writes canonical JSON. It is not a production indexer. + +## Commands + +From the repository root: + +```powershell +npm run index:fixtures +npm run demo:indexer +npm test --prefix services/indexer +``` + +`npm run index:fixtures` writes: + +```text +services/indexer/out/indexer-state.json +``` + +Use a custom output path: + +```powershell +npm run index:fixtures -- --out out/custom-state.json +``` + +## Fixtures + +Primary receipt fixtures: + +```text +services/indexer/fixtures/flowpulse-receipts.json +``` + +The fixture set covers: + +- valid rootfield registration +- valid root commit +- duplicate observation +- removed/reorg-style log +- invalid commitment input +- unresolved artifact input +- unsupported pulse type +- reverted receipt +- malformed log + +Legacy single-log fixture: + +```text +services/indexer/fixtures/flowpulse-logs.json +``` + +## Decoder + +The decoder accepts FlowPulse v0 logs with: + +```text +FlowPulse(bytes32,bytes32,address,uint8,bytes32,bytes32,bytes32,uint64,uint64,string) +``` + +`topic0` must equal: + +```text +0x5d07190b9ae441b4d7b16259a48424acd451492b12f5f99a29f5bfd992c13e43 +``` + +Decoded indexed topics: + +- `pulseId` +- `rootfieldId` +- `actor` + +Decoded data fields: + +- `pulseType` +- `subject` +- `commitment` +- `parentPulseId` +- `sequence` +- `occurredAt` +- `uri` + +Malformed logs are rejected into `rejectedLogs` with deterministic reason codes. + +## Observation Identity + +The contract emits `pulseId`. The indexer derives `observationId` only after receipt/log metadata exists: + +```text +keccak256(abi.encode( + "flowmemory.flowpulse.observation.v0", + chainId, + sourceContract, + txHash, + logIndex +)) +``` + +`txHash` and `logIndex` are receipt/log-derived. They are not known by contracts or hooks during execution. + +## Cursor Identity + +The indexer derives `cursorId` for scan progress: + +```text +keccak256(abi.encode( + "flowmemory.indexer.cursor.v0", + chainId, + sourceSetId, + blockNumber, + blockHash, + transactionIndex, + logIndex +)) +``` + +The `sourceSetId` is deterministic over chain id and the normalized emitting-contract set. + +## Lifecycle States + +V0 state values: + +- `observed` +- `pending` +- `finalized` +- `removed` +- `superseded` +- `reorged` + +The fixture state model supports a finality threshold and block-hash mismatch checks. It does not claim production reorg handling. + +## Duplicate Handling + +- `exactDuplicate`: same `observationId` and canonical observation JSON. +- `conflictingDuplicate`: same `observationId` with different canonical content. +- `pulseDuplicate`: same contract `pulseId` at a different observation location. +- `reorgReplacement`: same `pulseId` with changed block/log location. + +Exact duplicates are idempotent. Conflicting duplicates are an indexer integrity risk. Pulse duplicates and reorg replacements stay visible for verifier/operator policy. + +## Persistence + +The persisted file wraps indexer state with: + +```text +flowmemory.indexer.persistence.v0 +``` + +The state itself declares: + +```text +flowmemory.indexer.state.v0 +``` + +JSON output is deterministic and contains observations, cursors, batches, rootfields, pulses, rejected logs, and duplicate records. + +The JSON schema fixture lives at: + +```text +services/indexer/fixtures/indexer-state.schema.json +``` + +## Local RPC Boundary + +`readLocalRpcFlowPulseLogs` maps explicit JSON-RPC responses into the same raw fixture shape. It has no default RPC URL, no env file, no secrets, and tests use mocked fetch responses. Future live RPC indexing should be handled by a separate scoped issue. + +See [docs/INDEXER_VERIFIER_MVP.md](../../docs/INDEXER_VERIFIER_MVP.md) for the full pipeline. diff --git a/services/indexer/fixtures/flowpulse-logs.json b/services/indexer/fixtures/flowpulse-logs.json new file mode 100644 index 00000000..44bc290b --- /dev/null +++ b/services/indexer/fixtures/flowpulse-logs.json @@ -0,0 +1,21 @@ +{ + "logs": [ + { + "chainId": "8453", + "address": "0x1111111111111111111111111111111111111111", + "topics": [ + "0x5d07190b9ae441b4d7b16259a48424acd451492b12f5f99a29f5bfd992c13e43", + "0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "0x0000000000000000000000004444444444444444444444444444444444444444" + ], + "data": "0x0000000000000000000000000000000000000000000000000000000000000001bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb4122209ff672fc04b2ec3af31ab1af79813971f86de000aa6534038cc79de6b500000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000006a03e48000000000000000000000000000000000000000000000000000000000000000e0000000000000000000000000000000000000000000000000000000000000001e697066733a2f2f626166792d666c6f776d656d6f72792d6578616d706c650000", + "blockNumber": "123456", + "blockHash": "0x2222222222222222222222222222222222222222222222222222222222222222", + "transactionHash": "0x3333333333333333333333333333333333333333333333333333333333333333", + "transactionIndex": "7", + "logIndex": "2", + "receiptStatus": "success" + } + ] +} diff --git a/services/indexer/fixtures/flowpulse-receipts.json b/services/indexer/fixtures/flowpulse-receipts.json new file mode 100644 index 00000000..185d25c1 --- /dev/null +++ b/services/indexer/fixtures/flowpulse-receipts.json @@ -0,0 +1,205 @@ +{ + "schema": "flowmemory.indexer.receiptFixtures.v0", + "description": "Synthetic receipts for local FlowPulse indexer/verifier V0 tests. These are not live chain receipts.", + "receipts": [ + { + "name": "rootfield-registration-valid", + "chainId": "8453", + "blockNumber": "123456", + "blockHash": "0x2222222222222222222222222222222222222222222222222222222222222222", + "transactionHash": "0x3333333333333333333333333333333333333333333333333333333333333333", + "transactionIndex": "7", + "status": "success", + "logs": [ + { + "address": "0x1111111111111111111111111111111111111111", + "topics": [ + "0x5d07190b9ae441b4d7b16259a48424acd451492b12f5f99a29f5bfd992c13e43", + "0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "0x0000000000000000000000004444444444444444444444444444444444444444" + ], + "data": "0x0000000000000000000000000000000000000000000000000000000000000001bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb4122209ff672fc04b2ec3af31ab1af79813971f86de000aa6534038cc79de6b500000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000006a03e48000000000000000000000000000000000000000000000000000000000000000e0000000000000000000000000000000000000000000000000000000000000001e697066733a2f2f626166792d666c6f776d656d6f72792d6578616d706c650000", + "logIndex": "2" + } + ] + }, + { + "name": "root-commit-valid", + "chainId": "8453", + "blockNumber": "123457", + "blockHash": "0x2222222222222222222222222222222222222222222222222222222222222223", + "transactionHash": "0x3333333333333333333333333333333333333333333333333333333333333334", + "transactionIndex": "8", + "status": "success", + "logs": [ + { + "address": "0x1111111111111111111111111111111111111111", + "topics": [ + "0x5d07190b9ae441b4d7b16259a48424acd451492b12f5f99a29f5bfd992c13e43", + "0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab001", + "0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "0x0000000000000000000000004444444444444444444444444444444444444444" + ], + "data": "0x0000000000000000000000000000000000000000000000000000000000000002fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff169a9f953179a6ce2e08724dd759fb71b08b0759a5900fc0399a753ecf0557df5aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa0000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000006a03e4bc00000000000000000000000000000000000000000000000000000000000000e0000000000000000000000000000000000000000000000000000000000000001b666978747572653a2f2f726f6f742d636f6d6d69742d76616c69640000000000", + "logIndex": "3" + } + ] + }, + { + "name": "duplicate-registration", + "chainId": "8453", + "blockNumber": "123456", + "blockHash": "0x2222222222222222222222222222222222222222222222222222222222222222", + "transactionHash": "0x3333333333333333333333333333333333333333333333333333333333333333", + "transactionIndex": "7", + "status": "success", + "logs": [ + { + "address": "0x1111111111111111111111111111111111111111", + "topics": [ + "0x5d07190b9ae441b4d7b16259a48424acd451492b12f5f99a29f5bfd992c13e43", + "0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "0x0000000000000000000000004444444444444444444444444444444444444444" + ], + "data": "0x0000000000000000000000000000000000000000000000000000000000000001bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb4122209ff672fc04b2ec3af31ab1af79813971f86de000aa6534038cc79de6b500000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000006a03e48000000000000000000000000000000000000000000000000000000000000000e0000000000000000000000000000000000000000000000000000000000000001e697066733a2f2f626166792d666c6f776d656d6f72792d6578616d706c650000", + "logIndex": "2" + } + ] + }, + { + "name": "removed-root-commit", + "chainId": "8453", + "blockNumber": "123458", + "blockHash": "0x2222222222222222222222222222222222222222222222222222222222222224", + "transactionHash": "0x3333333333333333333333333333333333333333333333333333333333333335", + "transactionIndex": "9", + "status": "success", + "logs": [ + { + "address": "0x1111111111111111111111111111111111111111", + "topics": [ + "0x5d07190b9ae441b4d7b16259a48424acd451492b12f5f99a29f5bfd992c13e43", + "0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab002", + "0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "0x0000000000000000000000004444444444444444444444444444444444444444" + ], + "data": "0x0000000000000000000000000000000000000000000000000000000000000002fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff169a9f953179a6ce2e08724dd759fb71b08b0759a5900fc0399a753ecf0557df5aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa0000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000000000000000006a03e4bc00000000000000000000000000000000000000000000000000000000000000e0000000000000000000000000000000000000000000000000000000000000001b666978747572653a2f2f726f6f742d636f6d6d69742d76616c69640000000000", + "logIndex": "4", + "removed": true + } + ] + }, + { + "name": "root-commit-invalid-commitment", + "chainId": "8453", + "blockNumber": "123459", + "blockHash": "0x2222222222222222222222222222222222222222222222222222222222222225", + "transactionHash": "0x3333333333333333333333333333333333333333333333333333333333333336", + "transactionIndex": "10", + "status": "success", + "logs": [ + { + "address": "0x1111111111111111111111111111111111111111", + "topics": [ + "0x5d07190b9ae441b4d7b16259a48424acd451492b12f5f99a29f5bfd992c13e43", + "0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab003", + "0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "0x0000000000000000000000004444444444444444444444444444444444444444" + ], + "data": "0x0000000000000000000000000000000000000000000000000000000000000002fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff1ccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa0000000000000000000000000000000000000000000000000000000000000003000000000000000000000000000000000000000000000000000000006a03e4f800000000000000000000000000000000000000000000000000000000000000e0000000000000000000000000000000000000000000000000000000000000001b666978747572653a2f2f726f6f742d636f6d6d69742d76616c69640000000000", + "logIndex": "5" + } + ] + }, + { + "name": "root-commit-missing-evidence", + "chainId": "8453", + "blockNumber": "123460", + "blockHash": "0x2222222222222222222222222222222222222222222222222222222222222226", + "transactionHash": "0x3333333333333333333333333333333333333333333333333333333333333337", + "transactionIndex": "11", + "status": "success", + "logs": [ + { + "address": "0x1111111111111111111111111111111111111111", + "topics": [ + "0x5d07190b9ae441b4d7b16259a48424acd451492b12f5f99a29f5bfd992c13e43", + "0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab004", + "0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "0x0000000000000000000000004444444444444444444444444444444444444444" + ], + "data": "0x0000000000000000000000000000000000000000000000000000000000000002fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff169a9f953179a6ce2e08724dd759fb71b08b0759a5900fc0399a753ecf0557df5aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa0000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000006a03e53400000000000000000000000000000000000000000000000000000000000000e0000000000000000000000000000000000000000000000000000000000000001a666978747572653a2f2f6d697373696e672d6172746966616374000000000000", + "logIndex": "6" + } + ] + }, + { + "name": "unsupported-pulse-type", + "chainId": "8453", + "blockNumber": "123461", + "blockHash": "0x2222222222222222222222222222222222222222222222222222222222222227", + "transactionHash": "0x3333333333333333333333333333333333333333333333333333333333333338", + "transactionIndex": "12", + "status": "success", + "logs": [ + { + "address": "0x1111111111111111111111111111111111111111", + "topics": [ + "0x5d07190b9ae441b4d7b16259a48424acd451492b12f5f99a29f5bfd992c13e43", + "0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab005", + "0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "0x0000000000000000000000004444444444444444444444444444444444444444" + ], + "data": "0x0000000000000000000000000000000000000000000000000000000000000063fffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff169a9f953179a6ce2e08724dd759fb71b08b0759a5900fc0399a753ecf0557df5aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa0000000000000000000000000000000000000000000000000000000000000005000000000000000000000000000000000000000000000000000000006a03e57000000000000000000000000000000000000000000000000000000000000000e00000000000000000000000000000000000000000000000000000000000000015666978747572653a2f2f756e737570706f727465640000000000000000000000", + "logIndex": "7" + } + ] + }, + { + "name": "failed-receipt", + "chainId": "8453", + "blockNumber": "123462", + "blockHash": "0x2222222222222222222222222222222222222222222222222222222222222228", + "transactionHash": "0x3333333333333333333333333333333333333333333333333333333333333339", + "transactionIndex": "13", + "status": "reverted", + "logs": [ + { + "address": "0x1111111111111111111111111111111111111111", + "topics": [ + "0x5d07190b9ae441b4d7b16259a48424acd451492b12f5f99a29f5bfd992c13e43", + "0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab006", + "0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "0x0000000000000000000000004444444444444444444444444444444444444444" + ], + "data": "0x0000000000000000000000000000000000000000000000000000000000000001bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb4122209ff672fc04b2ec3af31ab1af79813971f86de000aa6534038cc79de6b500000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000006a03e48000000000000000000000000000000000000000000000000000000000000000e0000000000000000000000000000000000000000000000000000000000000001e697066733a2f2f626166792d666c6f776d656d6f72792d6578616d706c650000", + "logIndex": "8" + } + ] + }, + { + "name": "malformed-log", + "chainId": "8453", + "blockNumber": "123463", + "blockHash": "0x2222222222222222222222222222222222222222222222222222222222222229", + "transactionHash": "0x3333333333333333333333333333333333333333333333333333333333333340", + "transactionIndex": "14", + "status": "success", + "logs": [ + { + "address": "0x1111111111111111111111111111111111111111", + "topics": [ + "0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff", + "0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab007", + "0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "0x0000000000000000000000004444444444444444444444444444444444444444" + ], + "data": "0x00", + "logIndex": "9" + } + ] + } + ] +} diff --git a/services/indexer/fixtures/indexer-state.schema.json b/services/indexer/fixtures/indexer-state.schema.json new file mode 100644 index 00000000..ee885d1b --- /dev/null +++ b/services/indexer/fixtures/indexer-state.schema.json @@ -0,0 +1,36 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "flowmemory.indexer.persistence.v0", + "type": "object", + "required": ["schema", "state"], + "properties": { + "schema": { "const": "flowmemory.indexer.persistence.v0" }, + "state": { + "type": "object", + "required": [ + "schema", + "source", + "observations", + "pulses", + "rootfields", + "cursors", + "batches", + "rejectedLogs", + "duplicates" + ], + "properties": { + "schema": { "const": "flowmemory.indexer.state.v0" }, + "source": { "enum": ["fixture", "local-rpc-placeholder"] }, + "observations": { "type": "array" }, + "pulses": { "type": "array" }, + "rootfields": { "type": "array" }, + "cursors": { "type": "array" }, + "batches": { "type": "array" }, + "rejectedLogs": { "type": "array" }, + "duplicates": { "type": "array" } + }, + "additionalProperties": false + } + }, + "additionalProperties": false +} diff --git a/services/indexer/out/indexer-state.json b/services/indexer/out/indexer-state.json new file mode 100644 index 00000000..001c723c --- /dev/null +++ b/services/indexer/out/indexer-state.json @@ -0,0 +1 @@ +{"schema":"flowmemory.indexer.persistence.v0","state":{"batches":[{"cursorCount":6,"observationCount":7,"rejectedLogCount":2,"schema":"flowmemory.indexer.batch.v0","source":"fixture","sourceSetId":"0x95b6a52c8fbf5a0f68120c877461f0e7f75dc191c79a03e7510078f14a55dfe4"}],"cursors":[{"blockHash":"0x2222222222222222222222222222222222222222222222222222222222222222","blockNumber":"123456","chainId":"8453","cursorId":"0x668270c7410fea7c1581c9f4e8b9ba5a5d3eeebb7b5b75ee3a602b096245a21f","logIndex":"2","sourceSetId":"0x95b6a52c8fbf5a0f68120c877461f0e7f75dc191c79a03e7510078f14a55dfe4","transactionIndex":"7"},{"blockHash":"0x2222222222222222222222222222222222222222222222222222222222222224","blockNumber":"123458","chainId":"8453","cursorId":"0x69fddcac0f6148d6193cda1a3bdacde7c15a5bf4589831dfcfc0c90062bd05b4","logIndex":"4","sourceSetId":"0x95b6a52c8fbf5a0f68120c877461f0e7f75dc191c79a03e7510078f14a55dfe4","transactionIndex":"9"},{"blockHash":"0x2222222222222222222222222222222222222222222222222222222222222225","blockNumber":"123459","chainId":"8453","cursorId":"0x9bdea7fca6fd5954e75608037b1b8bb92a2312cd58a10573b40f0312a29285f8","logIndex":"5","sourceSetId":"0x95b6a52c8fbf5a0f68120c877461f0e7f75dc191c79a03e7510078f14a55dfe4","transactionIndex":"10"},{"blockHash":"0x2222222222222222222222222222222222222222222222222222222222222226","blockNumber":"123460","chainId":"8453","cursorId":"0xc7ef88c914177a735c6757f6e85412a8bbcdad6404a4165a458a1a92eafc4686","logIndex":"6","sourceSetId":"0x95b6a52c8fbf5a0f68120c877461f0e7f75dc191c79a03e7510078f14a55dfe4","transactionIndex":"11"},{"blockHash":"0x2222222222222222222222222222222222222222222222222222222222222227","blockNumber":"123461","chainId":"8453","cursorId":"0xe7071042f8c40f0f7a698a510749ff76bc65c683dcf4914eff7e98d9df5974f9","logIndex":"7","sourceSetId":"0x95b6a52c8fbf5a0f68120c877461f0e7f75dc191c79a03e7510078f14a55dfe4","transactionIndex":"12"},{"blockHash":"0x2222222222222222222222222222222222222222222222222222222222222223","blockNumber":"123457","chainId":"8453","cursorId":"0xf5cd45f8e8fb8d3bdedcfe527bcb21f40d01591729e244bdf263f53f1498169f","logIndex":"3","sourceSetId":"0x95b6a52c8fbf5a0f68120c877461f0e7f75dc191c79a03e7510078f14a55dfe4","transactionIndex":"8"}],"duplicates":[{"kind":"exactDuplicate","observationId":"0x9d958aadf8bf46f989b51e541709a73d21970e7e79643f939c9a0000b50f9a91","pulseId":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"}],"observations":[{"actor":"0x4444444444444444444444444444444444444444","blockHash":"0x2222222222222222222222222222222222222222222222222222222222222222","blockNumber":"123456","canonicalObservationJson":"{\"actor\":\"0x4444444444444444444444444444444444444444\",\"blockHash\":\"0x2222222222222222222222222222222222222222222222222222222222222222\",\"blockNumber\":\"123456\",\"chainId\":\"8453\",\"commitment\":\"0x4122209ff672fc04b2ec3af31ab1af79813971f86de000aa6534038cc79de6b5\",\"cursorId\":\"0x668270c7410fea7c1581c9f4e8b9ba5a5d3eeebb7b5b75ee3a602b096245a21f\",\"emittingContract\":\"0x1111111111111111111111111111111111111111\",\"eventSignature\":\"0x5d07190b9ae441b4d7b16259a48424acd451492b12f5f99a29f5bfd992c13e43\",\"lifecycleState\":\"finalized\",\"logIndex\":\"2\",\"observationId\":\"0x9d958aadf8bf46f989b51e541709a73d21970e7e79643f939c9a0000b50f9a91\",\"occurredAt\":\"1778640000\",\"parentPulseId\":\"0x0000000000000000000000000000000000000000000000000000000000000000\",\"pulseId\":\"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\",\"pulseType\":\"1\",\"receiptStatus\":\"success\",\"rootfieldId\":\"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\",\"sequence\":\"1\",\"subject\":\"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\",\"transactionIndex\":\"7\",\"txHash\":\"0x3333333333333333333333333333333333333333333333333333333333333333\",\"uri\":\"ipfs://bafy-flowmemory-example\"}","chainId":"8453","commitment":"0x4122209ff672fc04b2ec3af31ab1af79813971f86de000aa6534038cc79de6b5","cursorId":"0x668270c7410fea7c1581c9f4e8b9ba5a5d3eeebb7b5b75ee3a602b096245a21f","duplicateKind":"unique","emittingContract":"0x1111111111111111111111111111111111111111","eventSignature":"0x5d07190b9ae441b4d7b16259a48424acd451492b12f5f99a29f5bfd992c13e43","lifecycleState":"finalized","logIndex":"2","observationId":"0x9d958aadf8bf46f989b51e541709a73d21970e7e79643f939c9a0000b50f9a91","occurredAt":"1778640000","parentPulseId":"0x0000000000000000000000000000000000000000000000000000000000000000","pulseId":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","pulseType":"1","receiptStatus":"success","rootfieldId":"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb","sequence":"1","subject":"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb","transactionIndex":"7","txHash":"0x3333333333333333333333333333333333333333333333333333333333333333","uri":"ipfs://bafy-flowmemory-example"},{"actor":"0x4444444444444444444444444444444444444444","blockHash":"0x2222222222222222222222222222222222222222222222222222222222222223","blockNumber":"123457","canonicalObservationJson":"{\"actor\":\"0x4444444444444444444444444444444444444444\",\"blockHash\":\"0x2222222222222222222222222222222222222222222222222222222222222223\",\"blockNumber\":\"123457\",\"chainId\":\"8453\",\"commitment\":\"0x69a9f953179a6ce2e08724dd759fb71b08b0759a5900fc0399a753ecf0557df5\",\"cursorId\":\"0xf5cd45f8e8fb8d3bdedcfe527bcb21f40d01591729e244bdf263f53f1498169f\",\"emittingContract\":\"0x1111111111111111111111111111111111111111\",\"eventSignature\":\"0x5d07190b9ae441b4d7b16259a48424acd451492b12f5f99a29f5bfd992c13e43\",\"lifecycleState\":\"finalized\",\"logIndex\":\"3\",\"observationId\":\"0x49c6cd59d1f1916bc5301308be09c14d64127d1566362272dc5aa201ed53bdea\",\"occurredAt\":\"1778640060\",\"parentPulseId\":\"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\",\"pulseId\":\"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab001\",\"pulseType\":\"2\",\"receiptStatus\":\"success\",\"rootfieldId\":\"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\",\"sequence\":\"2\",\"subject\":\"0xfffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff1\",\"transactionIndex\":\"8\",\"txHash\":\"0x3333333333333333333333333333333333333333333333333333333333333334\",\"uri\":\"fixture://root-commit-valid\"}","chainId":"8453","commitment":"0x69a9f953179a6ce2e08724dd759fb71b08b0759a5900fc0399a753ecf0557df5","cursorId":"0xf5cd45f8e8fb8d3bdedcfe527bcb21f40d01591729e244bdf263f53f1498169f","duplicateKind":"unique","emittingContract":"0x1111111111111111111111111111111111111111","eventSignature":"0x5d07190b9ae441b4d7b16259a48424acd451492b12f5f99a29f5bfd992c13e43","lifecycleState":"finalized","logIndex":"3","observationId":"0x49c6cd59d1f1916bc5301308be09c14d64127d1566362272dc5aa201ed53bdea","occurredAt":"1778640060","parentPulseId":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","pulseId":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab001","pulseType":"2","receiptStatus":"success","rootfieldId":"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb","sequence":"2","subject":"0xfffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff1","transactionIndex":"8","txHash":"0x3333333333333333333333333333333333333333333333333333333333333334","uri":"fixture://root-commit-valid"},{"actor":"0x4444444444444444444444444444444444444444","blockHash":"0x2222222222222222222222222222222222222222222222222222222222222222","blockNumber":"123456","canonicalObservationJson":"{\"actor\":\"0x4444444444444444444444444444444444444444\",\"blockHash\":\"0x2222222222222222222222222222222222222222222222222222222222222222\",\"blockNumber\":\"123456\",\"chainId\":\"8453\",\"commitment\":\"0x4122209ff672fc04b2ec3af31ab1af79813971f86de000aa6534038cc79de6b5\",\"cursorId\":\"0x668270c7410fea7c1581c9f4e8b9ba5a5d3eeebb7b5b75ee3a602b096245a21f\",\"emittingContract\":\"0x1111111111111111111111111111111111111111\",\"eventSignature\":\"0x5d07190b9ae441b4d7b16259a48424acd451492b12f5f99a29f5bfd992c13e43\",\"lifecycleState\":\"finalized\",\"logIndex\":\"2\",\"observationId\":\"0x9d958aadf8bf46f989b51e541709a73d21970e7e79643f939c9a0000b50f9a91\",\"occurredAt\":\"1778640000\",\"parentPulseId\":\"0x0000000000000000000000000000000000000000000000000000000000000000\",\"pulseId\":\"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\",\"pulseType\":\"1\",\"receiptStatus\":\"success\",\"rootfieldId\":\"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\",\"sequence\":\"1\",\"subject\":\"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\",\"transactionIndex\":\"7\",\"txHash\":\"0x3333333333333333333333333333333333333333333333333333333333333333\",\"uri\":\"ipfs://bafy-flowmemory-example\"}","chainId":"8453","commitment":"0x4122209ff672fc04b2ec3af31ab1af79813971f86de000aa6534038cc79de6b5","cursorId":"0x668270c7410fea7c1581c9f4e8b9ba5a5d3eeebb7b5b75ee3a602b096245a21f","duplicateKind":"exactDuplicate","emittingContract":"0x1111111111111111111111111111111111111111","eventSignature":"0x5d07190b9ae441b4d7b16259a48424acd451492b12f5f99a29f5bfd992c13e43","lifecycleState":"finalized","logIndex":"2","observationId":"0x9d958aadf8bf46f989b51e541709a73d21970e7e79643f939c9a0000b50f9a91","occurredAt":"1778640000","parentPulseId":"0x0000000000000000000000000000000000000000000000000000000000000000","pulseId":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","pulseType":"1","receiptStatus":"success","rootfieldId":"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb","sequence":"1","subject":"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb","transactionIndex":"7","txHash":"0x3333333333333333333333333333333333333333333333333333333333333333","uri":"ipfs://bafy-flowmemory-example"},{"actor":"0x4444444444444444444444444444444444444444","blockHash":"0x2222222222222222222222222222222222222222222222222222222222222224","blockNumber":"123458","canonicalObservationJson":"{\"actor\":\"0x4444444444444444444444444444444444444444\",\"blockHash\":\"0x2222222222222222222222222222222222222222222222222222222222222224\",\"blockNumber\":\"123458\",\"chainId\":\"8453\",\"commitment\":\"0x69a9f953179a6ce2e08724dd759fb71b08b0759a5900fc0399a753ecf0557df5\",\"cursorId\":\"0x69fddcac0f6148d6193cda1a3bdacde7c15a5bf4589831dfcfc0c90062bd05b4\",\"emittingContract\":\"0x1111111111111111111111111111111111111111\",\"eventSignature\":\"0x5d07190b9ae441b4d7b16259a48424acd451492b12f5f99a29f5bfd992c13e43\",\"lifecycleState\":\"removed\",\"logIndex\":\"4\",\"observationId\":\"0x223d74c971f301d800dd69aa30994bc3fa3089b34b0db1b0a7fc9d4e8d114c79\",\"occurredAt\":\"1778640060\",\"parentPulseId\":\"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\",\"pulseId\":\"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab002\",\"pulseType\":\"2\",\"receiptStatus\":\"success\",\"rootfieldId\":\"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\",\"sequence\":\"2\",\"subject\":\"0xfffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff1\",\"transactionIndex\":\"9\",\"txHash\":\"0x3333333333333333333333333333333333333333333333333333333333333335\",\"uri\":\"fixture://root-commit-valid\"}","chainId":"8453","commitment":"0x69a9f953179a6ce2e08724dd759fb71b08b0759a5900fc0399a753ecf0557df5","cursorId":"0x69fddcac0f6148d6193cda1a3bdacde7c15a5bf4589831dfcfc0c90062bd05b4","duplicateKind":"unique","emittingContract":"0x1111111111111111111111111111111111111111","eventSignature":"0x5d07190b9ae441b4d7b16259a48424acd451492b12f5f99a29f5bfd992c13e43","lifecycleState":"removed","logIndex":"4","observationId":"0x223d74c971f301d800dd69aa30994bc3fa3089b34b0db1b0a7fc9d4e8d114c79","occurredAt":"1778640060","parentPulseId":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","pulseId":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab002","pulseType":"2","receiptStatus":"success","rootfieldId":"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb","sequence":"2","subject":"0xfffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff1","transactionIndex":"9","txHash":"0x3333333333333333333333333333333333333333333333333333333333333335","uri":"fixture://root-commit-valid"},{"actor":"0x4444444444444444444444444444444444444444","blockHash":"0x2222222222222222222222222222222222222222222222222222222222222225","blockNumber":"123459","canonicalObservationJson":"{\"actor\":\"0x4444444444444444444444444444444444444444\",\"blockHash\":\"0x2222222222222222222222222222222222222222222222222222222222222225\",\"blockNumber\":\"123459\",\"chainId\":\"8453\",\"commitment\":\"0xcccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc\",\"cursorId\":\"0x9bdea7fca6fd5954e75608037b1b8bb92a2312cd58a10573b40f0312a29285f8\",\"emittingContract\":\"0x1111111111111111111111111111111111111111\",\"eventSignature\":\"0x5d07190b9ae441b4d7b16259a48424acd451492b12f5f99a29f5bfd992c13e43\",\"lifecycleState\":\"pending\",\"logIndex\":\"5\",\"observationId\":\"0xe4a7065f1578c4a232b41f5984942e9b7b07760ce7dfeba77b38eb03173491df\",\"occurredAt\":\"1778640120\",\"parentPulseId\":\"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\",\"pulseId\":\"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab003\",\"pulseType\":\"2\",\"receiptStatus\":\"success\",\"rootfieldId\":\"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\",\"sequence\":\"3\",\"subject\":\"0xfffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff1\",\"transactionIndex\":\"10\",\"txHash\":\"0x3333333333333333333333333333333333333333333333333333333333333336\",\"uri\":\"fixture://root-commit-valid\"}","chainId":"8453","commitment":"0xcccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc","cursorId":"0x9bdea7fca6fd5954e75608037b1b8bb92a2312cd58a10573b40f0312a29285f8","duplicateKind":"unique","emittingContract":"0x1111111111111111111111111111111111111111","eventSignature":"0x5d07190b9ae441b4d7b16259a48424acd451492b12f5f99a29f5bfd992c13e43","lifecycleState":"pending","logIndex":"5","observationId":"0xe4a7065f1578c4a232b41f5984942e9b7b07760ce7dfeba77b38eb03173491df","occurredAt":"1778640120","parentPulseId":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","pulseId":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab003","pulseType":"2","receiptStatus":"success","rootfieldId":"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb","sequence":"3","subject":"0xfffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff1","transactionIndex":"10","txHash":"0x3333333333333333333333333333333333333333333333333333333333333336","uri":"fixture://root-commit-valid"},{"actor":"0x4444444444444444444444444444444444444444","blockHash":"0x2222222222222222222222222222222222222222222222222222222222222226","blockNumber":"123460","canonicalObservationJson":"{\"actor\":\"0x4444444444444444444444444444444444444444\",\"blockHash\":\"0x2222222222222222222222222222222222222222222222222222222222222226\",\"blockNumber\":\"123460\",\"chainId\":\"8453\",\"commitment\":\"0x69a9f953179a6ce2e08724dd759fb71b08b0759a5900fc0399a753ecf0557df5\",\"cursorId\":\"0xc7ef88c914177a735c6757f6e85412a8bbcdad6404a4165a458a1a92eafc4686\",\"emittingContract\":\"0x1111111111111111111111111111111111111111\",\"eventSignature\":\"0x5d07190b9ae441b4d7b16259a48424acd451492b12f5f99a29f5bfd992c13e43\",\"lifecycleState\":\"pending\",\"logIndex\":\"6\",\"observationId\":\"0x92144fc24c81cdd6319598e6e0c58d84e3f18d9ad4a8be33bc956e32c2ae39f4\",\"occurredAt\":\"1778640180\",\"parentPulseId\":\"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\",\"pulseId\":\"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab004\",\"pulseType\":\"2\",\"receiptStatus\":\"success\",\"rootfieldId\":\"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\",\"sequence\":\"4\",\"subject\":\"0xfffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff1\",\"transactionIndex\":\"11\",\"txHash\":\"0x3333333333333333333333333333333333333333333333333333333333333337\",\"uri\":\"fixture://missing-artifact\"}","chainId":"8453","commitment":"0x69a9f953179a6ce2e08724dd759fb71b08b0759a5900fc0399a753ecf0557df5","cursorId":"0xc7ef88c914177a735c6757f6e85412a8bbcdad6404a4165a458a1a92eafc4686","duplicateKind":"unique","emittingContract":"0x1111111111111111111111111111111111111111","eventSignature":"0x5d07190b9ae441b4d7b16259a48424acd451492b12f5f99a29f5bfd992c13e43","lifecycleState":"pending","logIndex":"6","observationId":"0x92144fc24c81cdd6319598e6e0c58d84e3f18d9ad4a8be33bc956e32c2ae39f4","occurredAt":"1778640180","parentPulseId":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","pulseId":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab004","pulseType":"2","receiptStatus":"success","rootfieldId":"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb","sequence":"4","subject":"0xfffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff1","transactionIndex":"11","txHash":"0x3333333333333333333333333333333333333333333333333333333333333337","uri":"fixture://missing-artifact"},{"actor":"0x4444444444444444444444444444444444444444","blockHash":"0x2222222222222222222222222222222222222222222222222222222222222227","blockNumber":"123461","canonicalObservationJson":"{\"actor\":\"0x4444444444444444444444444444444444444444\",\"blockHash\":\"0x2222222222222222222222222222222222222222222222222222222222222227\",\"blockNumber\":\"123461\",\"chainId\":\"8453\",\"commitment\":\"0x69a9f953179a6ce2e08724dd759fb71b08b0759a5900fc0399a753ecf0557df5\",\"cursorId\":\"0xe7071042f8c40f0f7a698a510749ff76bc65c683dcf4914eff7e98d9df5974f9\",\"emittingContract\":\"0x1111111111111111111111111111111111111111\",\"eventSignature\":\"0x5d07190b9ae441b4d7b16259a48424acd451492b12f5f99a29f5bfd992c13e43\",\"lifecycleState\":\"pending\",\"logIndex\":\"7\",\"observationId\":\"0x5ba8c5a70c814d35482df5f6598e549f83f2d0e27f6e3d25b83eb2a0e1d56a98\",\"occurredAt\":\"1778640240\",\"parentPulseId\":\"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\",\"pulseId\":\"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab005\",\"pulseType\":\"99\",\"receiptStatus\":\"success\",\"rootfieldId\":\"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\",\"sequence\":\"5\",\"subject\":\"0xfffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff1\",\"transactionIndex\":\"12\",\"txHash\":\"0x3333333333333333333333333333333333333333333333333333333333333338\",\"uri\":\"fixture://unsupported\"}","chainId":"8453","commitment":"0x69a9f953179a6ce2e08724dd759fb71b08b0759a5900fc0399a753ecf0557df5","cursorId":"0xe7071042f8c40f0f7a698a510749ff76bc65c683dcf4914eff7e98d9df5974f9","duplicateKind":"unique","emittingContract":"0x1111111111111111111111111111111111111111","eventSignature":"0x5d07190b9ae441b4d7b16259a48424acd451492b12f5f99a29f5bfd992c13e43","lifecycleState":"pending","logIndex":"7","observationId":"0x5ba8c5a70c814d35482df5f6598e549f83f2d0e27f6e3d25b83eb2a0e1d56a98","occurredAt":"1778640240","parentPulseId":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","pulseId":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab005","pulseType":"99","receiptStatus":"success","rootfieldId":"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb","sequence":"5","subject":"0xfffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff1","transactionIndex":"12","txHash":"0x3333333333333333333333333333333333333333333333333333333333333338","uri":"fixture://unsupported"}],"pulses":[{"actor":"0x4444444444444444444444444444444444444444","commitment":"0x4122209ff672fc04b2ec3af31ab1af79813971f86de000aa6534038cc79de6b5","observationId":"0x9d958aadf8bf46f989b51e541709a73d21970e7e79643f939c9a0000b50f9a91","occurredAt":"1778640000","parentPulseId":"0x0000000000000000000000000000000000000000000000000000000000000000","pulseId":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","pulseType":"1","rootfieldId":"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb","sequence":"1","subject":"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb","uri":"ipfs://bafy-flowmemory-example"},{"actor":"0x4444444444444444444444444444444444444444","commitment":"0x69a9f953179a6ce2e08724dd759fb71b08b0759a5900fc0399a753ecf0557df5","observationId":"0x49c6cd59d1f1916bc5301308be09c14d64127d1566362272dc5aa201ed53bdea","occurredAt":"1778640060","parentPulseId":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","pulseId":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab001","pulseType":"2","rootfieldId":"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb","sequence":"2","subject":"0xfffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff1","uri":"fixture://root-commit-valid"},{"actor":"0x4444444444444444444444444444444444444444","commitment":"0x4122209ff672fc04b2ec3af31ab1af79813971f86de000aa6534038cc79de6b5","observationId":"0x9d958aadf8bf46f989b51e541709a73d21970e7e79643f939c9a0000b50f9a91","occurredAt":"1778640000","parentPulseId":"0x0000000000000000000000000000000000000000000000000000000000000000","pulseId":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","pulseType":"1","rootfieldId":"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb","sequence":"1","subject":"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb","uri":"ipfs://bafy-flowmemory-example"},{"actor":"0x4444444444444444444444444444444444444444","commitment":"0x69a9f953179a6ce2e08724dd759fb71b08b0759a5900fc0399a753ecf0557df5","observationId":"0x223d74c971f301d800dd69aa30994bc3fa3089b34b0db1b0a7fc9d4e8d114c79","occurredAt":"1778640060","parentPulseId":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","pulseId":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab002","pulseType":"2","rootfieldId":"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb","sequence":"2","subject":"0xfffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff1","uri":"fixture://root-commit-valid"},{"actor":"0x4444444444444444444444444444444444444444","commitment":"0xcccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc","observationId":"0xe4a7065f1578c4a232b41f5984942e9b7b07760ce7dfeba77b38eb03173491df","occurredAt":"1778640120","parentPulseId":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","pulseId":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab003","pulseType":"2","rootfieldId":"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb","sequence":"3","subject":"0xfffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff1","uri":"fixture://root-commit-valid"},{"actor":"0x4444444444444444444444444444444444444444","commitment":"0x69a9f953179a6ce2e08724dd759fb71b08b0759a5900fc0399a753ecf0557df5","observationId":"0x92144fc24c81cdd6319598e6e0c58d84e3f18d9ad4a8be33bc956e32c2ae39f4","occurredAt":"1778640180","parentPulseId":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","pulseId":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab004","pulseType":"2","rootfieldId":"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb","sequence":"4","subject":"0xfffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff1","uri":"fixture://missing-artifact"},{"actor":"0x4444444444444444444444444444444444444444","commitment":"0x69a9f953179a6ce2e08724dd759fb71b08b0759a5900fc0399a753ecf0557df5","observationId":"0x5ba8c5a70c814d35482df5f6598e549f83f2d0e27f6e3d25b83eb2a0e1d56a98","occurredAt":"1778640240","parentPulseId":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","pulseId":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab005","pulseType":"99","rootfieldId":"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb","sequence":"5","subject":"0xfffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff1","uri":"fixture://unsupported"}],"rejectedLogs":[{"blockHash":"0x2222222222222222222222222222222222222222222222222222222222222228","blockNumber":"123462","chainId":"8453","logIndex":"8","message":"receipt status is reverted","reasonCode":"receipt.reverted","transactionIndex":"13","txHash":"0x3333333333333333333333333333333333333333333333333333333333333339"},{"blockHash":"0x2222222222222222222222222222222222222222222222222222222222222229","blockNumber":"123463","chainId":"8453","logIndex":"9","message":"unsupported event signature: 0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff","reasonCode":"log.malformed","transactionIndex":"14","txHash":"0x3333333333333333333333333333333333333333333333333333333333333340"}],"rootfields":[{"firstObservationId":"0x9d958aadf8bf46f989b51e541709a73d21970e7e79643f939c9a0000b50f9a91","latestObservationId":"0x5ba8c5a70c814d35482df5f6598e549f83f2d0e27f6e3d25b83eb2a0e1d56a98","pulseCount":6,"rootfieldId":"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"}],"schema":"flowmemory.indexer.state.v0","source":"fixture"}} diff --git a/services/indexer/package.json b/services/indexer/package.json new file mode 100644 index 00000000..06da877b --- /dev/null +++ b/services/indexer/package.json @@ -0,0 +1,10 @@ +{ + "name": "@flowmemory/indexer-v0", + "private": true, + "type": "module", + "scripts": { + "demo": "node src/demo.ts", + "index:fixtures": "node src/index-fixtures.ts", + "test": "node --test test/*.test.ts" + } +} diff --git a/services/indexer/src/demo.ts b/services/indexer/src/demo.ts new file mode 100644 index 00000000..a40da5af --- /dev/null +++ b/services/indexer/src/demo.ts @@ -0,0 +1,16 @@ +import { indexFlowPulseReceipts } from "./indexer.ts"; +import { loadIndexerFixtureReceipts } from "./fixtures.ts"; + +const state = indexFlowPulseReceipts(loadIndexerFixtureReceipts(), { + finalizedBlockNumber: "123458", +}); + +console.log(JSON.stringify({ + service: "flowmemory-indexer-v0", + mode: "fixture", + observationCount: state.observations.length, + rootfieldCount: state.rootfields.length, + duplicateCount: state.duplicates.length, + rejectedLogCount: state.rejectedLogs.length, + observations: state.observations, +}, null, 2)); diff --git a/services/indexer/src/fixtures.ts b/services/indexer/src/fixtures.ts new file mode 100644 index 00000000..b9a06f18 --- /dev/null +++ b/services/indexer/src/fixtures.ts @@ -0,0 +1,19 @@ +import { readFileSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; + +import type { FlowPulseReceiptFixture, RawFlowPulseLogFixture } from "../../shared/src/index.ts"; + +const __dirname = dirname(fileURLToPath(import.meta.url)); + +export function loadIndexerFixtureLogs(): RawFlowPulseLogFixture[] { + const path = join(__dirname, "../fixtures/flowpulse-logs.json"); + const fixture = JSON.parse(readFileSync(path, "utf8")) as { logs: RawFlowPulseLogFixture[] }; + return fixture.logs; +} + +export function loadIndexerFixtureReceipts(): FlowPulseReceiptFixture[] { + const path = join(__dirname, "../fixtures/flowpulse-receipts.json"); + const fixture = JSON.parse(readFileSync(path, "utf8")) as { receipts: FlowPulseReceiptFixture[] }; + return fixture.receipts; +} diff --git a/services/indexer/src/index-fixtures.ts b/services/indexer/src/index-fixtures.ts new file mode 100644 index 00000000..044e0f3c --- /dev/null +++ b/services/indexer/src/index-fixtures.ts @@ -0,0 +1,23 @@ +import { resolve } from "node:path"; + +import { indexFlowPulseReceipts } from "./indexer.ts"; +import { loadIndexerFixtureReceipts } from "./fixtures.ts"; +import { writeIndexerState } from "./persistence.ts"; + +const outArgIndex = process.argv.indexOf("--out"); +const outputPath = outArgIndex >= 0 ? process.argv[outArgIndex + 1] : "out/indexer-state.json"; + +const state = indexFlowPulseReceipts(loadIndexerFixtureReceipts(), { + finalizedBlockNumber: "123458", +}); + +writeIndexerState(outputPath, state); + +console.log(JSON.stringify({ + service: "flowmemory-indexer-v0", + outputPath: resolve(outputPath), + observations: state.observations.length, + cursors: state.cursors.length, + rejectedLogs: state.rejectedLogs.length, + duplicates: state.duplicates.length, +}, null, 2)); diff --git a/services/indexer/src/index.ts b/services/indexer/src/index.ts new file mode 100644 index 00000000..c5959e71 --- /dev/null +++ b/services/indexer/src/index.ts @@ -0,0 +1,4 @@ +export * from "./fixtures.ts"; +export * from "./indexer.ts"; +export * from "./persistence.ts"; +export * from "./rpc.ts"; diff --git a/services/indexer/src/indexer.ts b/services/indexer/src/indexer.ts new file mode 100644 index 00000000..031afd2a --- /dev/null +++ b/services/indexer/src/indexer.ts @@ -0,0 +1,297 @@ +import { + canonicalJson, + deriveSourceSetId, + logsFromReceiptFixture, + parseFlowPulseLogFixture, + type FlowPulseReceiptFixture, + type ParsedFlowPulseObservation, + type RawFlowPulseLogFixture, +} from "../../shared/src/index.ts"; + +export type DuplicateKind = + | "unique" + | "exactDuplicate" + | "conflictingDuplicate" + | "pulseDuplicate" + | "reorgReplacement"; + +export interface IndexedObservation extends ParsedFlowPulseObservation { + duplicateKind: DuplicateKind; + canonicalObservationJson: string; +} + +export interface IndexRejectedLog { + chainId: string; + blockNumber: string; + blockHash: string; + txHash: string; + transactionIndex: string; + logIndex: string; + reasonCode: string; + message: string; +} + +export interface IndexedCursor { + cursorId: string; + chainId: string; + sourceSetId: string; + blockNumber: string; + blockHash: string; + transactionIndex: string; + logIndex: string; +} + +export interface IndexedBatch { + schema: "flowmemory.indexer.batch.v0"; + source: "fixture" | "local-rpc-placeholder"; + sourceSetId: string; + observationCount: number; + cursorCount: number; + rejectedLogCount: number; +} + +export interface IndexedPulse { + observationId: string; + pulseId: string; + rootfieldId: string; + pulseType: string; + subject: string; + commitment: string; + parentPulseId: string; + sequence: string; + occurredAt: string; + actor: string; + uri: string; +} + +export interface IndexedRootfield { + rootfieldId: string; + firstObservationId: string; + latestObservationId: string; + pulseCount: number; +} + +export interface IndexerState { + schema: "flowmemory.indexer.state.v0"; + source: "fixture" | "local-rpc-placeholder"; + observations: IndexedObservation[]; + pulses: IndexedPulse[]; + rootfields: IndexedRootfield[]; + cursors: IndexedCursor[]; + batches: IndexedBatch[]; + rejectedLogs: IndexRejectedLog[]; + duplicates: Array<{ + kind: DuplicateKind; + observationId: string; + pulseId: string; + }>; +} + +export interface IndexerStateOptions { + finalizedBlockNumber?: string | number | bigint; + canonicalBlockHashes?: Record; + sourceAddresses?: string[]; +} + +function canonicalObservationJson(observation: ParsedFlowPulseObservation): string { + return canonicalJson({ + actor: observation.actor, + blockHash: observation.blockHash, + blockNumber: observation.blockNumber, + chainId: observation.chainId, + commitment: observation.commitment, + cursorId: observation.cursorId, + emittingContract: observation.emittingContract, + eventSignature: observation.eventSignature, + lifecycleState: observation.lifecycleState, + logIndex: observation.logIndex, + observationId: observation.observationId, + occurredAt: observation.occurredAt, + parentPulseId: observation.parentPulseId, + pulseId: observation.pulseId, + pulseType: observation.pulseType, + receiptStatus: observation.receiptStatus, + rootfieldId: observation.rootfieldId, + sequence: observation.sequence, + subject: observation.subject, + transactionIndex: observation.transactionIndex, + txHash: observation.txHash, + uri: observation.uri, + }); +} + +function observationStatus( + observation: ParsedFlowPulseObservation, + options: IndexerStateOptions, +): ParsedFlowPulseObservation["lifecycleState"] { + if (observation.lifecycleState === "removed") { + return "removed"; + } + + const canonicalBlockHash = options.canonicalBlockHashes?.[observation.blockNumber]; + if (canonicalBlockHash !== undefined && canonicalBlockHash.toLowerCase() !== observation.blockHash.toLowerCase()) { + return "reorged"; + } + + if (options.finalizedBlockNumber !== undefined) { + return BigInt(observation.blockNumber) <= BigInt(options.finalizedBlockNumber) ? "finalized" : "pending"; + } + + return "observed"; +} + +function duplicateKindFor( + observation: ParsedFlowPulseObservation, + canonicalJsonValue: string, + seenByObservationId: Map, + seenByPulseId: Map, +): DuplicateKind { + const existingObservation = seenByObservationId.get(observation.observationId); + if (existingObservation !== undefined) { + return existingObservation === canonicalJsonValue ? "exactDuplicate" : "conflictingDuplicate"; + } + + const existingPulse = seenByPulseId.get(observation.pulseId); + if (existingPulse !== undefined && existingPulse.observationId !== observation.observationId) { + if (existingPulse.blockHash !== observation.blockHash || existingPulse.logIndex !== observation.logIndex) { + return "reorgReplacement"; + } + return "pulseDuplicate"; + } + + return "unique"; +} + +export function indexFlowPulseLogs(logs: RawFlowPulseLogFixture[], options: IndexerStateOptions = {}): IndexerState { + const seenByObservationId = new Map(); + const seenByPulseId = new Map(); + const rootfields = new Map(); + const observations: IndexedObservation[] = []; + const cursors = new Map(); + const rejectedLogs: IndexRejectedLog[] = []; + const duplicates: IndexerState["duplicates"] = []; + const sourceAddresses = options.sourceAddresses ?? logs.map((log) => log.address); + const sourceSetId = deriveSourceSetId(logs[0]?.chainId ?? "0", sourceAddresses); + + for (const log of logs) { + if (log.receiptStatus !== "success") { + rejectedLogs.push({ + chainId: log.chainId, + blockNumber: log.blockNumber, + blockHash: log.blockHash, + txHash: log.transactionHash, + transactionIndex: log.transactionIndex, + logIndex: log.logIndex, + reasonCode: "receipt.reverted", + message: "receipt status is reverted", + }); + continue; + } + + let observation: ParsedFlowPulseObservation; + try { + observation = parseFlowPulseLogFixture(log, { sourceSetId }); + observation.lifecycleState = observationStatus(observation, options); + } catch (error) { + rejectedLogs.push({ + chainId: log.chainId, + blockNumber: log.blockNumber, + blockHash: log.blockHash, + txHash: log.transactionHash, + transactionIndex: log.transactionIndex, + logIndex: log.logIndex, + reasonCode: "log.malformed", + message: error instanceof Error ? error.message : "unknown parse error", + }); + continue; + } + + const canonicalObservation = canonicalObservationJson(observation); + const duplicateKind = duplicateKindFor(observation, canonicalObservation, seenByObservationId, seenByPulseId); + const indexedObservation: IndexedObservation = { + ...observation, + duplicateKind, + canonicalObservationJson: canonicalObservation, + }; + + observations.push(indexedObservation); + + if (duplicateKind === "unique") { + seenByObservationId.set(observation.observationId, canonicalObservation); + seenByPulseId.set(observation.pulseId, observation); + } else { + duplicates.push({ + kind: duplicateKind, + observationId: observation.observationId, + pulseId: observation.pulseId, + }); + if (duplicateKind === "pulseDuplicate" || duplicateKind === "reorgReplacement") { + const previous = observations.find((candidate) => candidate.pulseId === observation.pulseId); + if (previous !== undefined && previous.lifecycleState !== "removed" && previous.lifecycleState !== "reorged") { + previous.lifecycleState = "superseded"; + } + } + } + + cursors.set(observation.cursorId, { + cursorId: observation.cursorId, + chainId: observation.chainId, + sourceSetId, + blockNumber: observation.blockNumber, + blockHash: observation.blockHash, + transactionIndex: observation.transactionIndex, + logIndex: observation.logIndex, + }); + + const existingRootfield = rootfields.get(observation.rootfieldId); + if (existingRootfield === undefined) { + rootfields.set(observation.rootfieldId, { + rootfieldId: observation.rootfieldId, + firstObservationId: observation.observationId, + latestObservationId: observation.observationId, + pulseCount: 1, + }); + } else if (duplicateKind !== "exactDuplicate") { + existingRootfield.latestObservationId = observation.observationId; + existingRootfield.pulseCount += 1; + } + } + + return { + schema: "flowmemory.indexer.state.v0", + source: "fixture", + batches: [{ + schema: "flowmemory.indexer.batch.v0", + source: "fixture", + sourceSetId, + observationCount: observations.length, + cursorCount: cursors.size, + rejectedLogCount: rejectedLogs.length, + }], + observations, + cursors: [...cursors.values()].sort((left, right) => left.cursorId.localeCompare(right.cursorId)), + pulses: observations.map((observation) => ({ + observationId: observation.observationId, + pulseId: observation.pulseId, + rootfieldId: observation.rootfieldId, + pulseType: observation.pulseType, + subject: observation.subject, + commitment: observation.commitment, + parentPulseId: observation.parentPulseId, + sequence: observation.sequence, + occurredAt: observation.occurredAt, + actor: observation.actor, + uri: observation.uri, + })), + rootfields: [...rootfields.values()].sort((left, right) => left.rootfieldId.localeCompare(right.rootfieldId)), + rejectedLogs, + duplicates, + }; +} + +export function indexFlowPulseReceipts( + receipts: FlowPulseReceiptFixture[], + options: IndexerStateOptions = {}, +): IndexerState { + return indexFlowPulseLogs(receipts.flatMap((receipt) => logsFromReceiptFixture(receipt)), options); +} diff --git a/services/indexer/src/persistence.ts b/services/indexer/src/persistence.ts new file mode 100644 index 00000000..e2ed0bc3 --- /dev/null +++ b/services/indexer/src/persistence.ts @@ -0,0 +1,26 @@ +import { mkdirSync, readFileSync, writeFileSync } from "node:fs"; +import { dirname } from "node:path"; + +import { canonicalJson } from "../../shared/src/index.ts"; +import type { IndexerState } from "./indexer.ts"; + +export interface PersistedIndexerState { + schema: "flowmemory.indexer.persistence.v0"; + state: IndexerState; +} + +export function persistedIndexerState(state: IndexerState): PersistedIndexerState { + return { + schema: "flowmemory.indexer.persistence.v0", + state, + }; +} + +export function writeIndexerState(path: string, state: IndexerState): void { + mkdirSync(dirname(path), { recursive: true }); + writeFileSync(path, `${canonicalJson(persistedIndexerState(state))}\n`, "utf8"); +} + +export function readIndexerState(path: string): PersistedIndexerState { + return JSON.parse(readFileSync(path, "utf8")) as PersistedIndexerState; +} diff --git a/services/indexer/src/rpc.ts b/services/indexer/src/rpc.ts new file mode 100644 index 00000000..307286f6 --- /dev/null +++ b/services/indexer/src/rpc.ts @@ -0,0 +1,100 @@ +import { FLOWPULSE_EVENT_TOPIC0, type RawFlowPulseLogFixture } from "../../shared/src/index.ts"; + +export interface LocalRpcReadOptions { + rpcUrl: string; + addresses: string[]; + fromBlock: string; + toBlock: string; + fetchImpl?: typeof fetch; +} + +interface JsonRpcResponse { + jsonrpc: "2.0"; + id: number; + result?: T; + error?: { + code: number; + message: string; + }; +} + +interface RpcLog { + address: string; + topics: string[]; + data: string; + blockNumber: string; + blockHash: string; + transactionHash: string; + transactionIndex: string; + logIndex: string; + removed?: boolean; +} + +interface RpcReceipt { + status: string; +} + +function quantityToDecimalString(quantity: string): string { + if (!/^0x[0-9a-fA-F]+$/.test(quantity)) { + throw new Error(`invalid JSON-RPC quantity: ${quantity}`); + } + return BigInt(quantity).toString(); +} + +async function rpc(fetchImpl: typeof fetch, rpcUrl: string, method: string, params: unknown[]): Promise { + const response = await fetchImpl(rpcUrl, { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ + jsonrpc: "2.0", + id: 1, + method, + params, + }), + }); + + if (!response.ok) { + throw new Error(`JSON-RPC HTTP error ${response.status}`); + } + + const payload = await response.json() as JsonRpcResponse; + if (payload.error !== undefined) { + throw new Error(`JSON-RPC error ${payload.error.code}: ${payload.error.message}`); + } + if (payload.result === undefined) { + throw new Error(`JSON-RPC ${method} returned no result`); + } + return payload.result; +} + +export async function readLocalRpcFlowPulseLogs(options: LocalRpcReadOptions): Promise { + const fetchImpl = options.fetchImpl ?? fetch; + const chainIdQuantity = await rpc(fetchImpl, options.rpcUrl, "eth_chainId", []); + const chainId = quantityToDecimalString(chainIdQuantity); + const logs = await rpc(fetchImpl, options.rpcUrl, "eth_getLogs", [{ + address: options.addresses, + fromBlock: options.fromBlock, + toBlock: options.toBlock, + topics: [FLOWPULSE_EVENT_TOPIC0], + }]); + + const rawLogs: RawFlowPulseLogFixture[] = []; + for (const log of logs) { + const receipt = await rpc(fetchImpl, options.rpcUrl, "eth_getTransactionReceipt", [log.transactionHash]); + rawLogs.push({ + chainId, + address: log.address, + topics: log.topics, + data: log.data, + blockNumber: quantityToDecimalString(log.blockNumber), + blockHash: log.blockHash, + transactionHash: log.transactionHash, + transactionIndex: quantityToDecimalString(log.transactionIndex), + logIndex: quantityToDecimalString(log.logIndex), + receiptStatus: receipt.status === "0x1" ? "success" : "reverted", + removed: log.removed, + }); + } + + return rawLogs; +} diff --git a/services/indexer/test/indexer.test.ts b/services/indexer/test/indexer.test.ts new file mode 100644 index 00000000..3b183a40 --- /dev/null +++ b/services/indexer/test/indexer.test.ts @@ -0,0 +1,139 @@ +import assert from "node:assert/strict"; +import { mkdtempSync, readFileSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import test from "node:test"; + +import { indexFlowPulseLogs, indexFlowPulseReceipts } from "../src/indexer.ts"; +import { loadIndexerFixtureLogs, loadIndexerFixtureReceipts } from "../src/fixtures.ts"; +import { readIndexerState, writeIndexerState } from "../src/persistence.ts"; +import { readLocalRpcFlowPulseLogs } from "../src/rpc.ts"; + +test("indexes FlowPulse fixture logs into canonical observations", () => { + const state = indexFlowPulseLogs(loadIndexerFixtureLogs()); + assert.equal(state.schema, "flowmemory.indexer.state.v0"); + assert.equal(state.source, "fixture"); + assert.equal(state.observations.length, 1); + assert.equal(state.observations[0].observationId, "0x9d958aadf8bf46f989b51e541709a73d21970e7e79643f939c9a0000b50f9a91"); + assert.equal(state.observations[0].lifecycleState, "observed"); + assert.equal(state.observations[0].duplicateKind, "unique"); + assert.equal(state.pulses.length, 1); + assert.equal(state.rootfields.length, 1); +}); + +test("detects exact duplicate observations", () => { + const logs = loadIndexerFixtureLogs(); + const state = indexFlowPulseLogs([logs[0], logs[0]]); + assert.equal(state.observations.length, 2); + assert.equal(state.observations[1].duplicateKind, "exactDuplicate"); + assert.equal(state.duplicates.length, 1); +}); + +test("ingests receipt fixtures and rejects reverted or malformed logs cleanly", () => { + const state = indexFlowPulseReceipts(loadIndexerFixtureReceipts(), { + finalizedBlockNumber: "123458", + }); + + assert.equal(state.observations.length, 7); + assert.equal(state.cursors.length, 6); + assert.equal(state.batches[0].observationCount, 7); + assert.equal(state.batches[0].cursorCount, 6); + assert.equal(state.rejectedLogs.length, 2); + assert.deepEqual(state.rejectedLogs.map((log) => log.reasonCode), ["receipt.reverted", "log.malformed"]); + assert.equal(state.duplicates.length, 1); + assert.equal(state.duplicates[0].kind, "exactDuplicate"); +}); + +test("models finality threshold without claiming production reorg handling", () => { + const [rootfieldReceipt, rootCommitReceipt] = loadIndexerFixtureReceipts(); + const state = indexFlowPulseReceipts([rootfieldReceipt, rootCommitReceipt], { + finalizedBlockNumber: "123456", + }); + + assert.equal(state.observations[0].lifecycleState, "finalized"); + assert.equal(state.observations[1].lifecycleState, "pending"); +}); + +test("marks block hash mismatches as reorged in fixture state", () => { + const [, rootCommitReceipt] = loadIndexerFixtureReceipts(); + const state = indexFlowPulseReceipts([rootCommitReceipt], { + canonicalBlockHashes: { + "123457": "0xfffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff0", + }, + }); + + assert.equal(state.observations[0].lifecycleState, "reorged"); +}); + +test("persists deterministic indexer state JSON", () => { + const state = indexFlowPulseReceipts(loadIndexerFixtureReceipts(), { + finalizedBlockNumber: "123458", + }); + const dir = mkdtempSync(join(tmpdir(), "flowmemory-indexer-")); + const path = join(dir, "state.json"); + + try { + writeIndexerState(path, state); + const firstWrite = readFileSync(path, "utf8"); + writeIndexerState(path, state); + const secondWrite = readFileSync(path, "utf8"); + const persisted = readIndexerState(path); + + assert.equal(firstWrite, secondWrite); + assert.equal(persisted.schema, "flowmemory.indexer.persistence.v0"); + assert.equal(persisted.state.observations.length, 7); + } finally { + rmSync(dir, { recursive: true, force: true }); + } +}); + +test("ships an indexer persistence JSON schema fixture", () => { + const schemaPath = join(process.cwd(), "fixtures", "indexer-state.schema.json"); + const schema = JSON.parse(readFileSync(schemaPath, "utf8")) as { $id: string }; + assert.equal(schema.$id, "flowmemory.indexer.persistence.v0"); +}); + +test("maps mocked local RPC logs into raw FlowPulse fixtures without secrets", async () => { + const [fixtureLog] = loadIndexerFixtureLogs(); + const calls: string[] = []; + const fetchImpl = async (_url: string, init?: RequestInit): Promise => { + const body = JSON.parse(String(init?.body)) as { method: string }; + calls.push(body.method); + if (body.method === "eth_chainId") { + return Response.json({ jsonrpc: "2.0", id: 1, result: "0x2105" }); + } + if (body.method === "eth_getLogs") { + return Response.json({ + jsonrpc: "2.0", + id: 1, + result: [{ + address: fixtureLog.address, + topics: fixtureLog.topics, + data: fixtureLog.data, + blockNumber: "0x1e240", + blockHash: fixtureLog.blockHash, + transactionHash: fixtureLog.transactionHash, + transactionIndex: "0x7", + logIndex: "0x2", + }], + }); + } + if (body.method === "eth_getTransactionReceipt") { + return Response.json({ jsonrpc: "2.0", id: 1, result: { status: "0x1" } }); + } + return Response.json({ jsonrpc: "2.0", id: 1, error: { code: -32601, message: "not found" } }); + }; + + const logs = await readLocalRpcFlowPulseLogs({ + rpcUrl: "http://127.0.0.1:8545", + addresses: [fixtureLog.address], + fromBlock: "0x1e240", + toBlock: "0x1e240", + fetchImpl, + }); + + assert.deepEqual(calls, ["eth_chainId", "eth_getLogs", "eth_getTransactionReceipt"]); + assert.equal(logs[0].chainId, "8453"); + assert.equal(logs[0].receiptStatus, "success"); + assert.equal(indexFlowPulseLogs(logs).observations[0].observationId, "0x9d958aadf8bf46f989b51e541709a73d21970e7e79643f939c9a0000b50f9a91"); +}); diff --git a/services/indexer/tsconfig.json b/services/indexer/tsconfig.json new file mode 100644 index 00000000..13b3dd7d --- /dev/null +++ b/services/indexer/tsconfig.json @@ -0,0 +1,4 @@ +{ + "extends": "../tsconfig.base.json", + "include": ["src/**/*.ts", "test/**/*.ts"] +} diff --git a/services/shared/CRYPTO_BOUNDARY.md b/services/shared/CRYPTO_BOUNDARY.md new file mode 100644 index 00000000..1f6c5cc2 --- /dev/null +++ b/services/shared/CRYPTO_BOUNDARY.md @@ -0,0 +1,54 @@ +# Crypto Integration Boundary + +This document advances issue #47 inside the current service-only scope. It does not edit the crypto package or define a production cryptography API. + +## Current V0 Position + +The local indexer/verifier package includes narrow helpers for: + +- EVM Keccak-256. +- ABI encoding of identity preimages. +- ABI decoding of the FlowPulse fixture event. +- Hex, address, and `bytes32` normalization. + +These helpers exist so the fixture package can run without live RPC, external dependencies, or changes outside `services/`. + +## Boundary Rule + +Service code should depend on a small crypto adapter surface, not directly on future crypto internals. + +Expected future adapter functions: + +- `keccak256Hex(bytes): 0x...` +- `encodeObservationIdentity(fields): Uint8Array` +- `encodeCursorIdentity(fields): Uint8Array` +- `encodeReportDigestInput(reportCore): Uint8Array | string` +- `normalizeAddress(value): 0x...` +- `normalizeBytes32(value): 0x...` + +The adapter must preserve current fixture outputs unless an explicit migration decision changes identities. + +## Compatibility Fixtures + +Before replacing local helpers with a dedicated crypto package, add compatibility tests for: + +- Known Keccak vectors. +- `observationId` fixture output. +- `cursorId` fixture output. +- `reportId` fixture output. +- Rootfield registration commitment. +- Root commitment commitment. + +These tests should run without RPC, secrets, production databases, or artifact fetching. + +## Non-Goals + +- No custom cryptographic primitives. +- No signature or attestation scheme in V0. +- No proof network. +- No edits to the crypto implementation from this service package task. +- No hardcoded private keys, API keys, seed phrases, or RPC secrets. + +## Migration Note + +When a dedicated crypto package is ready, services should switch through an adapter module and keep the current fixture outputs as regression tests. If any identity changes, record a new durable decision before changing code. diff --git a/services/shared/README.md b/services/shared/README.md new file mode 100644 index 00000000..c8cf1547 --- /dev/null +++ b/services/shared/README.md @@ -0,0 +1,100 @@ +# FlowMemory Shared V0 + +This package contains dependency-free shared helpers for the local indexer/verifier V0 package. It is pure TypeScript: no live RPC, no database, no secrets, and no production runtime. + +## Commands + +From the repository root: + +```powershell +npm test --prefix services/shared +npm test +``` + +## Shared Helpers + +The package provides: + +- FlowPulse v0 event signature and `topic0` constants. +- V0 verifier report status constants. +- Narrow ABI encoding/decoding helpers used by fixtures. +- EVM Keccak-256. +- Canonical JSON serialization. +- `deriveObservationId`. +- `deriveSourceSetId`. +- `deriveCursorId`. +- `deriveReportId`. +- `parseFlowPulseLogFixture`. + +The helpers are intentionally narrow. They support the fixture-first package without becoming a full ABI, RPC, or crypto framework. + +## Identity Terms + +`pulseId`: + +Contract-emitted FlowPulse id. It is decoded from the event and can link protocol actions, but it is not enough to identify a receipt/log occurrence. + +`observationId`: + +Indexer-derived id: + +```text +keccak256(abi.encode( + "flowmemory.flowpulse.observation.v0", + chainId, + sourceContract, + txHash, + logIndex +)) +``` + +`cursorId`: + +Indexer-derived scan cursor id: + +```text +keccak256(abi.encode( + "flowmemory.indexer.cursor.v0", + chainId, + sourceSetId, + blockNumber, + blockHash, + transactionIndex, + logIndex +)) +``` + +`reportId`: + +Verifier-derived digest: + +```text +keccak256(canonical_json(reportCore)) +``` + +## Canonicalization Rules + +- Hex strings are lower-case and `0x` prefixed. +- Addresses are lower-case 20-byte hex strings. +- `bytes32` values are lower-case 32-byte hex strings. +- Integer-like EVM fields are decimal strings in JSON. +- Canonical JSON sorts object keys lexicographically. +- Arrays preserve documented order. +- Report digests exclude wall-clock timestamps, local paths, signatures, and operator notes. + +## Crypto Boundary + +The current package includes local Keccak and ABI helpers so the service package can run without expanding scope into the crypto implementation. Future integration with a dedicated crypto package should happen through a narrow adapter and compatibility fixtures. + +See [CRYPTO_BOUNDARY.md](./CRYPTO_BOUNDARY.md). + +## Boundaries + +- No live RPC. +- No arbitrary artifact fetching. +- No database. +- No deployment config. +- No secrets in env files. +- No tokenomics or verifier economics. + +See [docs/INDEXER_VERIFIER_MVP.md](../../docs/INDEXER_VERIFIER_MVP.md) for the full pipeline. diff --git a/services/shared/fixtures/flowpulse-observation.json b/services/shared/fixtures/flowpulse-observation.json new file mode 100644 index 00000000..c4721e99 --- /dev/null +++ b/services/shared/fixtures/flowpulse-observation.json @@ -0,0 +1,79 @@ +{ + "description": "Synthetic FlowPulse observation fixture. Values are local test vectors only and do not reference live chain data.", + "identityInput": { + "chainId": "8453", + "emittingContract": "0x1111111111111111111111111111111111111111", + "blockNumber": "123456", + "blockHash": "0x2222222222222222222222222222222222222222222222222222222222222222", + "txHash": "0x3333333333333333333333333333333333333333333333333333333333333333", + "transactionIndex": "7", + "logIndex": "2", + "eventSignature": "0x5d07190b9ae441b4d7b16259a48424acd451492b12f5f99a29f5bfd992c13e43", + "pulseId": "0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "rootfieldId": "0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" + }, + "rawLog": { + "chainId": "8453", + "address": "0x1111111111111111111111111111111111111111", + "topics": [ + "0x5d07190b9ae441b4d7b16259a48424acd451492b12f5f99a29f5bfd992c13e43", + "0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "0x0000000000000000000000004444444444444444444444444444444444444444" + ], + "data": "0x0000000000000000000000000000000000000000000000000000000000000001bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb4122209ff672fc04b2ec3af31ab1af79813971f86de000aa6534038cc79de6b500000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000006a03e48000000000000000000000000000000000000000000000000000000000000000e0000000000000000000000000000000000000000000000000000000000000001e697066733a2f2f626166792d666c6f776d656d6f72792d6578616d706c650000", + "blockNumber": "123456", + "blockHash": "0x2222222222222222222222222222222222222222222222222222222222222222", + "transactionHash": "0x3333333333333333333333333333333333333333333333333333333333333333", + "transactionIndex": "7", + "logIndex": "2", + "receiptStatus": "success" + }, + "expected": { + "flowPulseTopic0": "0x5d07190b9ae441b4d7b16259a48424acd451492b12f5f99a29f5bfd992c13e43", + "observationId": "0x9d958aadf8bf46f989b51e541709a73d21970e7e79643f939c9a0000b50f9a91", + "reportId": "0xae35a9633973459196170e08410675c18666ccc582b2700761b54619fd8e9294" + }, + "reportCore": { + "schema": "flowmemory.verifier.report.v0", + "verifierSpecVersion": "0", + "resolverPolicyId": "flowmemory.resolver.policy.v0.fixture", + "status": "valid", + "observationId": "0x9d958aadf8bf46f989b51e541709a73d21970e7e79643f939c9a0000b50f9a91", + "observation": { + "chainId": "8453", + "emittingContract": "0x1111111111111111111111111111111111111111", + "eventSignature": "0x5d07190b9ae441b4d7b16259a48424acd451492b12f5f99a29f5bfd992c13e43", + "blockNumber": "123456", + "blockHash": "0x2222222222222222222222222222222222222222222222222222222222222222", + "txHash": "0x3333333333333333333333333333333333333333333333333333333333333333", + "transactionIndex": "7", + "logIndex": "2", + "pulseId": "0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "rootfieldId": "0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" + }, + "flowPulse": { + "actor": "0x4444444444444444444444444444444444444444", + "pulseType": "1", + "subject": "0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "commitment": "0x4122209ff672fc04b2ec3af31ab1af79813971f86de000aa6534038cc79de6b5", + "parentPulseId": "0x0000000000000000000000000000000000000000000000000000000000000000", + "sequence": "1", + "occurredAt": "1778640000", + "uri": "ipfs://bafy-flowmemory-example" + }, + "checks": [ + { + "id": "observation.decoded", + "passed": true + } + ], + "evidenceRefs": [ + { + "kind": "rootfield-registration", + "uri": "ipfs://bafy-flowmemory-example" + } + ], + "reasonCodes": [] + } +} diff --git a/services/shared/package.json b/services/shared/package.json new file mode 100644 index 00000000..411c11a0 --- /dev/null +++ b/services/shared/package.json @@ -0,0 +1,8 @@ +{ + "name": "@flowmemory/indexer-verifier-foundation", + "private": true, + "type": "module", + "scripts": { + "test": "node --test test/*.test.ts" + } +} diff --git a/services/shared/src/abi.ts b/services/shared/src/abi.ts new file mode 100644 index 00000000..fb622288 --- /dev/null +++ b/services/shared/src/abi.ts @@ -0,0 +1,128 @@ +import { hexToBytes, normalizeAddress, normalizeBytes32 } from "./hex.ts"; + +function concatBytes(parts: Uint8Array[]): Uint8Array { + const length = parts.reduce((sum, part) => sum + part.length, 0); + const output = new Uint8Array(length); + let offset = 0; + for (const part of parts) { + output.set(part, offset); + offset += part.length; + } + return output; +} + +export function encodeUint256(value: string | number | bigint): Uint8Array { + const bigintValue = BigInt(value); + if (bigintValue < 0n) { + throw new Error("uint256 cannot be negative"); + } + const output = new Uint8Array(32); + let remaining = bigintValue; + for (let index = 31; index >= 0; index -= 1) { + output[index] = Number(remaining & 0xffn); + remaining >>= 8n; + } + if (remaining !== 0n) { + throw new Error("uint256 overflow"); + } + return output; +} + +export function encodeAddress(value: string): Uint8Array { + const output = new Uint8Array(32); + output.set(hexToBytes(normalizeAddress(value)), 12); + return output; +} + +export function encodeBytes32(value: string): Uint8Array { + return hexToBytes(normalizeBytes32(value)); +} + +export function encodeStringTail(value: string): Uint8Array { + const bytes = new TextEncoder().encode(value); + const paddedLength = Math.ceil(bytes.length / 32) * 32; + const padded = new Uint8Array(paddedLength); + padded.set(bytes); + return concatBytes([encodeUint256(bytes.length), padded]); +} + +export function abiEncodeObservationIdentity(fields: { + domain: string; + chainId: string | number | bigint; + emittingContract: string; + txHash: string; + logIndex: string | number | bigint; +}): Uint8Array { + const headLength = 5 * 32; + const head = [ + encodeUint256(headLength), + encodeUint256(fields.chainId), + encodeAddress(fields.emittingContract), + encodeBytes32(fields.txHash), + encodeUint256(fields.logIndex), + ]; + return concatBytes([...head, encodeStringTail(fields.domain)]); +} + +export function abiEncodeCursorIdentity(fields: { + domain: string; + chainId: string | number | bigint; + sourceSetId: string; + blockNumber: string | number | bigint; + blockHash: string; + transactionIndex: string | number | bigint; + logIndex: string | number | bigint; +}): Uint8Array { + const headLength = 7 * 32; + const head = [ + encodeUint256(headLength), + encodeUint256(fields.chainId), + encodeBytes32(fields.sourceSetId), + encodeUint256(fields.blockNumber), + encodeBytes32(fields.blockHash), + encodeUint256(fields.transactionIndex), + encodeUint256(fields.logIndex), + ]; + return concatBytes([...head, encodeStringTail(fields.domain)]); +} + +function wordAt(data: Uint8Array, wordIndex: number): Uint8Array { + const start = wordIndex * 32; + const end = start + 32; + if (end > data.length) { + throw new Error(`missing ABI word ${wordIndex}`); + } + return data.subarray(start, end); +} + +export function decodeUint256Word(data: Uint8Array, wordIndex: number): bigint { + let value = 0n; + for (const byte of wordAt(data, wordIndex)) { + value = (value << 8n) | BigInt(byte); + } + return value; +} + +export function decodeBytes32Word(data: Uint8Array, wordIndex: number): `0x${string}` { + const bytes = wordAt(data, wordIndex); + return `0x${Array.from(bytes, (byte) => byte.toString(16).padStart(2, "0")).join("")}`; +} + +export function decodeAddressTopic(topic: string): `0x${string}` { + const bytes = hexToBytes(topic, 32); + return `0x${Array.from(bytes.subarray(12), (byte) => byte.toString(16).padStart(2, "0")).join("")}`; +} + +export function decodeString(data: Uint8Array, offset: bigint): string { + if (offset % 32n !== 0n) { + throw new Error("dynamic string offset must be word-aligned"); + } + const wordIndex = Number(offset / 32n); + const length = Number(decodeUint256Word(data, wordIndex)); + const start = Number(offset) + 32; + const end = start + length; + if (end > data.length) { + throw new Error("dynamic string exceeds ABI data length"); + } + return new TextDecoder().decode(data.subarray(start, end)); +} diff --git a/services/shared/src/canonical-json.ts b/services/shared/src/canonical-json.ts new file mode 100644 index 00000000..f0b29f06 --- /dev/null +++ b/services/shared/src/canonical-json.ts @@ -0,0 +1,24 @@ +type JsonValue = null | boolean | number | string | JsonValue[] | { [key: string]: JsonValue | undefined }; + +function normalize(value: JsonValue): JsonValue { + if (value === null || typeof value !== "object") { + return value; + } + + if (Array.isArray(value)) { + return value.map((entry) => normalize(entry)); + } + + const output: { [key: string]: JsonValue } = {}; + for (const key of Object.keys(value).sort()) { + const entry = value[key]; + if (entry !== undefined) { + output[key] = normalize(entry); + } + } + return output; +} + +export function canonicalJson(value: JsonValue): string { + return JSON.stringify(normalize(value)); +} diff --git a/services/shared/src/constants.ts b/services/shared/src/constants.ts new file mode 100644 index 00000000..3865e0d3 --- /dev/null +++ b/services/shared/src/constants.ts @@ -0,0 +1,28 @@ +export const FLOWPULSE_EVENT_SIGNATURE = + "FlowPulse(bytes32,bytes32,address,uint8,bytes32,bytes32,bytes32,uint64,uint64,string)"; + +export const FLOWPULSE_EVENT_TOPIC0 = + "0x5d07190b9ae441b4d7b16259a48424acd451492b12f5f99a29f5bfd992c13e43"; + +export const OBSERVATION_ID_DOMAIN = "flowmemory.flowpulse.observation.v0"; + +export const CURSOR_ID_DOMAIN = "flowmemory.indexer.cursor.v0"; + +export const SOURCE_SET_ID_DOMAIN = "flowmemory.indexer.source_set.v0"; + +export const VERIFIER_REPORT_SCHEMA = "flowmemory.verifier.report.v0"; + +export const VERIFIER_STATUSES = Object.freeze([ + "valid", + "invalid", + "unresolved", + "unsupported", + "reorged", +]); + +export type VerifierStatus = + | "valid" + | "invalid" + | "unresolved" + | "unsupported" + | "reorged"; diff --git a/services/shared/src/flowpulse.ts b/services/shared/src/flowpulse.ts new file mode 100644 index 00000000..3be61250 --- /dev/null +++ b/services/shared/src/flowpulse.ts @@ -0,0 +1,161 @@ +import { + decodeAddressTopic, + decodeBytes32Word, + decodeString, + decodeUint256Word, +} from "./abi.ts"; +import { FLOWPULSE_EVENT_TOPIC0 } from "./constants.ts"; +import { normalizeAddress, normalizeBytes32, normalizeHex, hexToBytes } from "./hex.ts"; +import { + deriveCursorId, + deriveObservationId, + deriveSourceSetId, + type ObservationLifecycleState, +} from "./observation.ts"; + +export interface RawFlowPulseLogFixture { + chainId: string; + address: string; + topics: string[]; + data: string; + blockNumber: string; + blockHash: string; + transactionHash: string; + transactionIndex: string; + logIndex: string; + receiptStatus: "success" | "reverted"; + removed?: boolean; +} + +export interface FlowPulseReceiptFixtureLog { + address: string; + topics: string[]; + data: string; + logIndex: string; + removed?: boolean; +} + +export interface FlowPulseReceiptFixture { + chainId: string; + blockNumber: string; + blockHash: string; + transactionHash: string; + transactionIndex: string; + status: "success" | "reverted"; + logs: FlowPulseReceiptFixtureLog[]; +} + +export interface ParsedFlowPulseObservation { + observationId: `0x${string}`; + cursorId: `0x${string}`; + lifecycleState: ObservationLifecycleState; + chainId: string; + emittingContract: `0x${string}`; + eventSignature: `0x${string}`; + blockNumber: string; + blockHash: `0x${string}`; + txHash: `0x${string}`; + transactionIndex: string; + logIndex: string; + receiptStatus: "success" | "reverted"; + pulseId: `0x${string}`; + rootfieldId: `0x${string}`; + actor: `0x${string}`; + pulseType: string; + subject: `0x${string}`; + commitment: `0x${string}`; + parentPulseId: `0x${string}`; + sequence: string; + occurredAt: string; + uri: string; +} + +export interface FlowPulseLogParseOptions { + sourceSetId?: string; + sourceAddresses?: string[]; +} + +export function parseFlowPulseLogFixture( + log: RawFlowPulseLogFixture, + options: FlowPulseLogParseOptions = {}, +): ParsedFlowPulseObservation { + if (log.topics.length !== 4) { + throw new Error(`FlowPulse log must have 4 topics, got ${log.topics.length}`); + } + + const eventSignature = normalizeBytes32(log.topics[0]); + if (eventSignature !== FLOWPULSE_EVENT_TOPIC0) { + throw new Error(`unsupported event signature: ${eventSignature}`); + } + + const data = hexToBytes(normalizeHex(log.data)); + const emittingContract = normalizeAddress(log.address); + const blockHash = normalizeBytes32(log.blockHash); + const txHash = normalizeBytes32(log.transactionHash); + const pulseId = normalizeBytes32(log.topics[1]); + const rootfieldId = normalizeBytes32(log.topics[2]); + const actor = decodeAddressTopic(log.topics[3]); + + const observationId = deriveObservationId({ + chainId: log.chainId, + emittingContract, + blockNumber: log.blockNumber, + blockHash, + txHash, + transactionIndex: log.transactionIndex, + logIndex: log.logIndex, + eventSignature, + pulseId, + rootfieldId, + }); + const sourceSetId = options.sourceSetId ?? deriveSourceSetId(log.chainId, options.sourceAddresses ?? [emittingContract]); + const cursorId = deriveCursorId({ + chainId: log.chainId, + sourceSetId, + blockNumber: log.blockNumber, + blockHash, + transactionIndex: log.transactionIndex, + logIndex: log.logIndex, + }); + + return { + observationId, + cursorId, + lifecycleState: log.removed ? "removed" : "observed", + chainId: log.chainId, + emittingContract, + eventSignature, + blockNumber: log.blockNumber, + blockHash, + txHash, + transactionIndex: log.transactionIndex, + logIndex: log.logIndex, + receiptStatus: log.receiptStatus, + pulseId, + rootfieldId, + actor, + pulseType: decodeUint256Word(data, 0).toString(), + subject: decodeBytes32Word(data, 1), + commitment: decodeBytes32Word(data, 2), + parentPulseId: decodeBytes32Word(data, 3), + sequence: decodeUint256Word(data, 4).toString(), + occurredAt: decodeUint256Word(data, 5).toString(), + uri: decodeString(data, decodeUint256Word(data, 6)), + }; +} + +export function logsFromReceiptFixture(receipt: FlowPulseReceiptFixture): RawFlowPulseLogFixture[] { + return receipt.logs.map((log) => ({ + chainId: receipt.chainId, + address: log.address, + topics: log.topics, + data: log.data, + blockNumber: receipt.blockNumber, + blockHash: receipt.blockHash, + transactionHash: receipt.transactionHash, + transactionIndex: receipt.transactionIndex, + logIndex: log.logIndex, + receiptStatus: receipt.status, + removed: log.removed, + })); +} diff --git a/services/shared/src/hex.ts b/services/shared/src/hex.ts new file mode 100644 index 00000000..08481b80 --- /dev/null +++ b/services/shared/src/hex.ts @@ -0,0 +1,43 @@ +export type Hex = `0x${string}`; + +export function normalizeHex(value: string, bytes?: number): Hex { + if (typeof value !== "string") { + throw new TypeError("hex value must be a string"); + } + + const prefixed = value.startsWith("0x") || value.startsWith("0X") ? value.slice(2) : value; + if (!/^[0-9a-fA-F]*$/.test(prefixed)) { + throw new Error(`invalid hex value: ${value}`); + } + + if (bytes !== undefined && prefixed.length !== bytes * 2) { + throw new Error(`expected ${bytes} bytes, got ${prefixed.length / 2}`); + } + + if (prefixed.length % 2 !== 0) { + throw new Error("hex value must have an even number of nibbles"); + } + + return `0x${prefixed.toLowerCase()}`; +} + +export function hexToBytes(value: string, bytes?: number): Uint8Array { + const normalized = normalizeHex(value, bytes).slice(2); + const output = new Uint8Array(normalized.length / 2); + for (let index = 0; index < output.length; index += 1) { + output[index] = Number.parseInt(normalized.slice(index * 2, index * 2 + 2), 16); + } + return output; +} + +export function bytesToHex(bytes: Uint8Array): Hex { + return `0x${Array.from(bytes, (byte) => byte.toString(16).padStart(2, "0")).join("")}`; +} + +export function normalizeAddress(value: string): Hex { + return normalizeHex(value, 20); +} + +export function normalizeBytes32(value: string): Hex { + return normalizeHex(value, 32); +} diff --git a/services/shared/src/index.ts b/services/shared/src/index.ts new file mode 100644 index 00000000..77f3874c --- /dev/null +++ b/services/shared/src/index.ts @@ -0,0 +1,8 @@ +export * from "./abi.ts"; +export * from "./canonical-json.ts"; +export * from "./constants.ts"; +export * from "./flowpulse.ts"; +export * from "./hex.ts"; +export * from "./keccak.ts"; +export * from "./observation.ts"; +export * from "./report.ts"; diff --git a/services/shared/src/keccak.ts b/services/shared/src/keccak.ts new file mode 100644 index 00000000..d3d4f3d1 --- /dev/null +++ b/services/shared/src/keccak.ts @@ -0,0 +1,126 @@ +import { bytesToHex } from "./hex.ts"; + +const MASK_64 = (1n << 64n) - 1n; + +const ROUND_CONSTANTS = [ + 0x0000000000000001n, + 0x0000000000008082n, + 0x800000000000808an, + 0x8000000080008000n, + 0x000000000000808bn, + 0x0000000080000001n, + 0x8000000080008081n, + 0x8000000000008009n, + 0x000000000000008an, + 0x0000000000000088n, + 0x0000000080008009n, + 0x000000008000000an, + 0x000000008000808bn, + 0x800000000000008bn, + 0x8000000000008089n, + 0x8000000000008003n, + 0x8000000000008002n, + 0x8000000000000080n, + 0x000000000000800an, + 0x800000008000000an, + 0x8000000080008081n, + 0x8000000000008080n, + 0x0000000080000001n, + 0x8000000080008008n, +]; + +const ROTATION_OFFSETS = [ + [0, 36, 3, 41, 18], + [1, 44, 10, 45, 2], + [62, 6, 43, 15, 61], + [28, 55, 25, 21, 56], + [27, 20, 39, 8, 14], +]; + +function rotateLeft64(value: bigint, shift: number): bigint { + if (shift === 0) { + return value & MASK_64; + } + const amount = BigInt(shift); + return ((value << amount) | (value >> (64n - amount))) & MASK_64; +} + +function keccakF1600(state: bigint[]): void { + for (const roundConstant of ROUND_CONSTANTS) { + const c = new Array(5); + const d = new Array(5); + + for (let x = 0; x < 5; x += 1) { + c[x] = state[x] ^ state[x + 5] ^ state[x + 10] ^ state[x + 15] ^ state[x + 20]; + } + + for (let x = 0; x < 5; x += 1) { + d[x] = c[(x + 4) % 5] ^ rotateLeft64(c[(x + 1) % 5], 1); + } + + for (let x = 0; x < 5; x += 1) { + for (let y = 0; y < 5; y += 1) { + state[x + 5 * y] = (state[x + 5 * y] ^ d[x]) & MASK_64; + } + } + + const b = new Array(25).fill(0n); + for (let x = 0; x < 5; x += 1) { + for (let y = 0; y < 5; y += 1) { + b[y + 5 * ((2 * x + 3 * y) % 5)] = rotateLeft64(state[x + 5 * y], ROTATION_OFFSETS[x][y]); + } + } + + for (let x = 0; x < 5; x += 1) { + for (let y = 0; y < 5; y += 1) { + state[x + 5 * y] = (b[x + 5 * y] ^ ((~b[((x + 1) % 5) + 5 * y]) & b[((x + 2) % 5) + 5 * y])) & MASK_64; + } + } + + state[0] = (state[0] ^ roundConstant) & MASK_64; + } +} + +function xorBlock(state: bigint[], block: Uint8Array): void { + for (let index = 0; index < block.length; index += 1) { + const lane = Math.floor(index / 8); + const shift = BigInt((index % 8) * 8); + state[lane] = (state[lane] ^ (BigInt(block[index]) << shift)) & MASK_64; + } +} + +export function keccak256(bytes: Uint8Array): Uint8Array { + const rateBytes = 136; + const state = new Array(25).fill(0n); + let offset = 0; + + while (offset + rateBytes <= bytes.length) { + xorBlock(state, bytes.subarray(offset, offset + rateBytes)); + keccakF1600(state); + offset += rateBytes; + } + + const finalBlock = new Uint8Array(rateBytes); + finalBlock.set(bytes.subarray(offset)); + finalBlock[bytes.length - offset] ^= 0x01; + finalBlock[rateBytes - 1] ^= 0x80; + xorBlock(state, finalBlock); + keccakF1600(state); + + const output = new Uint8Array(32); + for (let index = 0; index < output.length; index += 1) { + const lane = Math.floor(index / 8); + const shift = BigInt((index % 8) * 8); + output[index] = Number((state[lane] >> shift) & 0xffn); + } + + return output; +} + +export function keccak256Hex(bytes: Uint8Array): `0x${string}` { + return bytesToHex(keccak256(bytes)); +} + +export function keccak256Utf8(value: string): `0x${string}` { + return keccak256Hex(new TextEncoder().encode(value)); +} diff --git a/services/shared/src/observation.ts b/services/shared/src/observation.ts new file mode 100644 index 00000000..cb8aa6d7 --- /dev/null +++ b/services/shared/src/observation.ts @@ -0,0 +1,96 @@ +import { encodeAddress, encodeBytes32, encodeUint256, abiEncodeCursorIdentity } from "./abi.ts"; +import { CURSOR_ID_DOMAIN, SOURCE_SET_ID_DOMAIN } from "./constants.ts"; +import { normalizeAddress, normalizeBytes32 } from "./hex.ts"; +import { keccak256Hex, keccak256Utf8 } from "./keccak.ts"; + +export type ObservationLifecycleState = "observed" | "pending" | "finalized" | "removed" | "superseded" | "reorged"; + +export interface ObservationIdentityInput { + chainId: string | number | bigint; + emittingContract: string; + blockNumber: string | number | bigint; + blockHash: string; + txHash: string; + transactionIndex: string | number | bigint; + logIndex: string | number | bigint; + eventSignature: string; + pulseId: string; + rootfieldId: string; +} + +const FLOWPULSE_OBSERVATION_TYPE = + "FlowPulseObservationV0(uint256 chainId,address emittingContract,uint64 blockNumber,bytes32 blockHash,bytes32 txHash,uint32 transactionIndex,uint32 logIndex,bytes32 eventSignature,bytes32 pulseId,bytes32 rootfieldId)"; + +function concatBytes(parts: Uint8Array[]): Uint8Array { + const length = parts.reduce((sum, part) => sum + part.length, 0); + const output = new Uint8Array(length); + let offset = 0; + for (const part of parts) { + output.set(part, offset); + offset += part.length; + } + return output; +} + +export function normalizeObservationIdentityInput(input: ObservationIdentityInput): Required { + return { + chainId: input.chainId, + emittingContract: normalizeAddress(input.emittingContract), + blockNumber: input.blockNumber, + blockHash: normalizeBytes32(input.blockHash), + txHash: normalizeBytes32(input.txHash), + transactionIndex: input.transactionIndex, + logIndex: input.logIndex, + eventSignature: normalizeBytes32(input.eventSignature), + pulseId: normalizeBytes32(input.pulseId), + rootfieldId: normalizeBytes32(input.rootfieldId), + }; +} + +export function encodeObservationIdentity(input: ObservationIdentityInput): Uint8Array { + const normalized = normalizeObservationIdentityInput(input); + return concatBytes([ + encodeBytes32(keccak256Utf8(FLOWPULSE_OBSERVATION_TYPE)), + encodeUint256(normalized.chainId), + encodeAddress(normalized.emittingContract), + encodeUint256(normalized.blockNumber), + encodeBytes32(normalized.blockHash), + encodeBytes32(normalized.txHash), + encodeUint256(normalized.transactionIndex), + encodeUint256(normalized.logIndex), + encodeBytes32(normalized.eventSignature), + encodeBytes32(normalized.pulseId), + encodeBytes32(normalized.rootfieldId), + ]); +} + +export function deriveObservationId(input: ObservationIdentityInput): `0x${string}` { + return keccak256Hex(encodeObservationIdentity(input)); +} + +export interface CursorIdentityInput { + chainId: string | number | bigint; + sourceSetId: string; + blockNumber: string | number | bigint; + blockHash: string; + transactionIndex: string | number | bigint; + logIndex: string | number | bigint; +} + +export function deriveSourceSetId(chainId: string | number | bigint, addresses: string[]): `0x${string}` { + const normalizedAddresses = [...new Set(addresses.map((address) => normalizeAddress(address)))].sort(); + const payload = `${SOURCE_SET_ID_DOMAIN}|${BigInt(chainId).toString()}|${normalizedAddresses.join(",")}`; + return keccak256Hex(new TextEncoder().encode(payload)); +} + +export function deriveCursorId(input: CursorIdentityInput): `0x${string}` { + return keccak256Hex(abiEncodeCursorIdentity({ + domain: CURSOR_ID_DOMAIN, + chainId: input.chainId, + sourceSetId: normalizeBytes32(input.sourceSetId), + blockNumber: input.blockNumber, + blockHash: normalizeBytes32(input.blockHash), + transactionIndex: input.transactionIndex, + logIndex: input.logIndex, + })); +} diff --git a/services/shared/src/report.ts b/services/shared/src/report.ts new file mode 100644 index 00000000..b86b703b --- /dev/null +++ b/services/shared/src/report.ts @@ -0,0 +1,80 @@ +import { canonicalJson } from "./canonical-json.ts"; +import { VERIFIER_REPORT_SCHEMA, VERIFIER_STATUSES, type VerifierStatus } from "./constants.ts"; +import { normalizeAddress, normalizeBytes32 } from "./hex.ts"; +import { keccak256Hex } from "./keccak.ts"; + +export interface VerifierReportCore { + schema: typeof VERIFIER_REPORT_SCHEMA; + verifierSpecVersion: string; + resolverPolicyId: string; + status: VerifierStatus; + observationId: string; + observation: { + chainId: string; + emittingContract: string; + eventSignature: string; + blockNumber: string; + blockHash: string; + txHash: string; + transactionIndex: string; + logIndex: string; + pulseId: string; + rootfieldId: string; + }; + flowPulse: { + actor: string; + pulseType: string; + subject: string; + commitment: string; + parentPulseId: string; + sequence: string; + occurredAt: string; + uri: string; + }; + checks: Array>; + evidenceRefs: Array>; + reasonCodes: string[]; +} + +export function isVerifierStatus(value: string): value is VerifierStatus { + return VERIFIER_STATUSES.includes(value as VerifierStatus); +} + +export function normalizeReportCore(report: VerifierReportCore): VerifierReportCore { + if (report.schema !== VERIFIER_REPORT_SCHEMA) { + throw new Error(`unsupported report schema: ${report.schema}`); + } + if (!isVerifierStatus(report.status)) { + throw new Error(`unsupported verifier status: ${report.status}`); + } + + return { + ...report, + observationId: normalizeBytes32(report.observationId), + observation: { + ...report.observation, + emittingContract: normalizeAddress(report.observation.emittingContract), + eventSignature: normalizeBytes32(report.observation.eventSignature), + blockHash: normalizeBytes32(report.observation.blockHash), + txHash: normalizeBytes32(report.observation.txHash), + pulseId: normalizeBytes32(report.observation.pulseId), + rootfieldId: normalizeBytes32(report.observation.rootfieldId), + }, + flowPulse: { + ...report.flowPulse, + actor: normalizeAddress(report.flowPulse.actor), + subject: normalizeBytes32(report.flowPulse.subject), + commitment: normalizeBytes32(report.flowPulse.commitment), + parentPulseId: normalizeBytes32(report.flowPulse.parentPulseId), + }, + reasonCodes: [...report.reasonCodes].sort(), + }; +} + +export function canonicalReportJson(report: VerifierReportCore): string { + return canonicalJson(normalizeReportCore(report)); +} + +export function deriveReportId(report: VerifierReportCore): `0x${string}` { + return keccak256Hex(new TextEncoder().encode(canonicalReportJson(report))); +} diff --git a/services/shared/test/foundation.test.ts b/services/shared/test/foundation.test.ts new file mode 100644 index 00000000..35ae7e80 --- /dev/null +++ b/services/shared/test/foundation.test.ts @@ -0,0 +1,98 @@ +import assert from "node:assert/strict"; +import { readFileSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; +import test from "node:test"; + +import { + FLOWPULSE_EVENT_SIGNATURE, + FLOWPULSE_EVENT_TOPIC0, + VERIFIER_STATUSES, + canonicalJson, + deriveObservationId, + deriveReportId, + isVerifierStatus, + keccak256Utf8, + normalizeAddress, + parseFlowPulseLogFixture, + type ObservationIdentityInput, + type RawFlowPulseLogFixture, + type VerifierReportCore, +} from "../src/index.ts"; + +const __dirname = dirname(fileURLToPath(import.meta.url)); +const fixture = JSON.parse( + readFileSync(join(__dirname, "../fixtures/flowpulse-observation.json"), "utf8"), +) as { + identityInput: ObservationIdentityInput; + rawLog: RawFlowPulseLogFixture; + expected: { + flowPulseTopic0: string; + observationId: string; + reportId: string; + }; + reportCore: VerifierReportCore; +}; + +test("computes EVM Keccak-256 test vectors", () => { + assert.equal( + keccak256Utf8(""), + "0xc5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470", + ); + assert.equal(keccak256Utf8(FLOWPULSE_EVENT_SIGNATURE), FLOWPULSE_EVENT_TOPIC0); + assert.equal(FLOWPULSE_EVENT_TOPIC0, fixture.expected.flowPulseTopic0); +}); + +test("derives canonical FlowPulse observation id from receipt/log metadata", () => { + assert.equal(deriveObservationId(fixture.identityInput), fixture.expected.observationId); +}); + +test("parses a fixture-only FlowPulse log without live RPC", () => { + const parsed = parseFlowPulseLogFixture(fixture.rawLog); + assert.equal(parsed.observationId, fixture.expected.observationId); + assert.equal(parsed.lifecycleState, "observed"); + assert.equal(parsed.pulseId, fixture.reportCore.observation.pulseId); + assert.equal(parsed.rootfieldId, fixture.reportCore.observation.rootfieldId); + assert.equal(parsed.actor, fixture.reportCore.flowPulse.actor); + assert.equal(parsed.pulseType, fixture.reportCore.flowPulse.pulseType); + assert.equal(parsed.subject, fixture.reportCore.flowPulse.subject); + assert.equal(parsed.commitment, fixture.reportCore.flowPulse.commitment); + assert.equal(parsed.sequence, fixture.reportCore.flowPulse.sequence); + assert.equal(parsed.occurredAt, fixture.reportCore.flowPulse.occurredAt); + assert.equal(parsed.uri, fixture.reportCore.flowPulse.uri); +}); + +test("rejects non-FlowPulse fixture logs", () => { + const badLog = structuredClone(fixture.rawLog); + badLog.topics[0] = "0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff"; + assert.throws(() => parseFlowPulseLogFixture(badLog), /expected 32 bytes|unsupported event signature/); +}); + +test("normalizes EVM addresses and rejects malformed addresses", () => { + assert.equal( + normalizeAddress("0xAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"), + "0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + ); + assert.throws(() => normalizeAddress("0x1234"), /expected 20 bytes/); +}); + +test("exposes the v0 verifier status vocabulary", () => { + assert.deepEqual(VERIFIER_STATUSES, [ + "valid", + "invalid", + "unresolved", + "unsupported", + "reorged", + ]); + assert.equal(isVerifierStatus("valid"), true); + assert.equal(isVerifierStatus("verified"), false); +}); + +test("canonical JSON sorts object keys while preserving array order", () => { + assert.equal(canonicalJson({ b: "2", a: { d: "4", c: "3" } }), '{"a":{"c":"3","d":"4"},"b":"2"}'); + assert.equal(canonicalJson(["b", "a"]), '["b","a"]'); +}); + +test("derives deterministic verifier report id from canonical report core", () => { + assert.equal(deriveReportId(fixture.reportCore), fixture.expected.reportId); +}); diff --git a/services/shared/tsconfig.json b/services/shared/tsconfig.json new file mode 100644 index 00000000..13b3dd7d --- /dev/null +++ b/services/shared/tsconfig.json @@ -0,0 +1,4 @@ +{ + "extends": "../tsconfig.base.json", + "include": ["src/**/*.ts", "test/**/*.ts"] +} diff --git a/services/tsconfig.base.json b/services/tsconfig.base.json new file mode 100644 index 00000000..1c43fc26 --- /dev/null +++ b/services/tsconfig.base.json @@ -0,0 +1,12 @@ +{ + "compilerOptions": { + "target": "ES2024", + "module": "NodeNext", + "moduleResolution": "NodeNext", + "strict": true, + "noEmit": true, + "skipLibCheck": true, + "allowImportingTsExtensions": true, + "types": ["node"] + } +} diff --git a/services/verifier/README.md b/services/verifier/README.md index 2be275e2..a7811c4d 100644 --- a/services/verifier/README.md +++ b/services/verifier/README.md @@ -1,82 +1,142 @@ -# FlowMemory Verifier MVP +# FlowMemory Verifier V0 -Status: specification draft. +This package is a local, fixture-first verifier. It consumes indexed FlowPulse observations, resolves only local fixture artifacts, applies deterministic commitment checks, and writes canonical verification reports. -The verifier MVP reads FlowPulse observations, reconstructs commitments, checks evidence, and emits deterministic reports. It does not run a production service, proof network, or token system. +It is not a verifier network, production proof service, token system, staking system, or full trustless verifier layer. -## Inputs +## Commands -- decoded FlowPulse log -- receipt/log metadata used to derive `observationId` -- artifact manifest and Merkle openings, when required -- storage receipt commitment openings, when required -- worker signature envelope, when required -- verifier policy document hash +From the repository root: -## Import Strategy +```powershell +npm run verify:fixtures +npm run demo:verifier +npm test --prefix services/verifier +``` -There is no `services/shared/` package yet. Until one exists, verifier code should import or mirror the package under `crypto/`: +`npm run verify:fixtures` reads `services/indexer/out/indexer-state.json` when present. If the indexer output is missing, it builds indexer state from fixtures. It writes: -```js -import { - flowPulseObservationId, - flowPulseEventArgsHash, - receiptHash, - verifierReportHash, - verifierSignaturePayload, - verifyDigest -} from "../../crypto/src/index.js"; +```text +services/verifier/out/reports.json ``` -If the verifier cannot import directly, it must run equivalent tests against `crypto/fixtures/`, `crypto/fixtures/vectors.json`, and `crypto/test-vectors/`. The compatibility gate is: +Use custom paths: ```powershell -cd E:\FlowMemory\flowmemory-crypto\crypto -npm test -npm run validate:vectors -python validate_test_vectors.py +npm run verify:fixtures -- --input ../indexer/out/indexer-state.json --out out/custom-reports.json +``` + +## Resolver Policy + +V0 uses local fixture resolver policy: + +```text +flowmemory.resolver.policy.v0.fixture +``` + +Artifact fixtures live at: + +```text +services/verifier/fixtures/artifacts.json +``` + +The verifier does not fetch arbitrary HTTP or IPFS content. `uri` is advisory and only acts as a lookup key inside the local fixture resolver. The resolver supports an optional maximum artifact byte size and returns `unresolved` when evidence is missing or policy-rejected. + +## Report Statuses + +The verifier report package uses these deterministic report statuses: + +- `valid`: supported checks passed. +- `invalid`: supported checks ran and at least one required check failed. +- `unresolved`: required fixture evidence is missing or policy-rejected. +- `unsupported`: pulse type or artifact semantics are outside V0 rules. +- `reorged`: observation is removed or reorged. + +The status vocabulary is documented in [VERIFIER_STATUS_VOCABULARY.md](./VERIFIER_STATUS_VOCABULARY.md). + +## Flow Memory Status Mapping + +Flow Memory and dashboard surfaces may use the launch vocabulary: + +- `verified` +- `unresolved` +- `unsupported` +- `failed` +- `reorged` + +The V0 adapter rule is: + +- verifier `valid` maps to Flow Memory `verified`. +- verifier `invalid` maps to Flow Memory `failed`. +- verifier `unresolved`, `unsupported`, and `reorged` keep the same meaning. + +Do not silently collapse `unresolved` into `failed`; missing evidence and failed evidence are different states. + +## Commitment Checks + +For `ROOTFIELD_REGISTERED` (`pulseType = 1`): + +```text +subject == rootfieldId +commitment == keccak256(abi.encode(schemaHash, metadataHash)) +``` + +For `ROOT_COMMITTED` (`pulseType = 2`): + +```text +subject == root +commitment == keccak256(abi.encode(root, artifactCommitment)) +``` + +Unsupported pulse types return `unsupported`. Missing artifacts return `unresolved`. Subject or commitment mismatches return `invalid`. + +## Reports + +Each report has: + +- `reportId` +- `reportDigest` +- `reportCore` + +`reportId` and `reportDigest` are the same V0 digest: + +```text +keccak256(canonical_json(reportCore)) ``` -## Deterministic Report Flow +The digest excludes wall-clock timestamps, local file paths, signatures, operator notes, and other mutable presentation data. + +The JSON schema fixture lives at: -1. Recompute `observationId`. -2. Recompute `eventArgsHash`. -3. Recompute `receiptHash`. -4. Recompute artifact root and storage commitment if openings are supplied. -5. Verify worker signatures if policy requires them. -6. Evaluate finality and reorg status. -7. Produce canonical checks JSON or a checks Merkle root. -8. Produce `reportId`. -9. Sign `reportId` with the verifier signature envelope. +```text +services/verifier/fixtures/verification-report.schema.json +``` -## Status Vocabulary +## Persistence + +The persisted report file declares: ```text -0 = reserved -1 = observed -2 = verified -3 = unresolved -4 = unsupported -5 = failed -6 = reorged -7 = superseded +flowmemory.verifier.persistence.v0 ``` -Minimum requirements: +Each report core declares: -- `observed`: receipt/log parsed and observation id computed. -- `verified`: all policy-required checks passed. -- `unresolved`: required evidence is missing or unavailable. -- `unsupported`: schema, pulse type, root scheme, or key type is unknown. -- `failed`: a policy-required check failed. -- `reorged`: block/log is not canonical under finality policy. -- `superseded`: a newer report intentionally replaces this report. +```text +flowmemory.verifier.report.v0 +``` + +JSON output is deterministic across repeated runs with the same fixtures. ## Non-Goals -- No live RPC integration in this spec. -- No database schema. -- No API server. -- No zk proof implementation. - No verifier economics. -- No tokenomics. +- No staking, rewards, or slashing. +- No production verifier network. +- No live artifact fetching. +- No production database. +- No report signing or attestations yet. +- No zk proof implementation. +- No API server. + +See [docs/INDEXER_VERIFIER_MVP.md](../../docs/INDEXER_VERIFIER_MVP.md) for the full local pipeline. diff --git a/services/verifier/VERIFIER_STATUS_VOCABULARY.md b/services/verifier/VERIFIER_STATUS_VOCABULARY.md new file mode 100644 index 00000000..d73239c3 --- /dev/null +++ b/services/verifier/VERIFIER_STATUS_VOCABULARY.md @@ -0,0 +1,126 @@ +# Verifier Status Vocabulary + +This document defines the V0 verifier report statuses used by the runnable fixture package. + +## V0 Report Statuses + +### valid + +Meaning: Supported deterministic checks passed against allowed fixture evidence. + +Minimum evidence: + +- Complete indexed observation. +- Observation is not `removed` or `reorged`. +- Pulse type has a V0 verifier rule. +- Required fixture artifact exists under the resolver policy. +- Subject and commitment checks pass. + +### invalid + +Meaning: Supported deterministic checks ran and at least one required check failed. + +Minimum evidence: + +- Complete indexed observation. +- Pulse type has a V0 verifier rule. +- Required fixture artifact exists. +- A deterministic mismatch was found. + +Typical reason codes: + +- `subject.mismatch` +- `commitment.mismatch` +- `artifact.schema_mismatch` + +### unresolved + +Meaning: Required evidence is missing or rejected by resolver policy. + +Minimum evidence: + +- Complete indexed observation. +- Resolver policy produced a deterministic failure reason. + +Typical reason codes: + +- `artifact.unavailable` +- `artifact.too_large` + +### unsupported + +Meaning: The verifier cannot evaluate this observation under V0 rules. + +Minimum evidence: + +- Observation was decoded far enough to identify unsupported semantics. + +Typical reason codes: + +- `pulse.type.unsupported` + +### reorged + +Meaning: The observation is removed or no longer canonical. + +Minimum evidence: + +- Indexer lifecycle state is `removed` or `reorged`. + +Typical reason codes: + +- `observation.reorged` + +## Status Selection Order + +When multiple statuses might apply, V0 chooses: + +1. `reorged` +2. `unsupported` +3. `unresolved` +4. `invalid` +5. `valid` + +Reorged observations should not become `valid`. Unsupported pulse types should not be marked unresolved merely because their URI has no fixture artifact. + +## Compatibility With Earlier Planning Terms + +Issue #14 originally used a broader vocabulary: + +- `observed` +- `verified` +- `unresolved` +- `unsupported` +- `failed` +- `reorged` +- `stale` +- `disputed` +- `superseded` + +For the runnable V0 report schema: + +- `verified` maps to `valid`. +- `failed` maps to `invalid`. +- `observed` is an indexer/display state, not a verifier result claim. +- `stale`, `disputed`, and `superseded` are future report lifecycle overlays, not V0 terminal report statuses. + +The broader terms can return later as dashboard or attestation lifecycle fields without changing the V0 report result enum. + +## Report Digest Rule + +The `reportId` is: + +```text +keccak256(canonical_json(reportCore)) +``` + +`reportCore.status` is included in the digest. Wall-clock timestamps, local paths, signatures, operator notes, and presentation metadata are excluded. + +## Non-Goals + +- No verifier rewards. +- No slashing. +- No proof network. +- No production API. +- No production database schema. +- No report signing or attestation envelope yet. diff --git a/services/verifier/fixtures/artifacts.json b/services/verifier/fixtures/artifacts.json new file mode 100644 index 00000000..56040f8e --- /dev/null +++ b/services/verifier/fixtures/artifacts.json @@ -0,0 +1,15 @@ +{ + "resolverPolicyId": "flowmemory.resolver.policy.v0.fixture", + "artifactsByUri": { + "ipfs://bafy-flowmemory-example": { + "kind": "rootfield-registration", + "schemaHash": "0xdddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd", + "metadataHash": "0xeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee" + }, + "fixture://root-commit-valid": { + "kind": "root-commitment", + "root": "0xfffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff1", + "artifactCommitment": "0x9999999999999999999999999999999999999999999999999999999999999999" + } + } +} diff --git a/services/verifier/fixtures/verification-report.schema.json b/services/verifier/fixtures/verification-report.schema.json new file mode 100644 index 00000000..885b83ef --- /dev/null +++ b/services/verifier/fixtures/verification-report.schema.json @@ -0,0 +1,39 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "flowmemory.verifier.report.v0", + "type": "object", + "required": ["reportId", "reportDigest", "reportCore"], + "properties": { + "reportId": { "type": "string", "pattern": "^0x[0-9a-f]{64}$" }, + "reportDigest": { "type": "string", "pattern": "^0x[0-9a-f]{64}$" }, + "reportCore": { + "type": "object", + "required": [ + "schema", + "verifierSpecVersion", + "resolverPolicyId", + "status", + "observationId", + "observation", + "flowPulse", + "checks", + "evidenceRefs", + "reasonCodes" + ], + "properties": { + "schema": { "const": "flowmemory.verifier.report.v0" }, + "verifierSpecVersion": { "type": "string" }, + "resolverPolicyId": { "type": "string" }, + "status": { "enum": ["valid", "invalid", "unresolved", "unsupported", "reorged"] }, + "observationId": { "type": "string", "pattern": "^0x[0-9a-f]{64}$" }, + "observation": { "type": "object" }, + "flowPulse": { "type": "object" }, + "checks": { "type": "array" }, + "evidenceRefs": { "type": "array" }, + "reasonCodes": { "type": "array", "items": { "type": "string" } } + }, + "additionalProperties": false + } + }, + "additionalProperties": false +} diff --git a/services/verifier/out/reports.json b/services/verifier/out/reports.json new file mode 100644 index 00000000..f813d04d --- /dev/null +++ b/services/verifier/out/reports.json @@ -0,0 +1 @@ +{"reports":[{"reportCore":{"checks":[{"id":"observation.decoded","passed":true},{"id":"subject.rootfield_matches","passed":true},{"id":"commitment.rootfield_registration","passed":true}],"evidenceRefs":[{"kind":"rootfield-registration","uri":"ipfs://bafy-flowmemory-example"}],"flowPulse":{"actor":"0x4444444444444444444444444444444444444444","commitment":"0x4122209ff672fc04b2ec3af31ab1af79813971f86de000aa6534038cc79de6b5","occurredAt":"1778640000","parentPulseId":"0x0000000000000000000000000000000000000000000000000000000000000000","pulseType":"1","sequence":"1","subject":"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb","uri":"ipfs://bafy-flowmemory-example"},"observation":{"blockHash":"0x2222222222222222222222222222222222222222222222222222222222222222","blockNumber":"123456","chainId":"8453","emittingContract":"0x1111111111111111111111111111111111111111","eventSignature":"0x5d07190b9ae441b4d7b16259a48424acd451492b12f5f99a29f5bfd992c13e43","logIndex":"2","pulseId":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","rootfieldId":"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb","transactionIndex":"7","txHash":"0x3333333333333333333333333333333333333333333333333333333333333333"},"observationId":"0x9d958aadf8bf46f989b51e541709a73d21970e7e79643f939c9a0000b50f9a91","reasonCodes":[],"resolverPolicyId":"flowmemory.resolver.policy.v0.fixture","schema":"flowmemory.verifier.report.v0","status":"valid","verifierSpecVersion":"0"},"reportDigest":"0x03af9720a87f7d72ce2ce95c1d04d50134649da51ace802d88818ad67fb2ebb3","reportId":"0x03af9720a87f7d72ce2ce95c1d04d50134649da51ace802d88818ad67fb2ebb3"},{"reportCore":{"checks":[{"id":"observation.decoded","passed":true},{"id":"subject.root_matches","passed":true},{"id":"commitment.root","passed":true}],"evidenceRefs":[{"kind":"root-commitment","uri":"fixture://root-commit-valid"}],"flowPulse":{"actor":"0x4444444444444444444444444444444444444444","commitment":"0x69a9f953179a6ce2e08724dd759fb71b08b0759a5900fc0399a753ecf0557df5","occurredAt":"1778640060","parentPulseId":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","pulseType":"2","sequence":"2","subject":"0xfffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff1","uri":"fixture://root-commit-valid"},"observation":{"blockHash":"0x2222222222222222222222222222222222222222222222222222222222222223","blockNumber":"123457","chainId":"8453","emittingContract":"0x1111111111111111111111111111111111111111","eventSignature":"0x5d07190b9ae441b4d7b16259a48424acd451492b12f5f99a29f5bfd992c13e43","logIndex":"3","pulseId":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab001","rootfieldId":"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb","transactionIndex":"8","txHash":"0x3333333333333333333333333333333333333333333333333333333333333334"},"observationId":"0x49c6cd59d1f1916bc5301308be09c14d64127d1566362272dc5aa201ed53bdea","reasonCodes":[],"resolverPolicyId":"flowmemory.resolver.policy.v0.fixture","schema":"flowmemory.verifier.report.v0","status":"valid","verifierSpecVersion":"0"},"reportDigest":"0xf1dfced6038cfa79928e1888e611da6bb05e7a14393cb20e2d5a5cb90c825c35","reportId":"0xf1dfced6038cfa79928e1888e611da6bb05e7a14393cb20e2d5a5cb90c825c35"},{"reportCore":{"checks":[{"id":"observation.decoded","passed":true},{"id":"subject.rootfield_matches","passed":true},{"id":"commitment.rootfield_registration","passed":true}],"evidenceRefs":[{"kind":"rootfield-registration","uri":"ipfs://bafy-flowmemory-example"}],"flowPulse":{"actor":"0x4444444444444444444444444444444444444444","commitment":"0x4122209ff672fc04b2ec3af31ab1af79813971f86de000aa6534038cc79de6b5","occurredAt":"1778640000","parentPulseId":"0x0000000000000000000000000000000000000000000000000000000000000000","pulseType":"1","sequence":"1","subject":"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb","uri":"ipfs://bafy-flowmemory-example"},"observation":{"blockHash":"0x2222222222222222222222222222222222222222222222222222222222222222","blockNumber":"123456","chainId":"8453","emittingContract":"0x1111111111111111111111111111111111111111","eventSignature":"0x5d07190b9ae441b4d7b16259a48424acd451492b12f5f99a29f5bfd992c13e43","logIndex":"2","pulseId":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","rootfieldId":"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb","transactionIndex":"7","txHash":"0x3333333333333333333333333333333333333333333333333333333333333333"},"observationId":"0x9d958aadf8bf46f989b51e541709a73d21970e7e79643f939c9a0000b50f9a91","reasonCodes":[],"resolverPolicyId":"flowmemory.resolver.policy.v0.fixture","schema":"flowmemory.verifier.report.v0","status":"valid","verifierSpecVersion":"0"},"reportDigest":"0x03af9720a87f7d72ce2ce95c1d04d50134649da51ace802d88818ad67fb2ebb3","reportId":"0x03af9720a87f7d72ce2ce95c1d04d50134649da51ace802d88818ad67fb2ebb3"},{"reportCore":{"checks":[{"id":"observation.decoded","passed":true}],"evidenceRefs":[],"flowPulse":{"actor":"0x4444444444444444444444444444444444444444","commitment":"0x69a9f953179a6ce2e08724dd759fb71b08b0759a5900fc0399a753ecf0557df5","occurredAt":"1778640060","parentPulseId":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","pulseType":"2","sequence":"2","subject":"0xfffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff1","uri":"fixture://root-commit-valid"},"observation":{"blockHash":"0x2222222222222222222222222222222222222222222222222222222222222224","blockNumber":"123458","chainId":"8453","emittingContract":"0x1111111111111111111111111111111111111111","eventSignature":"0x5d07190b9ae441b4d7b16259a48424acd451492b12f5f99a29f5bfd992c13e43","logIndex":"4","pulseId":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab002","rootfieldId":"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb","transactionIndex":"9","txHash":"0x3333333333333333333333333333333333333333333333333333333333333335"},"observationId":"0x223d74c971f301d800dd69aa30994bc3fa3089b34b0db1b0a7fc9d4e8d114c79","reasonCodes":["observation.reorged"],"resolverPolicyId":"flowmemory.resolver.policy.v0.fixture","schema":"flowmemory.verifier.report.v0","status":"reorged","verifierSpecVersion":"0"},"reportDigest":"0xb7cef189a48b8c2697603c3704a52bb3a0322b375fdbea68d2bbbadb65a92ec1","reportId":"0xb7cef189a48b8c2697603c3704a52bb3a0322b375fdbea68d2bbbadb65a92ec1"},{"reportCore":{"checks":[{"id":"observation.decoded","passed":true},{"id":"subject.root_matches","passed":true},{"id":"commitment.root","passed":false}],"evidenceRefs":[{"kind":"root-commitment","uri":"fixture://root-commit-valid"}],"flowPulse":{"actor":"0x4444444444444444444444444444444444444444","commitment":"0xcccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc","occurredAt":"1778640120","parentPulseId":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","pulseType":"2","sequence":"3","subject":"0xfffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff1","uri":"fixture://root-commit-valid"},"observation":{"blockHash":"0x2222222222222222222222222222222222222222222222222222222222222225","blockNumber":"123459","chainId":"8453","emittingContract":"0x1111111111111111111111111111111111111111","eventSignature":"0x5d07190b9ae441b4d7b16259a48424acd451492b12f5f99a29f5bfd992c13e43","logIndex":"5","pulseId":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab003","rootfieldId":"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb","transactionIndex":"10","txHash":"0x3333333333333333333333333333333333333333333333333333333333333336"},"observationId":"0xe4a7065f1578c4a232b41f5984942e9b7b07760ce7dfeba77b38eb03173491df","reasonCodes":["commitment.mismatch"],"resolverPolicyId":"flowmemory.resolver.policy.v0.fixture","schema":"flowmemory.verifier.report.v0","status":"invalid","verifierSpecVersion":"0"},"reportDigest":"0xabc22e2203a9404c832aa792f7a569e17e599078fc9ebacf2c1307bbf6f1d7bb","reportId":"0xabc22e2203a9404c832aa792f7a569e17e599078fc9ebacf2c1307bbf6f1d7bb"},{"reportCore":{"checks":[{"id":"observation.decoded","passed":true}],"evidenceRefs":[],"flowPulse":{"actor":"0x4444444444444444444444444444444444444444","commitment":"0x69a9f953179a6ce2e08724dd759fb71b08b0759a5900fc0399a753ecf0557df5","occurredAt":"1778640180","parentPulseId":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","pulseType":"2","sequence":"4","subject":"0xfffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff1","uri":"fixture://missing-artifact"},"observation":{"blockHash":"0x2222222222222222222222222222222222222222222222222222222222222226","blockNumber":"123460","chainId":"8453","emittingContract":"0x1111111111111111111111111111111111111111","eventSignature":"0x5d07190b9ae441b4d7b16259a48424acd451492b12f5f99a29f5bfd992c13e43","logIndex":"6","pulseId":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab004","rootfieldId":"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb","transactionIndex":"11","txHash":"0x3333333333333333333333333333333333333333333333333333333333333337"},"observationId":"0x92144fc24c81cdd6319598e6e0c58d84e3f18d9ad4a8be33bc956e32c2ae39f4","reasonCodes":["artifact.unavailable"],"resolverPolicyId":"flowmemory.resolver.policy.v0.fixture","schema":"flowmemory.verifier.report.v0","status":"unresolved","verifierSpecVersion":"0"},"reportDigest":"0xb3798cf34e65a4af4017fe312cd60646a730fd74dbf36ea8d3a5b8b88502e294","reportId":"0xb3798cf34e65a4af4017fe312cd60646a730fd74dbf36ea8d3a5b8b88502e294"},{"reportCore":{"checks":[{"id":"observation.decoded","passed":true}],"evidenceRefs":[],"flowPulse":{"actor":"0x4444444444444444444444444444444444444444","commitment":"0x69a9f953179a6ce2e08724dd759fb71b08b0759a5900fc0399a753ecf0557df5","occurredAt":"1778640240","parentPulseId":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa","pulseType":"99","sequence":"5","subject":"0xfffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff1","uri":"fixture://unsupported"},"observation":{"blockHash":"0x2222222222222222222222222222222222222222222222222222222222222227","blockNumber":"123461","chainId":"8453","emittingContract":"0x1111111111111111111111111111111111111111","eventSignature":"0x5d07190b9ae441b4d7b16259a48424acd451492b12f5f99a29f5bfd992c13e43","logIndex":"7","pulseId":"0xaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab005","rootfieldId":"0xbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb","transactionIndex":"12","txHash":"0x3333333333333333333333333333333333333333333333333333333333333338"},"observationId":"0x5ba8c5a70c814d35482df5f6598e549f83f2d0e27f6e3d25b83eb2a0e1d56a98","reasonCodes":["pulse.type.unsupported"],"resolverPolicyId":"flowmemory.resolver.policy.v0.fixture","schema":"flowmemory.verifier.report.v0","status":"unsupported","verifierSpecVersion":"0"},"reportDigest":"0x61b984c1055654334a7a7a191887779c8f2be937b08590e8a90927ee645042f5","reportId":"0x61b984c1055654334a7a7a191887779c8f2be937b08590e8a90927ee645042f5"}],"schema":"flowmemory.verifier.persistence.v0"} diff --git a/services/verifier/package.json b/services/verifier/package.json new file mode 100644 index 00000000..7b4f00d9 --- /dev/null +++ b/services/verifier/package.json @@ -0,0 +1,10 @@ +{ + "name": "@flowmemory/verifier-v0", + "private": true, + "type": "module", + "scripts": { + "demo": "node src/demo.ts", + "verify:fixtures": "node src/verify-fixtures.ts", + "test": "node --test test/*.test.ts" + } +} diff --git a/services/verifier/src/demo.ts b/services/verifier/src/demo.ts new file mode 100644 index 00000000..591188e9 --- /dev/null +++ b/services/verifier/src/demo.ts @@ -0,0 +1,17 @@ +import { indexFlowPulseReceipts } from "../../indexer/src/index.ts"; +import { loadIndexerFixtureReceipts } from "../../indexer/src/fixtures.ts"; +import { loadVerifierArtifactFixture } from "./fixtures.ts"; +import { verifyObservations } from "./verifier.ts"; + +const indexerState = indexFlowPulseReceipts(loadIndexerFixtureReceipts(), { + finalizedBlockNumber: "123458", +}); +const reports = verifyObservations(indexerState.observations, loadVerifierArtifactFixture()); + +console.log(JSON.stringify({ + service: "flowmemory-verifier-v0", + mode: "fixture", + reportCount: reports.length, + statuses: reports.map((report) => report.reportCore.status), + reports, +}, null, 2)); diff --git a/services/verifier/src/fixtures.ts b/services/verifier/src/fixtures.ts new file mode 100644 index 00000000..e70af14a --- /dev/null +++ b/services/verifier/src/fixtures.ts @@ -0,0 +1,12 @@ +import { readFileSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; + +import type { ArtifactResolverFixture } from "./verifier.ts"; + +const __dirname = dirname(fileURLToPath(import.meta.url)); + +export function loadVerifierArtifactFixture(): ArtifactResolverFixture { + const path = join(__dirname, "../fixtures/artifacts.json"); + return JSON.parse(readFileSync(path, "utf8")) as ArtifactResolverFixture; +} diff --git a/services/verifier/src/index.ts b/services/verifier/src/index.ts new file mode 100644 index 00000000..203122b9 --- /dev/null +++ b/services/verifier/src/index.ts @@ -0,0 +1,3 @@ +export * from "./fixtures.ts"; +export * from "./persistence.ts"; +export * from "./verifier.ts"; diff --git a/services/verifier/src/persistence.ts b/services/verifier/src/persistence.ts new file mode 100644 index 00000000..472c5598 --- /dev/null +++ b/services/verifier/src/persistence.ts @@ -0,0 +1,26 @@ +import { mkdirSync, readFileSync, writeFileSync } from "node:fs"; +import { dirname } from "node:path"; + +import { canonicalJson } from "../../shared/src/index.ts"; +import type { VerifierReport } from "./verifier.ts"; + +export interface PersistedVerifierReports { + schema: "flowmemory.verifier.persistence.v0"; + reports: VerifierReport[]; +} + +export function persistedVerifierReports(reports: VerifierReport[]): PersistedVerifierReports { + return { + schema: "flowmemory.verifier.persistence.v0", + reports, + }; +} + +export function writeVerifierReports(path: string, reports: VerifierReport[]): void { + mkdirSync(dirname(path), { recursive: true }); + writeFileSync(path, `${canonicalJson(persistedVerifierReports(reports))}\n`, "utf8"); +} + +export function readVerifierReports(path: string): PersistedVerifierReports { + return JSON.parse(readFileSync(path, "utf8")) as PersistedVerifierReports; +} diff --git a/services/verifier/src/verifier.ts b/services/verifier/src/verifier.ts new file mode 100644 index 00000000..33622a10 --- /dev/null +++ b/services/verifier/src/verifier.ts @@ -0,0 +1,238 @@ +import { + VERIFIER_REPORT_SCHEMA, + deriveReportId, + encodeBytes32, + keccak256Hex, + normalizeBytes32, + type VerifierReportCore, + type VerifierStatus, +} from "../../shared/src/index.ts"; + +export interface VerifiableObservation { + observationId: string; + lifecycleState?: string; + chainId: string; + emittingContract: string; + eventSignature: string; + blockNumber: string; + blockHash: string; + txHash: string; + transactionIndex: string; + logIndex: string; + pulseId: string; + rootfieldId: string; + actor: string; + pulseType: string; + subject: string; + commitment: string; + parentPulseId: string; + sequence: string; + occurredAt: string; + uri: string; +} + +export interface RootfieldRegistrationArtifact { + kind: "rootfield-registration"; + schemaHash: string; + metadataHash: string; +} + +export interface RootCommitmentArtifact { + kind: "root-commitment"; + root: string; + artifactCommitment: string; +} + +export type VerifierArtifact = RootfieldRegistrationArtifact | RootCommitmentArtifact; + +export interface ArtifactResolverFixture { + resolverPolicyId: string; + maxArtifactBytes?: number; + artifactsByUri: Record; +} + +export interface VerifierReport { + reportId: string; + reportDigest: string; + reportCore: VerifierReportCore; +} + +function concatBytes(parts: Uint8Array[]): Uint8Array { + const output = new Uint8Array(parts.reduce((sum, part) => sum + part.length, 0)); + let offset = 0; + for (const part of parts) { + output.set(part, offset); + offset += part.length; + } + return output; +} + +export function rootfieldRegistrationCommitment(artifact: RootfieldRegistrationArtifact): `0x${string}` { + return keccak256Hex(concatBytes([ + encodeBytes32(artifact.schemaHash), + encodeBytes32(artifact.metadataHash), + ])); +} + +export function rootCommitment(artifact: RootCommitmentArtifact): `0x${string}` { + return keccak256Hex(concatBytes([ + encodeBytes32(artifact.root), + encodeBytes32(artifact.artifactCommitment), + ])); +} + +function baseReportCore( + observation: VerifiableObservation, + resolverPolicyId: string, + status: VerifierStatus, + checks: Array>, + evidenceRefs: Array>, + reasonCodes: string[], +): VerifierReportCore { + return { + schema: VERIFIER_REPORT_SCHEMA, + verifierSpecVersion: "0", + resolverPolicyId, + status, + observationId: observation.observationId, + observation: { + chainId: observation.chainId, + emittingContract: observation.emittingContract, + eventSignature: observation.eventSignature, + blockNumber: observation.blockNumber, + blockHash: observation.blockHash, + txHash: observation.txHash, + transactionIndex: observation.transactionIndex, + logIndex: observation.logIndex, + pulseId: observation.pulseId, + rootfieldId: observation.rootfieldId, + }, + flowPulse: { + actor: observation.actor, + pulseType: observation.pulseType, + subject: observation.subject, + commitment: observation.commitment, + parentPulseId: observation.parentPulseId, + sequence: observation.sequence, + occurredAt: observation.occurredAt, + uri: observation.uri, + }, + checks, + evidenceRefs, + reasonCodes, + }; +} + +function finalizeReport(reportCore: VerifierReportCore): VerifierReport { + const reportDigest = deriveReportId(reportCore); + return { + reportId: reportDigest, + reportDigest, + reportCore, + }; +} + +function artifactSize(artifact: VerifierArtifact): number { + return new TextEncoder().encode(JSON.stringify(artifact)).byteLength; +} + +export function verifyObservation( + observation: VerifiableObservation, + resolver: ArtifactResolverFixture, +): VerifierReport { + const checks: Array> = [ + { id: "observation.decoded", passed: true }, + ]; + const evidenceRefs: Array> = []; + const reasonCodes: string[] = []; + + if (observation.lifecycleState === "reorged" || observation.lifecycleState === "removed") { + reasonCodes.push("observation.reorged"); + return finalizeReport(baseReportCore(observation, resolver.resolverPolicyId, "reorged", checks, evidenceRefs, reasonCodes)); + } + + if (observation.pulseType !== "1" && observation.pulseType !== "2") { + reasonCodes.push("pulse.type.unsupported"); + return finalizeReport(baseReportCore(observation, resolver.resolverPolicyId, "unsupported", checks, evidenceRefs, reasonCodes)); + } + + const artifact = resolver.artifactsByUri[observation.uri]; + if (artifact === undefined) { + reasonCodes.push("artifact.unavailable"); + return finalizeReport(baseReportCore(observation, resolver.resolverPolicyId, "unresolved", checks, evidenceRefs, reasonCodes)); + } + + evidenceRefs.push({ uri: observation.uri, kind: artifact.kind }); + if (resolver.maxArtifactBytes !== undefined && artifactSize(artifact) > resolver.maxArtifactBytes) { + reasonCodes.push("artifact.too_large"); + return finalizeReport(baseReportCore(observation, resolver.resolverPolicyId, "unresolved", checks, evidenceRefs, reasonCodes)); + } + + if (observation.pulseType === "1") { + if (artifact.kind !== "rootfield-registration") { + reasonCodes.push("artifact.schema_mismatch"); + return finalizeReport(baseReportCore(observation, resolver.resolverPolicyId, "invalid", checks, evidenceRefs, reasonCodes)); + } + + const expectedCommitment = rootfieldRegistrationCommitment(artifact); + const subjectMatches = normalizeBytes32(observation.subject) === normalizeBytes32(observation.rootfieldId); + const commitmentMatches = normalizeBytes32(observation.commitment) === expectedCommitment; + checks.push({ id: "subject.rootfield_matches", passed: subjectMatches }); + checks.push({ id: "commitment.rootfield_registration", passed: commitmentMatches }); + + if (!subjectMatches) { + reasonCodes.push("subject.mismatch"); + } + if (!commitmentMatches) { + reasonCodes.push("commitment.mismatch"); + } + + return finalizeReport(baseReportCore( + observation, + resolver.resolverPolicyId, + reasonCodes.length === 0 ? "valid" : "invalid", + checks, + evidenceRefs, + reasonCodes, + )); + } + + if (observation.pulseType === "2") { + if (artifact.kind !== "root-commitment") { + reasonCodes.push("artifact.schema_mismatch"); + return finalizeReport(baseReportCore(observation, resolver.resolverPolicyId, "invalid", checks, evidenceRefs, reasonCodes)); + } + + const expectedCommitment = rootCommitment(artifact); + const subjectMatches = normalizeBytes32(observation.subject) === normalizeBytes32(artifact.root); + const commitmentMatches = normalizeBytes32(observation.commitment) === expectedCommitment; + checks.push({ id: "subject.root_matches", passed: subjectMatches }); + checks.push({ id: "commitment.root", passed: commitmentMatches }); + + if (!subjectMatches) { + reasonCodes.push("subject.mismatch"); + } + if (!commitmentMatches) { + reasonCodes.push("commitment.mismatch"); + } + + return finalizeReport(baseReportCore( + observation, + resolver.resolverPolicyId, + reasonCodes.length === 0 ? "valid" : "invalid", + checks, + evidenceRefs, + reasonCodes, + )); + } + + reasonCodes.push("pulse.type.unsupported"); + return finalizeReport(baseReportCore(observation, resolver.resolverPolicyId, "unsupported", checks, evidenceRefs, reasonCodes)); +} + +export function verifyObservations( + observations: VerifiableObservation[], + resolver: ArtifactResolverFixture, +): VerifierReport[] { + return observations.map((observation) => verifyObservation(observation, resolver)); +} diff --git a/services/verifier/src/verify-fixtures.ts b/services/verifier/src/verify-fixtures.ts new file mode 100644 index 00000000..7737d089 --- /dev/null +++ b/services/verifier/src/verify-fixtures.ts @@ -0,0 +1,31 @@ +import { existsSync } from "node:fs"; +import { resolve } from "node:path"; + +import { indexFlowPulseReceipts, readIndexerState } from "../../indexer/src/index.ts"; +import { loadIndexerFixtureReceipts } from "../../indexer/src/fixtures.ts"; +import { loadVerifierArtifactFixture } from "./fixtures.ts"; +import { verifyObservations } from "./verifier.ts"; +import { writeVerifierReports } from "./persistence.ts"; + +const outArgIndex = process.argv.indexOf("--out"); +const inputArgIndex = process.argv.indexOf("--input"); +const outputPath = outArgIndex >= 0 ? process.argv[outArgIndex + 1] : "out/reports.json"; +const inputPath = inputArgIndex >= 0 ? process.argv[inputArgIndex + 1] : "../indexer/out/indexer-state.json"; + +const indexerState = existsSync(inputPath) + ? readIndexerState(inputPath).state + : indexFlowPulseReceipts(loadIndexerFixtureReceipts(), { finalizedBlockNumber: "123458" }); + +const reports = verifyObservations(indexerState.observations, loadVerifierArtifactFixture()); +writeVerifierReports(outputPath, reports); + +console.log(JSON.stringify({ + service: "flowmemory-verifier-v0", + inputPath: resolve(inputPath), + outputPath: resolve(outputPath), + reports: reports.length, + statuses: reports.reduce>((counts, report) => { + counts[report.reportCore.status] = (counts[report.reportCore.status] ?? 0) + 1; + return counts; + }, {}), +}, null, 2)); diff --git a/services/verifier/test/verifier.test.ts b/services/verifier/test/verifier.test.ts new file mode 100644 index 00000000..c42bc401 --- /dev/null +++ b/services/verifier/test/verifier.test.ts @@ -0,0 +1,109 @@ +import assert from "node:assert/strict"; +import { mkdtempSync, readFileSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import test from "node:test"; + +import { indexFlowPulseLogs, indexFlowPulseReceipts } from "../../indexer/src/index.ts"; +import { loadIndexerFixtureLogs, loadIndexerFixtureReceipts } from "../../indexer/src/fixtures.ts"; +import { loadVerifierArtifactFixture } from "../src/fixtures.ts"; +import { readVerifierReports, writeVerifierReports } from "../src/persistence.ts"; +import { verifyObservation, verifyObservations } from "../src/verifier.ts"; + +test("generates valid deterministic reports from fixture observations", () => { + const indexerState = indexFlowPulseLogs(loadIndexerFixtureLogs()); + const reports = verifyObservations(indexerState.observations, loadVerifierArtifactFixture()); + assert.equal(reports.length, 1); + assert.equal(reports[0].reportCore.status, "valid"); + assert.match(reports[0].reportId, /^0x[0-9a-f]{64}$/); +}); + +test("marks missing artifacts unresolved", () => { + const [observation] = indexFlowPulseLogs(loadIndexerFixtureLogs()).observations; + const report = verifyObservation(observation, { + resolverPolicyId: "flowmemory.resolver.policy.v0.empty", + artifactsByUri: {}, + }); + assert.equal(report.reportCore.status, "unresolved"); + assert.deepEqual(report.reportCore.reasonCodes, ["artifact.unavailable"]); +}); + +test("marks commitment mismatch as invalid", () => { + const [observation] = indexFlowPulseLogs(loadIndexerFixtureLogs()).observations; + const resolver = loadVerifierArtifactFixture(); + resolver.artifactsByUri[observation.uri] = { + kind: "rootfield-registration", + schemaHash: "0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff", + metadataHash: "0xeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee", + }; + + const report = verifyObservation(observation, resolver); + assert.equal(report.reportCore.status, "invalid"); + assert.equal(report.reportCore.reasonCodes.includes("commitment.mismatch"), true); +}); + +test("marks unknown pulse types unsupported", () => { + const [observation] = indexFlowPulseLogs(loadIndexerFixtureLogs()).observations; + const report = verifyObservation({ ...observation, pulseType: "99" }, loadVerifierArtifactFixture()); + assert.equal(report.reportCore.status, "unsupported"); +}); + +test("marks artifacts over fixture policy size unresolved", () => { + const [observation] = indexFlowPulseLogs(loadIndexerFixtureLogs()).observations; + const report = verifyObservation(observation, { + ...loadVerifierArtifactFixture(), + maxArtifactBytes: 1, + }); + + assert.equal(report.reportCore.status, "unresolved"); + assert.deepEqual(report.reportCore.reasonCodes, ["artifact.too_large"]); +}); + +test("generates all verifier statuses from receipt fixtures", () => { + const indexerState = indexFlowPulseReceipts(loadIndexerFixtureReceipts(), { + finalizedBlockNumber: "123458", + }); + const reports = verifyObservations(indexerState.observations, loadVerifierArtifactFixture()); + const counts = reports.reduce>((acc, report) => { + acc[report.reportCore.status] = (acc[report.reportCore.status] ?? 0) + 1; + return acc; + }, {}); + + assert.deepEqual(counts, { + invalid: 1, + reorged: 1, + unresolved: 1, + unsupported: 1, + valid: 3, + }); +}); + +test("persists deterministic verifier report JSON", () => { + const indexerState = indexFlowPulseReceipts(loadIndexerFixtureReceipts(), { + finalizedBlockNumber: "123458", + }); + const reports = verifyObservations(indexerState.observations, loadVerifierArtifactFixture()); + const dir = mkdtempSync(join(tmpdir(), "flowmemory-verifier-")); + const path = join(dir, "reports.json"); + + try { + writeVerifierReports(path, reports); + const firstWrite = readFileSync(path, "utf8"); + writeVerifierReports(path, reports); + const secondWrite = readFileSync(path, "utf8"); + const persisted = readVerifierReports(path); + + assert.equal(firstWrite, secondWrite); + assert.equal(persisted.schema, "flowmemory.verifier.persistence.v0"); + assert.equal(persisted.reports.length, 7); + assert.equal("createdAt" in persisted.reports[0].reportCore, false); + } finally { + rmSync(dir, { recursive: true, force: true }); + } +}); + +test("ships a verifier report JSON schema fixture", () => { + const schemaPath = join(process.cwd(), "fixtures", "verification-report.schema.json"); + const schema = JSON.parse(readFileSync(schemaPath, "utf8")) as { $id: string }; + assert.equal(schema.$id, "flowmemory.verifier.report.v0"); +}); diff --git a/services/verifier/tsconfig.json b/services/verifier/tsconfig.json new file mode 100644 index 00000000..13b3dd7d --- /dev/null +++ b/services/verifier/tsconfig.json @@ -0,0 +1,4 @@ +{ + "extends": "../tsconfig.base.json", + "include": ["src/**/*.ts", "test/**/*.ts"] +}