BIP375: Add test vectors + validator

[macgyver13]

This PR provides bip-0375/bip375_test_vectors.json and a reference bip-0375/validator/validate_psbt.py for validating Sending Silent Payments with PSBTs.
PSBTs are validated against v2 requirements with BIP-375 rules (building on BIP-352 silent payment derivation and BIP-374 DLEQ proofs).

Changes since last force-push:

• Add vendored dependencies in bip-0375/deps:
  • secp256k1lab (ECC operations)
  • bitcoin_test (PSBT parsing adapted from Bitcoin Core test framework)
• Implement 4-stage sequential PSBT validation: structure, ECDH coverage + DLEQ correctness, input eligibility, and output script verification
• Expanded test vector coverage:
  • mixed + ineligible inputs
  • output script validation with same scan key / spend key, same scan / different spend key, different scan key
  • introduce P2TR inputs
• Add test vector version field (starting at 1.1)
• Update BIP-0375 Test Vectors section

Open Questions:

• Should BIP text be clarified for Computing the Output Scripts as follows? - added 'from eligible inputs'

Compute the PSBT_OUT_SCRIPT using the procedure in BIP352 but substituting a·Bscan with the PSBT_GLOBAL_SP_ECDH_SHARE for that scan key if available, or the sum of all PSBT_IN_SP_ECDH_SHAREs from eligible inputs for that scan key.

Feedback welcome @andrewtoth @achow101 @theStack

Details

Test Runner Output

Description: BIP-375 Test Vectors
Version: 1.1

Invalid PSBTs: 22
psbt structure: missing PSBT_OUT_SP_V0_INFO field when PSBT_OUT_SP_V0_LABEL set
psbt structure: incorrect byte length for PSBT_OUT_SP_V0_INFO field
psbt structure: incorrect byte length for PSBT_IN_SP_ECDH_SHARE field
psbt structure: incorrect byte length for PSBT_IN_SP_DLEQ field
psbt structure: PSBT_GLOBAL_TX_MODIFIABLE field is non-zero when PSBT_OUT_SCRIPT set for sp output
psbt structure: missing PSBT_OUT_SCRIPT field when sending to non-sp output
ecdh coverage: P2TR input with NUMS internal key cannot derive sp output
ecdh coverage: only one ineligible P2MS input when PSBT_OUT_SCRIPT set for sp output
ecdh coverage: missing PSBT_IN_SP_ECDH_SHARE field for input 0 when PSBT_OUT_SCRIPT set for sp output
ecdh coverage: missing PSBT_IN_SP_DLEQ field for input when PSBT_IN_SP_ECDH_SHARE set
ecdh coverage: missing PSBT_GLOBAL_SP_DLEQ field when PSBT_GLOBAL_SP_ECDH_SHARE set
ecdh coverage: invalid proof in PSBT_IN_SP_DLEQ field
ecdh coverage: invalid proof in PSBT_GLOBAL_SP_DLEQ field
ecdh coverage: missing PSBT_IN_BIP32_DERIVATION field for input when PSBT_IN_SP_DLEQ set
ecdh coverage: output 1 missing ECDH share for scan key with one input / three sp outputs (different scan keys)
ecdh coverage: input 1 missing ECDH share for output 1 with two inputs / two sp outputs (different scan keys)
ecdh coverage: input 1 missing ECDH share for scan key with two inputs / one sp output
input eligibility: segwit version greater than 1 in transaction inputs with sp output
input eligibility: non-SIGHASH_ALL signature on input with sp output
output scripts: PSBT_OUT_SCRIPT does not match derived sp output
output scripts: two sp outputs (same scan / different spend keys) not sorted lexicographically by spend key
output scripts: k values assigned to wrong output indices with three sp outputs (same scan / spend keys)

Valid PSBTs: 19
can finalize: one P2PKH input single-signer
can finalize: two inputs single-signer using global ECDH share
can finalize: two inputs single-signer using per-input ECDH shares
can finalize: two inputs / two sp outputs with mixed global and per-input ECDH shares
can finalize: one input / one sp output with both global and per-input ECDH shares
can finalize: three sp outputs (different scan keys) with multiple global ECDH shares
can finalize: one P2WPKH input / two mixed outputs - labeled sp output and BIP 32 change
can finalize: one input / two sp outputs - output 0 has no label / output 1 uses label=0 convention for sp change
can finalize: two sp outputs - output 0 uses label=3 / output 1 uses label=1
can finalize: two inputs using per-input ECDH shares - only eligible inputs contribute shares (P2SH excluded)
can finalize: two inputs using global ECDH share - only eligible inputs contribute shares (P2SH excluded)
can finalize: two mixed input types - only eligible inputs contribute ECDH shares (NUMS internal key excluded)
can finalize: three sp outputs (same scan key) - each output has distinct k value
can finalize: three sp outputs (same scan key) / two regular outputs - k values assigned independently of output index
in progress: two P2TR inputs, neither is signed
in progress: one P2TR input / one sp output with no ECDH shares when PSBT_OUT_SCRIPT field is not set
in progress: two inputs / one sp output, input 1 missing ECDH share when PSBT_OUT_SCRIPT field is not set
in progress: one input / two sp outputs, input 0 missing ECDH share for output 0 when PSBT_OUT_SCRIPT field is not set
in progress: large PSBT with nine mixed inputs / six outputs - some inputs signed

Summary: 41 passed, 0 failed

Test vector generator is available in an external repo

[nymius]

cACK 4510632

• I haven't been able to decode the PSBT's using bitcoin-cli decodepsbt in a per test basis. Did you have any issues?

• I would consider copying the approach taken in #2084, and use secp256k1lab here.

• As each PSBT change is reflected in BIP 174, maybe is worth moving the parsing, fields and PSBT validation logic to BIP 174 directory, and keep the role logic of each protocol on its own directory, like here.

[macgyver13]

cACK 4510632

• I haven't been able to decode the PSBT's using bitcoin-cli decodepsbt in a per test basis. Did you have any issues?

Details in comment above - PSBT v2 not supported

• I would consider copying the approach taken in BIP-374: vendor secp256k1lab and use it for reference implementation #2084, and use secp256k1lab here.

I agree that approach is better than the current bip-0374 import. The current import process would have to adapt to that change regardless. I'll put that on the list for the next revision.

• As each PSBT change is reflected in BIP 174, maybe is worth moving the parsing, fields and PSBT validation logic to BIP 174 directory, and keep the role logic of each protocol on its own directory, like here.

Interesting idea. Are you suggesting a storage change for this PR or expand the scope of the PSBT parsing, fields and validation to v0 and other v2 fields and move to other BIPS directories?

[nymius]

Interesting idea. Are you suggesting a storage change for this PR or expand the scope of the PSBT parsing, fields and validation to v0 and other v2 fields and move to other BIPS directories?

I'm suggesting the second one, but as it seems to involve more components, I wouldn't pursue it in this PR.

[macgyver13]

This PR depends on #2084.

Converting to draft while dependent work is evaluated. In parallel reworking a few test cases for better coverage of spec. ie. (ECDH share coverage and unique identification)

[murchandamus]

@macgyver13: It looks like your dependency moved forward. Could you take another look whether this is ready for an update?

[macgyver13]

Ready for review again — appreciate the patience. Main additions since marking as draft are in the PR description above.

Will need feedback on the two open questions before this PR can be finalized.

[murchandamus]

I see no review by the BIP owners so far. Are you talking to them out of band, or should the be requested here?

[andrewtoth]

Sorry, haven't had a chance to look at this yet. Will review shortly.

[macgyver13]

I see no review by the BIP owners so far. Are you talking to them out of band, or should the be requested here?

theStack mentioned he would take a look soon when we spoke last week.

[andrewtoth]

This is awesome, thank you!

BIP-375 allows PSBT_OUT_SCRIPT field to be optional if PSBT_OUT_SP_V0_INFO field is present for an output, should PSBT_OUT_SCRIPT field with empty value_data be considered valid? - I prefer invalid

This seems out of scope for BIP375. I believe if present, the PSBT_OUT_SCRIPT should have a valid script. So, empty is invalid.

Should BIP text be clarified for Computing the Output Scripts as follows? - added 'from eligible inputs'

Or perhaps the BIP can be modified to prohibit adding a share for ineligible inputs. I think that would fix this ambiguity and also catch issues sooner. Wdyt?

I don't see a test case for a P2PKH input, can we add one?
Also, a test case of a global ECDH share (so signer has all keys) for a mix of eligible and ineligible inputs could be added.

[macgyver13]

This is awesome, thank you!

Appreciate the review, happy to help!

BIP-375 allows PSBT_OUT_SCRIPT field to be optional if PSBT_OUT_SP_V0_INFO field is present for an output, should PSBT_OUT_SCRIPT field with empty value_data be considered valid? - I prefer invalid

This seems out of scope for BIP375. I believe if present, the PSBT_OUT_SCRIPT should have a valid script. So, empty is invalid.

I agree that this is out of scope for BIP-375.
Removed: psbt structure: empty PSBT_OUT_SCRIPT field when sending to non-sp output

Should BIP text be clarified for Computing the Output Scripts as follows? - added 'from eligible inputs'

Or perhaps the BIP can be modified to prohibit adding a share for ineligible inputs. I think that would fix this ambiguity and also catch issues sooner. Wdyt?

I prefer the prohibition of shares for ineligible inputs - it's the cleanest way to handle validation based on my experience. I have updated the Signer section of the BIP to reflect this; let me know what you think.
Modified test name only:
output scripts: P2TR input with NUMS internal key cannot derive sp output
ecdh coverage: P2TR input with NUMS internal key cannot derive sp output

I don't see a test case for a P2PKH input, can we add one? Also, a test case of a global ECDH share (so signer has all keys) for a mix of eligible and ineligible inputs could be added.

Modified one of the many P2WPKH test to use P2PKH instead to give more coverage.
can finalize: one input single-signer
can finalize: one P2PKH input single-signer

Good idea on global ECDH share with a mix of eligible and ineligible inputs -- this new test caught an invalid assumption in my Rust validator implementation.
Added: can finalize: two inputs using global ECDH share - only eligible inputs contribute shares (P2SH excluded)
Modified test name only:
can finalize: two mixed input types - only eligible inputs contribute ECDH shares (P2SH excluded)
can finalize: two inputs using per-input ECDH shares - only eligible inputs contribute shares (P2SH excluded)

[andrewtoth]

I prefer the prohibition of shares for ineligible inputs - it's the cleanest way to handle validation based on my experience. I have updated the Signer section of the BIP to reflect this; let me know what you think.

Thinking about this more, this is backwards incompatible change. So, modifying the BIP in this way needs a mailing list post and some time to gather feedback from existing implementers.

[macgyver13]

After reflecting on the previous version of the Signer text change I realized I had inserted a 'must' bullet into a 'should' section. I think the addition of eligible as a 'should' resolves the ambiguity and I don't see this as a backward incompatible change.

I chose to isolate this change in a single commit for easy review and linkage if necessary.

@andrewtoth - If you think a Mailing List Update is warranted let me know.