add watchtower querier trait by doitian · Pull Request #1153 · nervosnetwork/fiber

doitian · 2026-02-28T06:44:18Z

feat: implement watchtower querier trait and integrate with network actor

Allow network/channel actor to query tlc status from watchtower no matter it is in the same process or runs as a standalone service.

Refs #1077

doitian · 2026-03-11T06:59:47Z

Need to add e2e test

Copilot

Pull request overview

Implements a watchtower query abstraction so the network/channel flows can check TLC settlement/preimage status via either the built-in watchtower store or a standalone watchtower RPC, and adds coverage via new unit/e2e tests.

Changes:

Add WatchtowerQuerier + query actor to asynchronously fetch TLC status (preimage + settled) and integrate results into NetworkActor.
Expose new watchtower RPC method query_tlc_status (types, server, auth rule, docs).
Add a new Bruno e2e scenario for “force-close preimage settled by recipient” and add unit tests for the querier integration.

Reviewed changes

Copilot reviewed 44 out of 44 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/01-node1-connect-node2.bru	New e2e step: connect peers (N1↔N2).
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/02-node2-connect-node3.bru	New e2e step: connect peers (N2↔N3).
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/03-node1-node2-open-channel.bru	New e2e step: open N1→N2 channel.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/04-node2-get-auto-accepted-channel.bru	New e2e step: fetch N1-N2 channel id.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/05-ckb-generate-blocks.bru	New e2e step: mine epochs for channel confirmation.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/06-node2-node3-open-channel.bru	New e2e step: open N2→N3 channel.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/07-node3-get-auto-accepted-channel.bru	New e2e step: fetch N2-N3 channel id.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/08-ckb-generate-blocks.bru	New e2e step: mine epochs for channel confirmation.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/09-get-node1-funding-script.bru	New e2e step: capture node1 default funding lock script.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/10-get-node3-funding-script.bru	New e2e step: capture node3 default funding lock script.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/11-get-node1-balance.bru	New e2e step: record node1 on-chain balance.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/12-get-node3-balance.bru	New e2e step: record node3 on-chain balance.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/13-node3-gen-invoice.bru	New e2e step: generate hold invoice + payment hash/preimage vars.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/14-node1-send-payment-with-invoice.bru	New e2e step: send payment using invoice.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/15-node2-force-close-channel.bru	New e2e step: force-close N2-N3 channel.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/16-node2-disconnect-node3.bru	New e2e step: disconnect N2 from N3.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/17-ckb-generate-blocks-for-force-close-tx.bru	New e2e step: mine epoch for force-close tx.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/18-node3-remove-tlc.bru	New e2e step: node3 reveals preimage on-chain (remove_tlc).
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/19-ckb-generate-blocks-for-settlement-tx-preimage.bru	New e2e step: mine epochs for settlement tx (preimage path).
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/20-ckb-generate-blocks-for-final-settlement-tx.bru	New e2e step: mine epochs for final settlement.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/21-ckb-generate-blocks-for-final-settlement-tx-commit.bru	New e2e step: mine epoch for final settlement commit.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/22-check-channel1-balance.bru	New e2e assertion: verify intermediary balance increased.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/23-check-payment-status.bru	New e2e assertion: payment status is Success.
tests/bruno/e2e/watchtower/force-close-preimage-settled-by-recipient/24-disconnect.bru	New e2e step: disconnect peers after scenario.
docs/biscuit-auth.md	Document Biscuit auth rule for `query_tlc_status`.
crates/fiber-wasm/src/lib.rs	Wire optional watchtower querier into WASM network startup.
crates/fiber-types/src/schema.rs	Gate WATCHTOWER_TLC_SETTLED_PREFIX behind `watchtower` feature.
crates/fiber-lib/src/watchtower/store.rs	Extend watchtower store with TLC status query helpers.
crates/fiber-lib/src/tests/test_utils.rs	Allow injecting a watchtower querier into test NetworkNode config.
crates/fiber-lib/src/store/store_impl/mod.rs	Move TLC-settled lookup into `WatchtowerStore`; implement `WatchtowerQuerier` for Store.
crates/fiber-lib/src/rpc/watchtower.rs	Add `query_tlc_status` RPC + client-side querier wrapper.
crates/fiber-lib/src/rpc/biscuit.rs	Add Biscuit auth rule for `query_tlc_status`.
crates/fiber-lib/src/rpc/README.md	Add method docs for `query_tlc_status`.
crates/fiber-lib/src/fiber/watchtower_query_actor.rs	New actor to perform watchtower queries off the network actor hot path.
crates/fiber-lib/src/fiber/watchtower_query.rs	New `WatchtowerQuerier` trait + status struct.
crates/fiber-lib/src/fiber/tests/watchtower_query.rs	New unit tests covering payment success via watchtower-provided preimage.
crates/fiber-lib/src/fiber/tests/settle_tlc_set_command_tests.rs	Update mock store for removed `is_tlc_settled` channel-store API.
crates/fiber-lib/src/fiber/tests/mod.rs	Register new watchtower_query tests (non-wasm).
crates/fiber-lib/src/fiber/network.rs	Integrate watchtower querying into CheckChannels + handle results via new commands.
crates/fiber-lib/src/fiber/mod.rs	Export watchtower query types; include new actor module.
crates/fiber-lib/src/fiber/channel.rs	Remove `is_tlc_settled` from ChannelActorStateStore trait.
crates/fiber-json-types/src/watchtower.rs	Add JSON-RPC types for `query_tlc_status`.
crates/fiber-bin/src/main.rs	Construct watchtower querier (prefer store; fallback to standalone watchtower RPC) and pass to `start_network`.
.github/workflows/e2e.yml	Add the new watchtower e2e scenario to CI matrix.

Copilot · 2026-03-23T01:53:15Z

+                self.store.insert_preimage(payment_hash, preimage);
+                debug!(
+                    "Found watchtower preimage for channel {:?} tlc {:?}",
+                    channel_id, tlc_id
+                );
+                if self
+                    .store
+                    .get_invoice_status(&payment_hash)
+                    .is_some_and(|s| {
+                        !matches!(s, CkbInvoiceStatus::Open | CkbInvoiceStatus::Received)
+                    })
+                {
+                    return Ok(());
+                }
+                // Only send RemoveTlc if the channel still has this received TLC (it may have
+                // been removed already, e.g. by a concurrent payment winning the race).
+                let tlc_still_exists =
+                    self.store
+                        .get_channel_actor_state(&channel_id)


When a watchtower-sourced preimage arrives, it’s inserted into the node’s preimage store before any validation. If the watchtower is misconfigured/malicious and returns a wrong preimage, the node will stop querying (because get_preimage becomes Some) and repeatedly fail TLC settlement with FinalIncorrectPreimage. Recommend validating preimage against the TLC’s hash_algorithm/payment_hash (and only persisting it if it matches), and ideally only persisting if the received TLC still exists.

Copilot · 2026-03-23T01:53:15Z

+                                        let _ = wt_actor.send_message(
+                                            WatchtowerQueryActorMessage::QueryPreimage {
+                                                channel_id,
+                                                tlc_id: tlc_id.into(),
+                                                payment_hash,
+                                            },
+                                        );


This send ignores any error from watchtower_query_actor (e.g., mailbox closed/full), which can silently prevent TLC settlement retries via watchtower. Consider handling the send_message result (at least log at debug/warn) so failures are observable.

Suggested change

let _ = wt_actor.send_message(

WatchtowerQueryActorMessage::QueryPreimage {

channel_id,

tlc_id: tlc_id.into(),

payment_hash,

},

);

if let Err(err) = wt_actor.send_message(

WatchtowerQueryActorMessage::QueryPreimage {

channel_id,

tlc_id: tlc_id.into(),

payment_hash,

},

) {

warn!(

"Failed to send QueryPreimage to watchtower for channel {:?}, tlc {:?}, payment_hash {:?}: {}",

channel_id,

tlc_id,

payment_hash,

err

);

}

Copilot · 2026-03-23T01:53:16Z

+#[serde_as]
+#[derive(Serialize, Deserialize, Debug, Clone, JsonSchema)]
+pub struct QueryTlcStatusResult {
+    /// Found preimage when the TLC has been settled on chain with the preimage path.


preimage is documented as only being present when the TLC has been settled on-chain with the preimage path, but the implementation returns whatever preimage the watchtower knows (e.g., inserted via create_preimage), independent of is_settled. Please adjust the wording to match actual behavior (e.g., “preimage if known”).

Suggested change

/// Found preimage when the TLC has been settled on chain with the preimage path.

/// Preimage if known (independent of `is_settled`).

Copilot · 2026-03-23T01:53:16Z

+
+##### Returns
+
+* `preimage` - <em>Option<[Hash256](#type-hash256)></em>, Found preimage when the TLC has been settled on chain with the preimage path.


The return docs for query_tlc_status say the preimage is only found when the TLC has been settled on-chain with the preimage path, but the RPC returns any stored/known preimage regardless of is_settled (e.g., preimages created via create_preimage). Update this description to avoid implying preimage => on-chain settlement.

Suggested change

* `preimage` - <em>Option<[Hash256](#type-hash256)></em>, Found preimage when the TLC has been settled on chain with the preimage path.

* `preimage` - <em>Option<[Hash256](#type-hash256)></em>, Known preimage for this payment hash, if any. May be present regardless of whether the TLC has been settled on chain.

Copilot · 2026-03-23T01:53:16Z

+            // Construct watchtower querier for WASM (standalone watchtower only)
+            let watchtower_querier: Option<std::sync::Arc<dyn fnn::fiber::WatchtowerQuerier>> =
+                if let Some(url) = fiber_config.standalone_watchtower_rpc_url.clone() {
+                    let querier_client =
+                        WasmClientBuilder::default()
+                            .build(url)
+                            .await
+                            .map_err(|err| {
+                                ExitMessage(format!(
+                                    "failed to create watchtower rpc client: {}",
+                                    err
+                                ))
+                            })?;
+                    Some(std::sync::Arc::new(
+                        fnn::rpc::watchtower::WatchtowerRpcQuerier::new(querier_client),
+                    ))
+                } else {
+                    None
+                };


In WASM builds, the standalone watchtower RPC client is created without using fiber_config.standalone_watchtower_token, so query_tlc_status (and later event forwarding) will fail against an authenticated watchtower endpoint. Please either add token/header support for the wasm client builder (if supported by jsonrpsee wasm-client) or document/guard that auth tokens aren’t supported in wasm mode.

Copilot · 2026-03-23T01:53:16Z

+            // Construct watchtower querier for WASM (standalone watchtower only)
+            let watchtower_querier: Option<std::sync::Arc<dyn fnn::fiber::WatchtowerQuerier>> =
+                if let Some(url) = fiber_config.standalone_watchtower_rpc_url.clone() {
+                    let querier_client =
+                        WasmClientBuilder::default()
+                            .build(url)
+                            .await
+                            .map_err(|err| {
+                                ExitMessage(format!(
+                                    "failed to create watchtower rpc client: {}",
+                                    err
+                                ))
+                            })?;
+                    Some(std::sync::Arc::new(
+                        fnn::rpc::watchtower::WatchtowerRpcQuerier::new(querier_client),
+                    ))


This block builds a dedicated wasm watchtower client for the querier, but the file later constructs another client for forwarding events to the watchtower. If they’re targeting the same URL, consider reusing a single client instance (or a clonable handle) to avoid duplicate connections/config drift (e.g., headers/timeouts).

Copilot · 2026-03-23T01:53:16Z

+    b.with_rule(
+        "query_tlc_status",
+        AuthRule::new(r#"allow if read("watchtower");"#).with_require_rpc_context(true),
+    );


The new query_tlc_status Biscuit rule only checks read("watchtower"), but the Biscuit auth doc example adds a right({channel_id}, "watchtower") constraint. Either update the documentation to match the actual enforcement, or extend the auth/middleware to inject channel_id facts and enforce channel-scoped rights for this method (and other watchtower methods, if intended).

Copilot · 2026-03-23T01:53:17Z

+rule(
+    "query_tlc_status",
+    r#"
+    allow if read("watchtower");
+    allow if right({channel_id}, "watchtower");
+    "#,
+);


The example rule for query_tlc_status includes right({channel_id}, "watchtower"), but the current Biscuit middleware/rule set doesn’t appear to inject channel_id into the authorizer context for RPC calls (it only injects node_id). As written, this example is not enforceable and may mislead users into thinking channel-scoped authorization is active for this method.

…ctor Allow network/channel actor to query tlc status from watchtower no matter it is in the same process or runs as a standalone service.

- Add WatchtowerQueryActor to run query_tlc_status without blocking Network Actor - Queue QueryPreimage and QuerySettled jobs from CheckChannels; handle results via WatchtowerPreimageResult and WatchtowerSettledResult commands - Pass tlc_id through preimage flow so result handler can send RemoveTlc when preimage is found; only send if channel still has that received TLC Made-with: Cursor

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: doitian <35768+doitian@users.noreply.github.com>

Verify that in a 3-node multi-hop payment (N1->N2->N3) using hold invoice, when N2 force-closes the N2-N3 channel, N3 reveals the preimage on-chain and N2's watchtower picks it up to settle the TLC on the still-open N1-N2 channel.

The openrpc spec generator requires all RPC parameter and result types to implement schemars::JsonSchema. QueryTlcStatusParams and QueryTlcStatusResult were missing it, breaking the CI job.

doitian · 2026-03-24T08:25:49Z

Troubleshooting #1222

doitian · 2026-03-25T02:44:38Z

After discussion with Quake, we prioritize fixing the issue when watchtower is running together with fiber node. I'll close this PR and create a new one based on that assumption.

doitian force-pushed the add-watchtower-querier-trait branch from fc28803 to 01cae25 Compare February 28, 2026 08:58

jjyr approved these changes Mar 2, 2026

View reviewed changes

jjyr reviewed Mar 2, 2026

View reviewed changes

Comment thread crates/fiber-lib/src/fiber/network.rs Outdated

doitian marked this pull request as draft March 2, 2026 04:41

doitian marked this pull request as ready for review March 2, 2026 08:18

doitian force-pushed the add-watchtower-querier-trait branch from 0f4d9b6 to 5af3805 Compare March 2, 2026 09:16

doitian requested a review from jjyr March 3, 2026 03:33

jjyr approved these changes Mar 4, 2026

View reviewed changes

doitian force-pushed the add-watchtower-querier-trait branch 3 times, most recently from 71cad1a to 7c7d65f Compare March 10, 2026 05:18

doitian marked this pull request as draft March 11, 2026 06:59

doitian marked this pull request as ready for review March 12, 2026 05:55

doitian force-pushed the add-watchtower-querier-trait branch from 0c7c1fc to 2372735 Compare March 12, 2026 07:21

doitian mentioned this pull request Mar 19, 2026

[CCH] CchOrder status is not updated when outgoing TLC is settled on-chain via fiber (e.g. force close / watchtower) #1222

Open

quake requested a review from Copilot March 23, 2026 01:46

Copilot started reviewing on behalf of quake March 23, 2026 01:47 View session

Copilot AI reviewed Mar 23, 2026

View reviewed changes

ian and others added 8 commits March 24, 2026 11:09

feat: implement watchtower querier trait and integrate with network a…

7da700d

…ctor Allow network/channel actor to query tlc status from watchtower no matter it is in the same process or runs as a standalone service.

feat: add watchtower querier trait and mock implementation for testing

16b91f6

Fix clippy redundant-field-names errors

6081b9f

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: doitian <35768+doitian@users.noreply.github.com>

refactor type annotations

7977c04

docs: generate the RPC doc

6b21916

fix: add missing JsonSchema derive for watchtower query types

cb5a69c

The openrpc spec generator requires all RPC parameter and result types to implement schemars::JsonSchema. QueryTlcStatusParams and QueryTlcStatusResult were missing it, breaking the CI job.

doitian force-pushed the add-watchtower-querier-trait branch from 2372735 to cb5a69c Compare March 24, 2026 03:09

doitian marked this pull request as draft March 24, 2026 08:25

doitian closed this Mar 25, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add watchtower querier trait#1153

add watchtower querier trait#1153
doitian wants to merge 8 commits intonervosnetwork:developfrom
doitian:add-watchtower-querier-trait

doitian commented Feb 28, 2026

Uh oh!

Uh oh!

doitian commented Mar 11, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Mar 23, 2026

Uh oh!

Copilot AI Mar 23, 2026

Uh oh!

Copilot AI Mar 23, 2026

Uh oh!

Copilot AI Mar 23, 2026

Uh oh!

Copilot AI Mar 23, 2026

Uh oh!

Copilot AI Mar 23, 2026

Uh oh!

Copilot AI Mar 23, 2026

Uh oh!

Copilot AI Mar 23, 2026

Uh oh!

doitian commented Mar 24, 2026

Uh oh!

doitian commented Mar 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	/// Found preimage when the TLC has been settled on chain with the preimage path.
	/// Preimage if known (independent of `is_settled`).


		##### Returns

		* `preimage` - <em>Option<[Hash256](#type-hash256)></em>, Found preimage when the TLC has been settled on chain with the preimage path.

Conversation

doitian commented Feb 28, 2026

Uh oh!

Uh oh!

doitian commented Mar 11, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

doitian commented Mar 24, 2026

Uh oh!

doitian commented Mar 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants