Surface HTTPS retrieval success rate

[lidel]

Looking at https://dashboard.filspark.com/ makes no distinction between http:// and https:// providers.

iiuc the current probe of HTTP retrieval is not requiring HTTPS, and both http:// and https:// providers are marked as successful if the response is valid:

spark-checker/lib/spark.js

Lines 143 to 147 in 799cc5e

	const url = getRetrievalUrl(protocol, address, cid)
	console.log(`Fetching: ${url}`)
	resetTimeout()
	const res = await this.#fetch(url, { signal })

spark-checker/lib/multiaddr.js

Lines 22 to 23 in 799cc5e

	} // Handle HTTP/HTTPs addresses using the default port
	} else if (multiAddrParts[0] === 'http' || multiAddrParts[0] === 'https') {

Problem

Direct retrieval from unencrypted http:// URL is impossible in web browser contexts due to secure context limitation.
Attempting http:// request from a website loaded via https:// URL will block the request and produce mixed-context error in browser console.

Cross-origin fetch in Web API requires https:// with a valid TLS cert (signed by CA set up).
Note, these TLS certs are not used for data integrity – they are there just to facilitate Secure Context in web browsers, and they also allow use of HTTP/2 for additional performance thanks to request multiplexing.

Suggested fix

Spark seems to have all information already – maybe just log the schema used for HTTP request, and have separate success metric only for https:// URLs?

This way no additional request needs to be made, just surface how many of successes were https:// ones.

Why HTTPS matters

• HTTPS is mandatory everywhere, and HTTP providers without it are ignored by Kubo, Rainbow, ipfs.io, IPFS Desktop etc.
  • The biggest gateway implementation (boxo/gateway, which ships in Rainbow and Kubo, and powers ipfs.io and dweb.link + is used by many gateway operators) also has the requirement of https://.
  • Rationale:
    1. have parity with browser-based gateways like https://inbrowser.link & https://www.npmjs.com/package/@helia/verified-fetch
    2. incentivize SPs to "do the right thing" if they want to be useful to IPFS Mainnet
    3. not leak client's IP+browsing history in cleartext on the wire in transit
• The lack of https:// impacts IPFS Foundation-driven work in Native HTTP across the IPFS Stack, to enable Filecoin direct retrieval — IPFS/2025 ipshipyard/roadmaps#9 & Reliable, decentralized, and trustless browser fetching of IPFS content — IPFS/2025 ipshipyard/roadmaps#4
  • If SP does not expose https:// it is effectively dead (unusable) for all web clients that would like to perform direct retrieval (example: https://inbrowser.link/, @helia/verified-fetch - https://blog.ipfs.tech/verified-fetch/ )
  • If ecosystem does not correctly probe for HTTPS retrieval, then we don't see how many SPs are actually usable to IPFS Mainnet clients.
  • If direct retrieval from SPs over https:// is not possible, then data will have to go over centralized proxies and relays that take care of terminating HTTPS, increasing cost and latency.

cc ipshipyard/roadmaps#9 @bajtos @mishmosh @hsanjuan

[bajtos]

@willscott What are your thoughts on this from the perspective of FIL+ allocator compliance process? If Spark starts reporting HTTP vs HTTPS scores, will FIL+ eventually enforce HTTPS retrievals in the compliance review process?

[willscott]

eventually i guess, but i think we have not expected significant end user traffic from browsers to SPs / they are not provisioned from it - so this case where it's primarily expected to be gateways or other retrieval layers fetching from SPs won't have this issue.

as such i see this as low priority for SPs / fil+ perspective, and given current state that it'll be a slow evolution to push compliance there.

[lidel]

Just flagging this is not limited to web clients – apologies if I gave that impression.
Clarified this in "Why this matters".