Add a speculative flag to Request as well as related processing

[yoavweiss]

As discussed on w3c/resource-hints#74 and on whatwg/html#4115, this PR adds a speculative flag to Request, which makes sure the request:

• Survives navigations
• Is low priority
• Can be ignored by the UA (haven't added that part yet)
• Adds the related request headers

I intend (as part of whatwg/html#4115) to then make sure prefetch requests are sent with this flag set.

---

Preview | Diff

[yoavweiss]

• Can be ignored by the UA (haven't added that part yet)

@annevk - any advice on how such language can be formulated?

[annevk]

I suppose you could return early somehow for a speculative fetch. What does ignoring mean though? Not invoking fetch at all?

[annevk]

This would only result in setting the header? You'd still get the response and such?

[yoavweiss]

Yeah, that's the current intention. Even though prefetch is destined for the next navigation, the current renderer can still get the response (e.g. onload fires for it).

[annevk]

Well, but this adds it to fetch() as a new feature. Was that intended? Or should this only be accessible via <link>?

[yoavweiss]

I thought it'd be needed to make those headers survive SWs (although that may vary based on where #880 ends up)

[annevk]

But this would also allow creation. Surviving SW should happen based on copying private state I think.

[domfarolino]

^That sounds right to me. This is similar to the idea that priority is a concept that gets preserved here but is never exposed to the Fetch API.

[yoavweiss]

Thanks for reviewing! :)

[yoavweiss]

Yeah, that's the current intention. Even though prefetch is destined for the next navigation, the current renderer can still get the response (e.g. onload fires for it).

[yoavweiss]

@annevk I wasn't sure what to write here, so started by writing something. It feels a bit fuzzy to be part of the processing model, so advice on making it better would be appreciated

[annevk]

How concrete is it in the implementation?

[yoavweiss]

Currently I'm not aware of implementations that do that at the moment. When discussing whatwg/html#4115 @youennf wanted to give UAs the concrete possibility to do so, while staying conformant.

[yoavweiss]

I suppose you could return early somehow for a speculative fetch. What does ignoring mean though? Not invoking fetch at all?

I added a step to the processing model, aborting early at the UA's discretion. I think it'd be better to notify the developer that prefetch is being ignored.

[youennf]

Overall, I would tend to go with the minimum proposal to have a functional targeting navigation loads.
Some additional questions/observations:

1. Is prefetch sometimes/mainly/solely targeting (top) navigation loads.
   It seems to be the case since we want prefetch (at least navigation cross origin prefetch) to escape cache partitioning for instance.
   Preload, on the other hand, is about subresource loads, and can be used for all other cases.
   I like the difference as this allows to express that:

• a web page might know that it will probably navigate to another page (prefetch)
• a web page might know its own resources (preloads)
• a web page might not try to (in fact fail to) determine the resources of the potential next page and try to over-optimize the next page, especially third-party ones.

If we narrow down the scope of prefetch to navigation loads, we might be able to simplify things without loosing too much functionality.

2. Does it make sense for prefetch loads to go through SW of the prefetching page?
   The response might depend on 1. Let's assume we target navigation loads.
   For third-party navigations, SW interception benefit is unclear, downside is increased processing/latency.
   For same-origin navigations, there might be some theoretical benefits.
   That said, the service worker intercepting the prefetch might not be the one intercepting the navigation load. Preloads might be good enough there.

We also do not want to have the prefetch load go to the SW of the future navigation page.

3. Should a SW that intercepts the navigation load know that a prefetch is available?
   We might expect the navigation load to go through SW before using the prefetch cache.
   It could be handy for the SW to say: yes, go use the prefetch cache.
   Might be dealt separately.

4. Do we need the boolean in the Request interface?
   If we allow prefetch loads SW interception, probably. Otherwise, question 5.

5. Do we want JS to be able to create Request with this flag set?
   If we narrow down the scope to navigation loads, I am not sure allowing fetch() with a speculative load is very useful. One use case might be to emulate form submissions.
   Not sure the complexity of handling long living requests with body is worth this usecase.

6. Should responses be made available to the web page doing the prefetch?
   onload/onerror might be fine to have for .
   This does not require to provide the response to the webpage.
   In general, we should be worried of communicating third party responses to the web page if mode is not cors. But cors might not be always suitable for prefetch. no-cors might not be suitable either, for instance due to CORP checks. Maybe prefetch would be better served with its own mode.

[yoavweiss]

@youennf thanks for your comments! :) (and apologies for my slowness, travel is my typical excuse)

Regarding 1-3, they are defined in whatwg/html#4115
Do you think we also need to add similar restrictions in Fetch?

Regarding 4 and 5, we can probably avoid those for the time being. The tradeoff here would be that prefetches would be somewhat magical, but maybe we can tackle that later?

Regarding 6, agree that there's no need to communicate the actual response bytes to the prefetching page. Not sure how to spell that out as a spec concept though.

[youennf]

To improve efficiency, a prefetch could also be allowed to consult the HTTP cache and/or prefetch cache.
That said, the onload/onerror event handlers may allow the web page triggering the prefetch to infer some user specific information. This might need some analysis.

[jkarlin]

Is prefetch sometimes/mainly/solely targeting (top) navigation loads.
It seems to be the case since we want prefetch (at least navigation cross origin prefetch) to escape cache partitioning for instance.

There are many cases where prefetch is used to load subresources. For instance, there are sites that will create a <link rel=prefetch> to critical-path subresources when you click on a link in order to save a RTT. I'd really like us to solve prefetch for both x-origin documents and subresources.

[youennf]

Subresources could potentially be optimized with link preloads in the prefetched resource or web packaging.

[jkarlin]

Link preloads in the document are still a full RTT slower than prefetching the subresource at navigation start.

[youennf]

Sure. In general though, it seems rather brittle for a page to try optimizing the loading of a cross-origin page that might change any minute. Also, if the subresource is already in the cache (say a JS library that is needed for all pages of the prefetched origin), that might actually slow things down to prefetch it if the subresource is already in the cache and prefetch is not allowed to look at the cache.
We can probably start with something simple that solve some of the main use cases and improve upon it.

[jkarlin]

Right. We'd need the prefetch to specify a cache key (destination origin) to use for the subresource, otherwise the prefetch could be wasted if the cache already has the resource

[domfarolino]

Probably this should be a boolean, as I think we're leaning more towards explicit booleans as opposed to "flags"s these days?

[jkarlin]

Sure. In general though, it seems rather brittle for a page to try optimizing the loading of a cross-origin page that might change any minute. Also, if the subresource is already in the cache (say a JS library that is needed for all pages of the prefetched origin), that might actually slow things down to prefetch it if the subresource is already in the cache and prefetch is not allowed to look at the cache.
We can probably start with something simple that solve some of the main use cases and improve upon it.

Sorry, I meant to reply to this earlier. This sounds good to me.