Integrate with Fetch Metadata.

[mikewest]

This patch integrates Fetch Metadata processing into Fetch's "main
fetch" algorithm, and defines a "user activation flag" on requests that
will be populated during HTML's "process a navigate fetch" algorithm.

Closes #885.

• At least two implementers are interested (and none opposed):
  • Chrome is shipping this as of 76.
  • Mozilla has labeled the feature as "worth prototyping".
• Tests are written and can be reviewed and commented upon at:
  • https://wpt.fyi/results/fetch/metadata?label=experimental&label=master&aligned
  • @mikewest is writing a test for the specific bit around Service Worker's access to the headers in WPT: Sec-Fetch-* headers aren't accessible in service workers. web-platform-tests/wpt#27857.
• Implementation bugs are filed:
  • Chrome: Shipped.
  • Firefox: https://bugzilla.mozilla.org/show_bug.cgi?id=1695911
  • Safari: https://bugs.webkit.org/show_bug.cgi?id=204744

(See WHATWG Working Mode: Changes for more details.)

---

Preview | Diff

[mikewest]

As discussed on public-webappsec@. WDYT, @annevk?

[mikewest]

Coming back to this as a prereq to whatwg/html#5203.

WDYT?

[annevk]

As Yoav once discovered, this does not work. As https://fetch.spec.whatwg.org/#cors-unsafe-request-header-names is currently invoked, setting Sec- headers before the network layer forces a preflight. We'd have to add an exception there (or its caller).

I think there is agreement that adding an exception is fine, provided the Sec- headers have reasonable restrictions on length and attacker-controlled data. (Perhaps we ought to document that in the process of enshrining this exception.)

[mikewest]

As Yoav once discovered, this does not work. As https://fetch.spec.whatwg.org/#cors-unsafe-request-header-names is currently invoked, setting Sec- headers before the network layer forces a preflight. We'd have to add an exception there (or its caller).

Would you like me to put these headers specifically into https://fetch.spec.whatwg.org/#cors-safelisted-request-header? Or would you like us to work out a Sec-* carveout generally?

[annevk]

I think we should carve it out generically, right?

Now actually copying @yoavweiss so he can share any thoughts as to why carving out Sec- generically might be bad.

[mikewest]

Would you like me to carve it out generically in this CL, or in a separate patch? Probably the latter, right? #1000

[annevk]

This also does not address w3c/webappsec-fetch-metadata#29 as far as I can tell. We need a solution for that.

Also, w3c/webappsec-fetch-metadata#38 changed right as this PR would expose them to service workers?

Could you maybe do another triage round so I know what upstream issues might end up impacting Fetch still?

[mikewest]

I merged the last few months of changes in f780097, and moved the integration out of "main fetch" and into "HTTP-network-or-cache" fetch in 58d9d18. I believe that addresses the following concerns:

• We do not expose the headers to the Request.headers accessor in Service workers, as they're added after SW have a shot at handling the request. That should allow us to resolve Exposing Sec-Fetch headers to Service Workers via Request().Headers() w3c/webappsec-fetch-metadata#38.
• The HTTPS->HTTP->HTTPS redirect behavior described in the spec (and in WPT) should fall naturally out of this change for the same reason (we recalculate on each hop), which should allow us to resolve Update Redirect section with HTTPS->HTTP behavior w3c/webappsec-fetch-metadata#29.
• We can punt the question of a more generic Sec-* carveout for CORS preflights, as this change ensures that headers are added after the point at which we would have send a preflight (in HTTP fetch).

I've skimmed very quickly through other bugs, but nothing other than whatwg/html#5203 jumps out at me as needing to be resolved before landing this integration. I'll try to label things accordingly if I can find some time.

WDYT?

[annevk]

Thanks for getting around to updating this. The plan looks good to me, I only have nits.

[annevk]

I've also added the pull request template to OP as this is a normative change. Mainly so we can find tests and bugs later. I think this has support from Chrome/Firefox.

Maybe @youennf can weigh in for Safari?

[annevk]

Yeah this looks good, thanks.

[mikewest]

Updated dependent PRs and specs, and filled out the template above.

[annevk]

Thanks @mikewest!

[annevk]

Unfortunately, I just realized I forgot to note whatwg/html#5203 in the commit message. Oh well.