Skip to content

CONTRACT.md

Latest commit

 

History

History
209 lines (150 loc) · 10.7 KB

CONTRACT.md

File metadata and controls

209 lines (150 loc) · 10.7 KB

Prerender Integration Contract

This document defines the wire-protocol contract that every Prerender integration library MUST satisfy. It is the single source of truth for what an integration sends to the Prerender service.

If you are writing a new Prerender integration (Express, Fastify, Rails, FastAPI, Spring, Gin, …) this is the spec to conform to. Run your integration against mock-server.mjs and the scenarios.json manifest to verify.

1. When to prerender

An integration MUST request a prerendered page from the Prerender service if all of the following hold:

  • HTTP method is GET.
  • A User-Agent request header is present and non-empty.
  • The request path does not end with one of the ignored static-asset extensions.
  • At least one of:
    • The query string contains the parameter _escaped_fragment_ (key alone is sufficient — value may be empty).
    • The request carries the header X-Bufferbot (any non-empty value).
    • The User-Agent header matches one of the crawler user agents (case-insensitive substring match).

Otherwise the integration MUST pass the request through to the underlying application unchanged.

2. Crawler user agents

Match is case-insensitive substring: lowercase the incoming UA and test ua.includes(token) for each token below.

googlebot
yahoo
bingbot
baiduspider
facebookexternalhit
twitterbot
rogerbot
linkedinbot
embedly
quora link preview
showyoubot
outbrain
pinterest
slackbot
developers.google.com/+/web/snippet
w3c_validator
perplexity
oai-searchbot
chatgpt-user
gptbot
claudebot
amazonbot

This list is the canonical source. Integrations SHOULD copy it verbatim. New entries are added here first and propagated to integrations in a coordinated bump.

3. Static-asset extensions

Match is case-insensitive suffix on request.path (not the full URL — exclude query string).

.js .css .xml .less .png .jpg .jpeg .gif .pdf .doc .txt .ico .rss .zip
.mp3 .rar .exe .wmv .avi .ppt .mpg .mpeg .tif .wav .mov .psd .ai .xls
.mp4 .m4a .swf .dat .dmg .iso .flv .m4v .torrent .ttf .woff .svg

4. Outgoing request to Prerender

When the integration decides to prerender (per §1), it MUST issue exactly one HTTP request to the Prerender service shaped as follows.

4.1 Method

GET

4.2 URL composition

{serviceUrl}/{originalScheme}://{originalHost}{originalPathAndQuery}

Where:

  • serviceUrl is the configured service URL (default https://service.prerender.io/). Trailing slash is normalized — the integration MUST ensure exactly one / between serviceUrl and the target URL.
  • originalScheme is the scheme of the incoming request (http or https), unless an explicit protocol override is configured.
  • originalHost is the value of the incoming Host header.
  • originalPathAndQuery is the request path plus query string (preserving raw encoding; query string included verbatim when non-empty, otherwise omitted entirely along with the ?).

Example: an incoming request to https://example.com/blog/post-1?ref=twitter with default service URL produces:

GET https://service.prerender.io/https://example.com/blog/post-1?ref=twitter

4.3 Required headers

Header Value
User-Agent The exact value of the original request's User-Agent header. The renderer needs the bot's UA to render the page as that bot would see it — integrations MUST NOT replace it with their own identifier.
X-Prerender-Int-Type The integration's canonical identifier — see §5.
X-Prerender-Int-Version The integration's own released version, e.g. 1.0.2. Hardcoded constant per integration; bumped on each release. Lets support tickets like "all v1.0.x koa requests fail after Y" be answered from logs alone.
X-Prerender-Request-Id A fresh UUID v4 generated by the integration for this single outgoing request. Lets support correlate "customer says X happened at 14:32" with a specific backend log line. MUST be a new value per outgoing request — never reused, never derived from the incoming request.

4.4 Optional headers

Header When sent Value
X-Prerender-Token When token is configured The configured token verbatim. Omit the header entirely when unconfigured — do not send empty string.

4.5 Headers the integration MUST NOT send

  • Authorization (Prerender uses X-Prerender-Token, not standard auth).
  • Hop-by-hop headers from the original request (Connection, Transfer-Encoding, Keep-Alive, etc.).
  • Cookies from the original request.

4.6 Redirect handling

The integration MUST NOT follow redirects from the Prerender service automatically. Redirect responses (3xx) MUST be passed through to the original client with their Location header intact.

5. X-Prerender-Int-Type canonical values

The header value identifies which integration emitted the request, used for backend telemetry and support.

Current convention (preserved for backwards compatibility): PascalCase, no separators.

Integration Value
Laravel Laravel
Java (Servlet Filter) Java
ASP.NET Core AspNetCore
Django Django
Koa Koa
Hapi Hapi

New integrations: pick a PascalCase identifier that names the framework, not the language (e.g., Express, Fastify, Rails, FastAPI, Spring, Gin, Phoenix). When a framework name has its own canonical casing (e.g., FastAPI, AspNetCore), preserve it.

Status: this convention is documented because it reflects what is already deployed in 6 integrations. Future work may normalize to lowercase-kebab (asp-net-core, fast-api) — if so, the backend will accept both.

6. Response handling

This section is bidirectional: it specifies what the Prerender service returns AND what the integration must do with it.

6.1 What the service returns

Happy path (200)

Aspect Value
Status 200
Body The rendered HTML, verbatim. No wrapper.
Content-Type text/html; charset=UTF-8
X-Prerender-RequestId Echo of the integration's X-Prerender-Request-Id, if present, else a server-generated UUID. Note casing inconsistency: integration sends Request-Id, service responds with RequestId. Both forms refer to the same value.
X-Prerender-User-Id Hashed user identifier, for support correlation.

Error responses

All error responses have empty bodies unless noted. The X-Prerender-Reject-Reason header carries a machine-readable reason code:

Status When X-Prerender-Reject-Reason
401 X-Prerender-Token was sent but doesn't match a known user invalid-x-prerender-token-provided
403 X-Prerender-Token header is missing entirely no-x-prerender-token-provided
404 The embedded URL after the service prefix is malformed and can't be parsed url-invalid
429 The customer's plan is in scheduled cancellation rate-limit-scheduled-cancellation
429 The customer's trial limit has been reached trial-limit-reached
429 Generic rate limiting (request floor) (none — JSON body from express-rate-limit)
502 Downstream renderer connection error connection-error
503 The user's account is cancelled and recaching is disabled invalid-x-prerender-token-provided
504 The embedded URL matches the customer's ignored-domains list ignored-domain
504 Downstream renderer error (timeout/crash) rendering-error

Notes on legacy inconsistencies (preserved for backwards compatibility):

  • 401 vs 403 are reversed relative to the usual HTTP convention: missing-credential returns 403, invalid-credential returns 401. New integrations should treat both as "auth failed" and avoid branching on the exact code.
  • The reject-reason invalid-x-prerender-token-provided is reused for both invalid-token (401) and cancelled-user-recaching-disabled (503). Branch on the status code, not the reason.
  • Behaviour for non-GET methods (POST, PUT, etc.) is currently undefined. Integrations MUST only send GET (per §4.1); they MUST NOT rely on any specific response for non-GET requests.

6.2 What the integration MUST do

  • Forward the response status code verbatim to the original client.
  • Forward the response body verbatim.
  • Forward response headers, EXCEPT the following hop-by-hop / framework-managed headers which MUST be dropped:
    • Content-Encoding
    • Content-Length
    • Transfer-Encoding
    • Connection
  • The integration SHOULD forward X-Prerender-Reject-Reason to the original client when present — it's useful for debugging customer integrations and has no security implication.
  • The integration MUST NOT follow redirects from the Prerender service automatically (already stated in §4.6).

6.3 Integration behavior on network errors

If the request to the Prerender service fails with a network/connection error (DNS failure, connection refused, timeout, TLS error), the integration MUST fall back to the underlying application (i.e., behave as if shouldPrerender had returned false). It MUST NOT propagate the network error to the original client.

If the Prerender service returns a non-2xx, non-3xx status, the integration SHOULD forward it as-is by default. Implementations MAY expose a softHttpCodes opt-in that converts 4xx/5xx into either a passthrough or a custom error page.

7. Conformance scenarios

Two scenario manifests, one per direction:

  • scenarios.json — consumer scenarios for integration libraries. Each scenario describes an incoming request to the integration and the expected outgoing request to the Prerender service.
  • provider-scenarios.json — provider scenarios for the Prerender service itself. Each scenario describes an incoming request as it would arrive from an integration and the expected response per §6.1.

Verifying an integration (consumer)

  1. Spawn the mock: node mock-server.mjs (defaults to :9090).
  2. Configure your integration with serviceUrl: http://localhost:9090/ and token: test-token.
  3. For each scenario in scenarios.json, hit your integration with the input request, then GET http://localhost:9090/__requests and assert the recorded outgoing request matches the expected shape.
  4. Reset between scenarios with POST http://localhost:9090/__reset.

Verifying the service (provider)

  1. Boot the Prerender service in test mode (with the rendering backend and DB mocked or stubbed).
  2. For each scenario in provider-scenarios.json, replay the described request against the service and assert the response status and headers match expectedResponse.

See README.md for a full example.