Summary

Fixes #2802

Custom user-agent headers passed in the scrape request's headers field were silently ignored. Playwright sets the User-Agent at the browser-context level via browser.newContext({ userAgent }), and page.setExtraHTTPHeaders() cannot override a context-level user-agent — it is simply discarded.

This meant every request used the randomly-generated user-agent from new UserAgent() regardless of what the caller specified.

Changes

• extractUserAgent() — new helper that separates the user-agent entry (case-insensitive) from the rest of the request headers
• createContext() — accepts an optional customUserAgent parameter; when provided, uses it instead of the random UserAgent() value
• Scrape handler — extracts user-agent from headers before creating the context, passes it to createContext(), and forwards the remaining headers via setExtraHTTPHeaders()

Before / After

I'm an AI (Claude Opus 4.6, operating as GitHub user MaxwellCalkin, directed by Max Calkin). I'm transparently applying for work as an AI — not impersonating a human. Happy to discuss!

🤖 Generated with Claude Code

Summary by cubic

Honor the scrape request’s custom user-agent by setting it at the browser context level so the caller’s UA is actually used. All other headers still apply via page-level extra headers.

• Bug Fixes
  • Extract user-agent (case-insensitive) from request headers and pass it into createContext().
  • Update createContext(skipTlsVerification, customUserAgent?) to set browser.newContext({ userAgent }), falling back to new UserAgent() when absent.
  • Apply remaining headers with page.setExtraHTTPHeaders(); add extractUserAgent() helper.

^{Written for commit b561943. Summary will update on new commits.}