Skip to content
Jobport

Member of Technical Staff (Web Access Engineer, Search Crawler)

Perplexity AI · London, United Kingdom 🇬🇧

HybridFull-timeLead / ManagerSoftware & IT

Skills mentioned

Perplexity is hiring a Web Access Engineer for the Crawler team to build reliable fetching and rendering systems for AI-powered search. You'll work on freely available public web content, within site permissions and approved access policies, and develop the infrastructure that supplies content to downstream indexing and retrieval systems.

Your focus will be diagnosing network errors, page-loading issues, and incomplete rendering. You'll trace problems from network protocols through client-side execution, distinguish technical failures from intentional access restrictions, and turn findings into production fixes that improve content quality and efficiency at scale.

Responsibilities

  • Own fetching and rendering improvements end to end, from investigation through implementation and production rollout, partnering with crawler engineers and SREs.

  • Investigate fetch and rendering failures across DNS, TLS, HTTP, caching, redirects, and client-side rendering, and classify root causes.

  • Improve page loading, completion detection, and content capture to preserve relevant content and handle dynamic pages reliably.

  • Ensure the crawler respects site-imposed access restrictions and rate limits, identifies itself clearly, and retries only transient technical errors.

  • Build tools to reproduce failures, compare crawler output against expected page content, and turn findings into regression tests.

  • Define metrics for fetching reliability, content completeness, and error classification, and use them to prioritize improvements.

Qualifications

  • 5+ years of software engineering experience in web infrastructure, browsers, CDNs, or crawling, with strong programming skills in at least one of Go, Python, Rust, or C++.

  • Deep understanding of HTTP/1.1, HTTP/2, and HTTP/3, including status semantics, caching, conditional requests, content encoding, and redirects; strong knowledge of TLS, DNS, cookies, and sessions.

  • Understanding of CDN caching and content delivery, HTTP error handling, retry backoff, and server-defined rate limits.

  • Knowledge of page lifecycle, network idle, service workers, client-side rendering, and shadow DOM; experience debugging JavaScript and fluency with Chrome DevTools, CDP, HAR files, curl, and packet captures.

  • Comfort working independently when the path forward is not fully defined.

Nice to have

  • Experience developing CDN, caching, or web performance systems.

  • Experience operating crawlers, headless browsers, or distributed content-acquisition systems at scale.

  • Contributions to a browser engine or JavaScript runtime.

Posted 6d ago (13 Nov 2026) · Source: Perplexity AI via Ashby

Similar jobs in United Kingdom

ServiceNow · Staines, , United Kingdom 🇬🇧

On-siteFull-time

Company Description It all started when engineer Fred Luddy wrote code that automated a tedious task for his coworker, Phyllis. She cried tears of joy. That moment inspired Fred to build a company that could do that for everyone—freeing people from busywork so they could focus on

5h ago · ServiceNow via SmartRecruiters

n8n · London Office, United Kingdom 🇬🇧

RemoteFull-timeVisa sponsorship

The AI orchestration of your wildest imagination. n8n is the open workflow orchestration platform built for the new era of AI. We give technical teams the freedom of code with the speed of no-code, so they can automate faster, smarter, and without limits. Backed by a fiercely inv

5h ago · n8n via Ashby

Genomics England · London, England, United Kingdom 🇬🇧

HybridFull-time

Company Description Genomics England is a global leader in enabling genomic medicine and research, focused on creating a world where everyone benefits from genomic healthcare. Building on the 100,000 Genomes Project, we support the NHS’s world-first national whole genome sequenci

6h ago · Genomics England via SmartRecruiters

Exegy · Belfast, United Kingdom 🇬🇧

HybridFull-time

About Exegy Exegy is a global leader in intelligent market data, advanced trading systems, and future-proof technology. Exegy serves as a trusted partner to the complete ecosystem of the buy-side, sell-side, exchanges, and financial services technology firms around the globe. Hea

7h ago · Exegy via Ashby