Research Crawling Engineer

ABOUT THE ROLE Join a data and infrastructure team building large-scale web data systems for AI research and model development. You will help deliver reliable, high-quality datasets by operating crawlers across complex and changing web environments. WHAT YOU'LL DO - Build and maintain large-scale crawlers across diverse domains, including social media, travel, and multilingual websites. - Design high-throughput, fault-tolerant collection systems capable of processing millions to billions of URLs per day. - Adapt crawling strategies to anti-bot measures, rate limits, and dynamic or JavaScript-heavy sites. - Develop pipelines to clean, deduplicate, filter, and normalize web data at terabyte to petabyte scale. - Build datasets for research and model training in collaboration with research teams. - Monitor crawl coverage, performance, and data quality, and improve systems as web environments change. - Optimize infrastructure for cost, latency, and reliability across cloud and bare-metal environments. WHAT WE'RE LOOKING FOR - Typically 4 or more years of relevant experience building web crawlers or large-scale data acquisition systems. Candidates with a PhD may qualify with 2 or more years. - Strong experience with Go, Rust, Python, Java, or C++, and distributed systems or parallel processing. - Experience operating data pipelines at terabyte scale and processing millions to billions of URLs per day. - Deep understanding of HTTP, networking, and browser behavior, including experience handling anti-bot systems, rate limits, and JavaScript-heavy websites. - Experience with headless browsers such as Playwright or Puppeteer, proxy systems, IP rotation, and request orchestration is valuable. - Background in data quality evaluation, web data processing, NLP pipelines, or machine-learning dataset curation is a plus. - Experience running data workloads on cloud or bare-metal infrastructure; a relevant technical degree is beneficial. COMPENSATION & BENEFITS Base salary: $160,000 to $250,000 USD annually. Visa sponsorship is not available. LOCATION Remote.