Research Crawling Engineer

ABOUT THE ROLE Join a data infrastructure team building large-scale web data collection systems for AI research and model development. This role focuses on reliable, high-volume crawling and dataset preparation across diverse and changing web environments. WHAT YOU'LL DO - Build and maintain web crawlers across social, travel, and multilingual websites. - Design high-throughput, fault-tolerant systems capable of collecting millions to billions of URLs per day. - Address anti-bot systems, rate limits, and dynamic websites using browser-based and other crawling techniques. - Develop pipelines to clean, deduplicate, filter, and normalize web data at terabyte to petabyte scale. - Collaborate with research teams to create datasets suited to model training and research needs. - Monitor crawl coverage, performance, and data quality, adapting systems as websites change. - Optimize crawling infrastructure for cost, latency, and reliability across cloud or bare-metal environments. WHAT WE'RE LOOKING FOR - At least 2 years of experience building web crawlers or large-scale data pipelines; typically 4 or more years, with 2 or more years considered for candidates with a PhD. - Experience operating data pipelines at terabyte scale or larger, including high-volume crawling or data acquisition. - Strong programming skills in Go, Rust, Python, Java, or C++. - Deep knowledge of HTTP, networking, browser behavior, distributed systems, and parallel processing. - Hands-on experience navigating anti-bot systems, rate limits, and JavaScript-heavy websites. - Experience with headless browsers such as Playwright or Puppeteer, proxy systems, and IP rotation. - Background in data quality evaluation, web data processing, or cloud and bare-metal infrastructure. - Experience with NLP pipelines or machine learning dataset curation is a plus. COMPENSATION & BENEFITS Annual salary range: $160,000 to $250,000 USD. Visa sponsorship is not available. LOCATION Remote.