About the Role
We are seeking a highly skilled senior python developer
to drive innovation and delivery within our
Web Harvesting team
. This role will lead efforts to build, scale, and enhance automated web scraping solutions powered by
AI-assisted configuration generation
. You will mentor junior developers, accelerate delivery timelines, and help architect solutions that handle increasingly complex data sources.
Our systems ingest data from multiple content types while ensuring AI is used responsibly – only to generate scraper configurations, not to extract the data itself. These configurations are transformed into Python-driven pipelines that power our harvesting at scale.
Key Responsibilities
- Lead the design, development, optimization and technical direction of Python-based web harvesting pipelines.
- Mentor and guide junior Python developers
to improve skills and accelerate delivery. - Collaborate with AI agent developers to improve JSON configuration generation and integrate them seamlessly into harvesting workflows.
- Extend and enhance configuration frameworks to handle more complex and dynamic web sources.
- Design and oversee ingestion strategies for structured and unstructured offline files
- Architect scalable solutions in
AWS cloud environments
, leveraging
Airflow
,
S3
,
Postgres
, and API-driven services. - Ensure best practices in data quality, reliability, and maintainability of harvesting pipelines.
- Partner with product and data teams to align harvesting output with business needs.
Required Skills & Experience
- 7+ years of experience in software engineering, with strong expertise in
Python
. - Proven
leadership/mentorship
experience guiding and developing junior engineers. - Deep understanding of
web scraping, HTML parsing, and handling complex document structures
(PDF, DOCx, CSV). - Strong experience with
AWS services
(S3, ECS, Lambda, IAM, etc.) and workflow orchestration tools (preferably Airflow). - Proficiency with relational databases (
Postgres
) and building robust data ingestion pipelines. - Knowledge of
REST API design and integration. - Ability to architect systems for scale, resilience, and performance
.
Preferred Skills
- Experience working with
AI/ML systems
for automation (e.g., using AI for config or workflow generation). - Familiarity with distributed systems and cloud-native design patterns.
- Strong background in software engineering best practices (CI/CD, testing, code reviews).