Lead Engineer, AI Platform at Circle

About the Role

The AI Quality engineering team at Circle is responsible for owning the foundation that measures, diagnoses, and improves the quality of Circle's AI-powered features. This team is focused on building the necessary infrastructure to measure, diagnose, and improve production AI systems, steering clear of ML research or model training.

We are seeking a Lead Engineer to assist us in building out the evaluation frameworks, observability tooling, and diagnostic infrastructure. This infrastructure must clearly indicate whether our AI Agents are performing well, pinpoint areas for improvement, and guide efforts to enhance speed and cost efficiency. This is a hands-on player-coach position, and you will also lead and manage the AI Quality engineering team, guiding an ambitious and growing roadmap.

If you are excited about ensuring AI systems operate reliably, efficiently, and at scale, this role is for you.

What You Will Do

• Build and own our evaluation infrastructure, including designing CI/CD pipelines, scorers, and datasets to validate the performance of Circle's AI agents, from single tool calls to complete multi-turn conversations.
• Diagnose precise locations where quality fails, tracing issues across the agent pipeline, including plan creation versus execution, tool selection, and trajectory in complex areas like workflows, site builder, and analytics. You will translate findings into prototypes that solve issues or define priorities for the AI core engineering team.
• Grow the datasets essential for evaluation by establishing annotation workflows and building out AI generated and simulated conversations to accelerate product coverage beyond manual labeling capabilities.
• Conduct structured experiments across prompts, models, and the agent harness. You will evaluate new and open-source models against our production baseline, build the framework for model transition decisions, and drive cost and latency gains through model swapping, caching, and routing based on plan complexity.
• Lead and scale the AI Quality engineering team by setting the technical direction and managing daily priorities while remaining actively involved in coding.
• Collaborate closely with the AI Core engineering team to provide them with concrete assurance that their changes improve quality rather than just introducing new features.

What You Need to Succeed

• A minimum of 7 years of experience shipping production software, with strong preference given to roles involving LLM powered agents that perform real actions within a product. You must have experience with complex, tool-using systems (multiple tools, planning, or sub-agents) beyond simple single-turn assistants, and you should be prepared to detail a project you shipped and how you verified its effectiveness.
• Familiarity with Ruby on Rails or Python, or a proven capacity to learn these quickly. Ruby on Rails serves as our production system and the foundation upon which Circle's AI Agents are constructed, while proficiency in Python is highly valued, particularly for data and evaluation tasks.
• Experience developing evaluation or observability infrastructure for Machine Learning or AI systems, including building eval pipelines, scorers, dashboards, or CI/CD processes for evaluations. Familiarity with evaluation frameworks such as Braintrust, LangSmith, or similar tools is required.
• Experience designing datasets, annotation workflows, or labeling pipelines specifically for ML/AI evaluation. You must understand how to transform raw examples into a reliable dataset through human annotation, AI generation, or simulation.
• A proven ability to learn rapidly, experiment aggressively, and thrive in a highly dynamic environment. You must be comfortable running numerous experiments and accepting outcomes, including those that fail initially.
• Comfort in fast paced settings characterized by ambiguity, necessitating the ability to quickly master new technologies as projects demand it.
• Strong alignment with Circle's company values.
• Proficiency in English across all forms (speaking, writing, reading) at a CEFR Level C2 / ILR Level 5.

Compensation and Benefits

Circle offers compensation benchmarked to U.S. standards globally, along with equity in the company through ongoing refresh grants, and 35 days of paid time off annually. As a remote-only organization, the team gathers twice a year for company retreats in world class destinations. Beyond this incredible flexibility and autonomy, we provide a comprehensive benefits package supporting health, wellbeing, and professional development.

Learn More

• Candidate Safety & Interview Process
• Diversity, Equity & Inclusion
• How We Use Candidate Data
• Equal Employment Opportunity

Visit our Candidate Hub to gain further insight into working at Circle, our benefits, and our hiring process.