Senior Evaluation Algorithm Engineer at Binance

About Binance

Binance stands as a leading global blockchain ecosystem, renowned for operating the world’s largest cryptocurrency exchange by both trading volume and registered users. With the trust of over 300 million people across 100+ countries, Binance is recognized for its industry-leading security, transparent user fund management, high-speed trading engine, deep liquidity, and an extensive portfolio of digital-asset products. Our diverse offerings span trading, finance, education, research, payments, institutional services, and advanced Web3 features. We are committed to harnessing the power of digital assets and blockchain technology to cultivate an inclusive financial ecosystem, thereby advancing the freedom of money and enhancing financial access for individuals globally.

About the Role: Senior Evaluation Algorithm Engineer

In the rapidly evolving landscape of artificial intelligence, large language models (LLMs) are fundamentally transforming core business scenarios, including dialogue systems and financial trading. The continuous advancement and iteration of model capabilities are critically dependent on a robust, scientific, and trustworthy evaluation system, essentially, the 'ruler' that accurately measures model quality and effectively guides research and development (R&D) efforts. We are actively seeking a highly skilled evaluation expert with a strong algorithmic background to spearhead the development of our LLM evaluation capabilities. This pivotal role will cover diverse scenarios such as dialogue and financial trading, utilizing professional evaluation methodologies to precisely quantify model performance, pinpoint specific issues, and drive a continuous cycle of model improvement.

Key Responsibilities

• LLM Evaluation Strategy: Design comprehensive, end-to-end LLM evaluation plans for critical business scenarios like dialogue and financial trading. Establish sophisticated evaluation metric systems and rubrics that convert subjective model performance judgments into quantifiable, reproducible, and explainable evaluation conclusions.
• Dataset Leadership: Lead the design and construction of high-quality evaluation datasets. Define precise evaluation dimensions and ensure thorough scenario coverage. Develop robust data annotation guidelines and stringent quality control processes to build benchmarks that authentically reflect business needs and possess strong discriminative power.
• Performance Analysis & Improvement: Conduct in-depth analysis of model capability boundaries and identify failure modes based on evaluation results. Generate actionable improvement recommendations and actively collaborate with algorithm and product teams to drive iterative model enhancements, firmly embedding evaluation as a critical component of the R&D loop.
• Automation & Scaling: Drive the automation and scaling of evaluation workflows. Construct sustainable evaluation platforms and toolchains designed to support the demands of high-frequency, stable evaluation requirements amidst rapid model iteration cycles.
• Cross-functional Collaboration: Partner closely with algorithm, product, and data teams to translate overarching business and model objectives into clear, measurable evaluation standards. Transform evaluation findings into concrete R&D directions and proactively drive their successful implementation.

Required Qualifications and Skills

• Educational Background: Master's degree or higher in Computer Science, Artificial Intelligence, Mathematics, Statistics, or a closely related quantitative field.
• Algorithmic & LLM Foundation: Possess a solid algorithmic foundation coupled with a deep understanding of Large Language Model (LLM) principles, including training and fine-tuning processes.
• Industry Experience: Proven hands-on LLM evaluation experience gained at a large technology company, specifically involving participation in commercial deployment evaluation, not solely academic research or offline benchmarking. Familiarity with the complete evaluation pipeline, from data preparation and rubrics design to evaluation-driven R&D cycles.
• Evaluation Methodologies: Proficient with mainstream evaluation methods such as human evaluation, model-based automatic evaluation (e.g., LLM-as-a-judge), and metric computation, understanding their applicable boundaries. Ability to define appropriate evaluation dimensions for diverse business scenarios and craft clear, actionable, and discriminative rubrics.
• Data Control & Reliability: Systematic control over evaluation data representativeness, annotation consistency, and result reliability to ensure scientific and trustworthy evaluation conclusions.
• Programming Proficiency: Expert in Python, with practical experience in automating evaluation workflows, constructing benchmarks, or developing evaluation platforms. Capable of independently managing data processing, writing evaluation scripts, and performing thorough result analysis.
• Business Acumen & Communication: Strong business understanding and excellent communication skills, enabling the translation of complex evaluation findings into clear improvement directions and effective cross-team collaboration.

Bonus Qualifications

• Experience in evaluating dialogue systems, AI Agents, or specialized financial/trading LLMs.
• Experience in building high-quality AI training/evaluation data or advanced data annotation systems.
• Familiarity with Reinforcement Learning from Human Feedback (RLHF), reward models, or preference data-related work.

Why Join Binance

Embark on a journey to shape the future within the world's leading blockchain ecosystem. Collaborate with world-class talent in a user-centric global organization characterized by a flat hierarchical structure. You will tackle unique, fast-paced projects with significant autonomy within an innovative and dynamic environment. Thrive in a results-driven workplace that offers abundant opportunities for career growth and continuous learning. We provide a competitive salary and comprehensive company benefits, along with a work-from-home arrangement (which may vary based on the specific business team's work nature).

Binance is deeply committed to being an equal opportunity employer, recognizing that a diverse workforce is fundamental to our continued success. By submitting your job application, you confirm that you have carefully read and agreed to our Candidate Privacy Notice.