Amazon is hiring an AI Benchmarking Lead for its Seller Assistant team in Hyderabad. The role focuses on evaluating AI model outputs for relevance, correctness and completeness, while improving the consistency and reliability of the team’s evaluation process.
You will review audits completed by the core auditing team, identify gaps in evaluation rubrics, validate model failures and help improve annotation methods and standard operating procedures. The role also involves working with program managers and applied scientists to track quality and productivity metrics.

What You’ll Do

• Quality-check audits completed by the core auditing team and measure auditor accuracy.
• Identify gaps and unclear areas in evaluation rubrics that can lead to inconsistent results.
• Validate and categorize AI model failures to help identify product issues.
• Support annotation activities across ML data processes and ensure quality execution.
• Update annotation methods and standard operating procedures when process improvements are identified.
• Document SOP changes, obtain approvals, communicate updates and check whether the new process is followed correctly.
• Test new SOPs and tools and provide feedback on their quality and usefulness.
• Collect and analyse data, track agreed metrics and report progress to stakeholders.
• Identify process and tooling issues and recommend improvements to productivity and quality.

What You Need

• Bachelor’s degree or equivalent qualification in a related field.
• Experience with natural language data labeling, data annotation, linguistic annotation or other forms of data markup.
• Proficiency in Microsoft Excel.
• Basic understanding of SQL and Python.
• Experience with Microsoft Office applications.
• Strong written and verbal English communication skills.
• Knowledge of SOA and processes related to sellers.

Preferred Qualifications

• 1–3 years of equivalent experience.
• Experience performing annotation tasks across ML data processes.
• Strong process documentation and analysis skills.
• Ability to use SQL and Python for data analysis.
• Strong analytical and problem-solving skills.
• Ability to work independently and as part of a team.

Before You Apply

• Highlight any experience with data annotation, linguistic annotation, data labeling or ML data workflows.
• Make sure your resume clearly shows your Excel, SQL and Python experience.
• Prepare examples of process documentation, quality checks or SOP improvements you have worked on.
• Be ready to explain how you identify inconsistencies in data or evaluation results.
• Prepare examples of how you tracked metrics and used analysis to improve quality or productivity.

Skills

Data Annotation, Data Labeling, Excel, SQL, Python, Data Analysis, Process Documentation, Quality Assurance, Natural Language Processing, Problem Solving