Job description
We are seeking a highly skilled Machine Learning Engineer who specializes in leveraging Large Language Models (LLMs) for automated evaluation and quality assessment. In this role, you will design and build systems that automatically measure and improve the accuracy, relevance, and consistency of model outputs. You will lead initiatives to create evaluation pipelines, develop metrics, and deliver actionable insights for continuous improvements. This position requires strong technical expertise, analytical problem-solving abilities, and the capacity to manage projects across multiple cross-functional teams.
Essential functions
Design and implement automated systems and pipelines for evaluating LLM outputs.Develop metrics and KPIs to measure output quality, accuracy, and consistency using LLM-based evaluationsCollaborate with Engineering teams to create automated logic checks and validation tools.Partner with Data Scientists to analyze evaluation results and optimize prompt and task structures.Provide feedback loops to ensure evaluation guidelines align with LLM-based assessments.Investigate how LLM-derived evaluations can enhance product reliability and user experience.Recommend refinements to prompt engineering, evaluation strategies, and automation tools.Stay informed on emerging trends in LLM evaluation, automated quality assessment, and AI toolchains.Continuously improve and expand automated evaluation processes based on industry best practices.
Qualifications
5+ years of experience in ML engineering, NLP, or AI/ML automation.Hands-on experience in prompt engineering and designing LLM-based evaluation systems is preferredStrong understanding of machine learning principles with focus on NLP and advanced LLM capabilities (e.g., Chain-of-Thought, agentic workflows)Expertise in building automated evaluation or QA pipelines.Excellent analytical and problem-solving skills with experience in root cause and error pattern analysis.Proven project management and cross-functional collaboration experience.Excellent communication skills to convey complex insights to technical and non-technical audiences.Detail-oriented mindset with a focus on evaluation metrics, prompt design, and automation.Ability to quickly adapt to new business rules and evaluation guidelines across diverse product domains.Strong programming skills in Python and SQL.Experience with big data technologies like PySpark for data aggregation and sampling is a strong plusBachelor's/Master's degree in Computer Science/ Engineering or a related field.
We offer
Opportunity to work on cutting-edge projects Work with a highly motivated and dedicated team Competitive salary Flexible schedule Benefits package - medical insurance, vision, dental, etc. Corporate social events Professional development opportunities Well-equipped office
About us
Grid Dynamics (NASDAQ: GDYN) is a leading provider of technology consulting, platform and product engineering, AI, and advanced analytics services. Fusing technical vision with business acumen, we solve the most pressing technical challenges and enable positive business outcomes for enterprise companies undergoing business transformation. A key differentiator for Grid Dynamics is our 8 years of experience and leadership in enterprise AI , supported by profound expertise and ongoing investment in data , analytics , cloud & DevOps , application modernization and customer experience . Founded in 2006, Grid Dynamics is headquartered in Silicon Valley with offices across the Americas, Europe, and India.
Prepare your application
Use the description above to assess your fit for Grid Dynamics Holdings, Inc. This checklist is guidance from Resumize, rather than additional requirements from the employer. Imported listings can be shortened or change after collection, so confirm the complete description and current availability before sending an application.
Match requirements to evidence
Identify the responsibilities and skills the employer explicitly names. For each important requirement, choose one example from your work, education, training, or projects. Explain what you did, how you did it, and what changed as a result. Include a measurable outcome when you can support it. If you lack a requirement, describe related experience accurately instead of adding an unfamiliar skill to your resume.
Put the most relevant examples near the top of your resume. Keep job titles, dates, qualifications, and contact details consistent across your application materials. Use the employer’s terminology when it describes your experience accurately; a copied list of keywords does not explain your contribution. A short, specific bullet is easier to review than a paragraph that mixes several unrelated duties.
Check the working arrangement
Confirm the location, schedule, employment type, and any eligibility requirements on the employer’s site. A remote label does not establish worldwide eligibility or flexible hours. If compensation, benefits, equipment, travel, or contract duration are missing from this listing, write down your questions for the recruiter. Do not assume details from another opening at the same company apply to this role.
Review before submitting
Follow the employer’s requested file format and application instructions. Open your exported resume and check that its text is selectable, its sections read in order, and its links work. Proofread names, dates, and contact information. Share confidential work samples only when you have permission, and use an anonymized example when necessary.
Submit through a trusted employer or recruiting destination. If the original listing has disappeared, search the employer’s careers page for the title rather than assuming the vacancy remains open. Save the application date, job reference, and the version of your resume you sent. Those details help you prepare for a later conversation and avoid submitting conflicting information through several job boards.
Review your resume for writing and readability feedback, or browse other openings if this opportunity is unavailable.