AI Agent Evaluation Engineer - Remote

YO AI LabsAnywhereAdded 2d ago

AI Evaluation Engineer

Free to start · No card · 5 credits the moment you sign up

Job description

Senior Software Engineer Job Type: Contractor (:15 hours/week) Location: Remote Job Summary We are seeking experienced Senior Software Engineers to support an AI training project by creating reinforcement learning environments that evaluate AI models on complex software engineering tasks using Model Context Protocol (MCP) tools. You will design reproducible environments, deterministic verification, and reference solutions for tasks such as bug fixing, feature implementation, codebase refactoring, and performance optimization. No prior AI experience is required. Key Responsibilities Create reinforcement learning environments for software engineering tasks.Design tasks involving bug fixing, feature development, refactoring, and performance optimization.Build deterministic verification systems and golden reference solutions.Evaluate AI agents' ability to reason through complex codebases and use MCP tools effectively.Develop realistic, reproducible software engineering scenarios.Ensure tasks accurately measure coding ability, problem-solving, and tool usage.Document solutions and provide clear technical feedback. Required Skills Strong proficiency in Python 3, Java, Rust, C++, or TypeScript.Strong understanding of algorithms and data structures.Experience with bug fixing and debugging complex software issues.Proven experience in feature implementation and codebase refactoring.Strong knowledge of performance optimization and tuning.Excellent written and verbal communication.Strong attention to detail. Preferred Qualifications Experience working with large-scale or distributed codebases.Familiarity with AI/ML systems is a plus but not required.Experience with rigorous code reviews and software engineering best practices.Experience working effectively in remote or cross-functional teams. Hiring Process Submit an application and screening questions.Complete an AI interview (:30 minutes).Complete a technical assessment, if required.Hiring Manager review. Compensation Compensation is output-based, with payment provided per task that meets project specifications. Minimum weekly submission requirements may apply. Availability Selected experts should be prepared to begin their first tasks within 24–48 hours of completing onboarding.

Prepare your application

Use the description above to assess your fit for YO AI Labs. This checklist is guidance from Resumize, rather than additional requirements from the employer. Imported listings can be shortened or change after collection, so confirm the complete description and current availability before sending an application.

Match requirements to evidence

Identify the responsibilities and skills the employer explicitly names. For each important requirement, choose one example from your work, education, training, or projects. Explain what you did, how you did it, and what changed as a result. Include a measurable outcome when you can support it. If you lack a requirement, describe related experience accurately instead of adding an unfamiliar skill to your resume.

Put the most relevant examples near the top of your resume. Keep job titles, dates, qualifications, and contact details consistent across your application materials. Use the employer’s terminology when it describes your experience accurately; a copied list of keywords does not explain your contribution. A short, specific bullet is easier to review than a paragraph that mixes several unrelated duties.

Check the working arrangement

Confirm the location, schedule, employment type, and any eligibility requirements on the employer’s site. A remote label does not establish worldwide eligibility or flexible hours. If compensation, benefits, equipment, travel, or contract duration are missing from this listing, write down your questions for the recruiter. Do not assume details from another opening at the same company apply to this role.

Review before submitting

Follow the employer’s requested file format and application instructions. Open your exported resume and check that its text is selectable, its sections read in order, and its links work. Proofread names, dates, and contact information. Share confidential work samples only when you have permission, and use an anonymized example when necessary.

Submit through a trusted employer or recruiting destination. If the original listing has disappeared, search the employer’s careers page for the title rather than assuming the vacancy remains open. Save the application date, job reference, and the version of your resume you sent. Those details help you prepare for a later conversation and avoid submitting conflicting information through several job boards.

Review your resume for writing and readability feedback, or browse other openings if this opportunity is unavailable.