Software Engineer – AI Evaluation Expert (Contractual)
Free to start · No card · 5 credits the moment you sign up
Job description
About Patronus AI
Patronus AI is a frontier lab developing simulation research and infrastructure to accelerate progress toward human-aligned AGI. We are on a mission to simulate all of the world’s intelligence.
We are the team behind some of the earliest and most influential research in AI evaluation likeFinanceBench, Lynx,SimpleSafetyTests,CopyrightCatcher,Humanity’s Last Exam, and more. We are formerly AI researchers and engineers from companies like Meta AI, Amazon AGI, and Google. Our customers include foundation model labs and Fortune 500 enterprises like Adobe. We are backed by top-tier investors like Lightspeed Venture Partners, Notable Capital, Stanford University, Noam Brown, Gokul Rajaram, and more.
About the roleWe are looking for experienced software engineers (task-based contract) to design realistic, challenging tasks for frontier AI models and to judge how well those models handle them. You'll apply your day-to-day engineering judgment to show where AI gets code, reasoning and technical instructions right, and where it falls short.
What you’ll do
Depending on the project, you may:
- Design realistic, multi-step coding and technical tasks that test an AI model's ability to reason, write code and follow precise instructions.
- Build tasks around realistic workspace materials such as codebases, technical specs, architecture docs and datasets.
- Evaluate AI-generated code and technical content for correctness, efficiency, clarity and alignment with real-world engineering practice.
- Grade model outputs against rubrics and ground-truth answers, and write clear, structured feedback.
- Help improve how AI understands programming, system design, data workflows and engineering best practices.
- Work independently and asynchronously, meet deadlines, and collaborate with the Patronus team to keep quality high.
- Test and refine content to reflect real-world complexity.
- Review other contributors’ work for accuracy, clarity, and practical relevance.
- Incorporate feedback promptly and flag gaps or unclear guidance.
What we’re looking for
- 2+ years of professional experience in software engineering, data science or analytics.
- Experience with system design, data pipelines, ML or LLM-based applications.
- Prior work in code review, technical writing, AI training or model evaluation.
- Strong coding skills in at least one major language, such as Python, Java, C++, JavaScript, TypeScript or Go.
- Experience building, reviewing and debugging production-grade code.
- Ability to follow detailed technical guidelines, work independently, respond to feedback, and meet deadlines.
- Excellent written communication and close attention to detail.
- A bachelor's degree in computer science, engineering or a related field, or equivalent experience.
Our standards
- All tasks and solutions must be human-authored. LLM-generated content is not permitted.
- Tasks must be original and non-searchable.
- Accurate, carefully checked contributions.
- Quality over volume.
Pay: $30–$100 per hour and some tasks would have a flat rate
Location: Remote (India)
Type: Hourly, project/task-based contract (not a full-time role)
Commitment: Flexible, typically 15–20 hours per week; can be done alongside a full-time job
Applications are reviewed on a rolling basis. Placement depends on project needs and assessment results.
Patronus AI is an equal opportunity employer. We celebrate diversity in our workplace, and all qualified applicants will receive consideration for employment without regard to age, ancestry, color, family or medical care leave, gender identity or expression, genetic information, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran status, race, religion, sex (including pregnancy), sexual orientation, or other legally protected characteristics.
By clicking ‘Apply’, you agree to Greenhouse's Terms of Service and Privacy Policy.
By clicking 'Apply', you agree to Patronus AI, Inc. Privacy Policy.
Originally posted on Himalayas
Prepare your application
Use the description above to assess your fit for Patronus AI. This checklist is guidance from Resumize, rather than additional requirements from the employer. Imported listings can be shortened or change after collection, so confirm the complete description and current availability before sending an application.
Match requirements to evidence
Identify the responsibilities and skills the employer explicitly names. For each important requirement, choose one example from your work, education, training, or projects. Explain what you did, how you did it, and what changed as a result. Include a measurable outcome when you can support it. If you lack a requirement, describe related experience accurately instead of adding an unfamiliar skill to your resume.
Put the most relevant examples near the top of your resume. Keep job titles, dates, qualifications, and contact details consistent across your application materials. Use the employer’s terminology when it describes your experience accurately; a copied list of keywords does not explain your contribution. A short, specific bullet is easier to review than a paragraph that mixes several unrelated duties.
Check the working arrangement
Confirm the location, schedule, employment type, and any eligibility requirements on the employer’s site. A remote label does not establish worldwide eligibility or flexible hours. If compensation, benefits, equipment, travel, or contract duration are missing from this listing, write down your questions for the recruiter. Do not assume details from another opening at the same company apply to this role.
Review before submitting
Follow the employer’s requested file format and application instructions. Open your exported resume and check that its text is selectable, its sections read in order, and its links work. Proofread names, dates, and contact information. Share confidential work samples only when you have permission, and use an anonymized example when necessary.
Submit through a trusted employer or recruiting destination. If the original listing has disappeared, search the employer’s careers page for the title rather than assuming the vacancy remains open. Save the application date, job reference, and the version of your resume you sent. Those details help you prepare for a later conversation and avoid submitting conflicting information through several job boards.
Review your resume for writing and readability feedback, or browse other openings if this opportunity is unavailable.