Senior AI Test Engineer
Free to start · No card · 5 credits the moment you sign up
Job description
This is a remote position.
Senior AI Test Engineer
Location: Remote
About the Role
Alternative Path is looking for a Senior AI Test Engineer to lead quality assurance on a client engagement with a leading financial data and analytics company that has built AI agent systems (chatbots, RAG-based research/report generation, MCP-based tool-calling agents) on top of its proprietary datasets for its own clients. This is a senior, client-facing role: you'll own the test strategy for these systems end-to-end, working directly with the client's engineering team to validate accuracy, reliability, and safety before these agents reach financial industry end-users — where hallucinations or bad data retrieval carry real business risk.
This is not conventional QA. You'll be building evaluation frameworks and datasets, running systematic evals, and defining quality bars for non-deterministic, LLM-driven systems operating over complex financial data.
What You'll Do
- Own the test and evaluation strategy for the client's AI agent systems — chatbots, RAG-based research/report generation, and MCP-based tool-calling agents
- Design and build eval suites covering accuracy, groundedness/faithfulness to source data, task completion, latency, and safety
- Build and curate golden datasets reflecting real client use cases and financial-data edge cases (ambiguous queries, stale data, conflicting sources, numerical precision)
- Use eval/observability tooling such as LangSmith (or equivalents — Ragas, DeepEval, Braintrust, Langfuse, PromptFoo) to trace, score, and monitor agent runs
- Define and track RAG-specific metrics (retrieval precision/recall, hallucination rate, citation/source accuracy) — critical given the systems sit on top of financial datasets where correctness is non-negotiable
- Evaluate MCP-based tool-calling behavior — correct tool selection, multi-step task execution, failure handling
- Design regression testing so prompt, model, or data-pipeline changes don't silently degrade quality
- Run structured red-teaming for hallucinations, prompt injection, and unsafe/incorrect financial outputs
- Act as the primary quality point of contact with the client's engineering stakeholders — translating findings into clear, actionable risk assessments
- Mentor/guide any junior QA resources staffed onto the engagement as it scales
- Build automation to integrate evals into CI pipelines where feasible
What We're Looking For
Must-haves
- 3+ years in software QA/testing, with hands-on senior-level experience evaluating LLM-based or agentic AI systems (not just traditional QA)
- Direct experience with eval frameworks/tools — LangSmith strongly preferred; also acceptable: Ragas, DeepEval, PromptFoo, Braintrust, Langfuse
- Strong understanding of RAG architecture and how to independently evaluate retrieval quality vs. generation quality
- Practical familiarity with MCP (Model Context Protocol) or comparable agent tool-calling frameworks
- Strong Python skills for building test harnesses, scoring pipelines, and automation
- Experience designing evaluation approaches for non-deterministic systems (probabilistic scoring, not binary pass/fail)
- Comfortable working directly with client engineering teams — strong communication, able to explain quality risk to both technical and business stakeholders
- Prior experience in a client services / consulting QA environment (this is a client-facing engagement, not an internal product team)
Nice-to-haves
- Experience testing AI systems in fintech, capital markets, or data/analytics domains
- Familiarity with financial data accuracy/compliance considerations (numerical precision, source traceability, audit trails)
- Exposure to LLM red-teaming/safety evaluation practices
- Experience with browser-automation agent testing (Playwright or similar)
Why This Role Matters
The client's end-users rely on these agents for financial research and decision-making — the cost of a hallucinated number or a wrong source is high. You'll be the senior technical authority ensuring these systems are trustworthy before they ship, working at the intersection of AI evaluation and financial-data correctness.
Originally posted on Himalayas
Prepare your application
Use the description above to assess your fit for Alternative Path Private Limited. This checklist is guidance from Resumize, rather than additional requirements from the employer. Imported listings can be shortened or change after collection, so confirm the complete description and current availability before sending an application.
Match requirements to evidence
Identify the responsibilities and skills the employer explicitly names. For each important requirement, choose one example from your work, education, training, or projects. Explain what you did, how you did it, and what changed as a result. Include a measurable outcome when you can support it. If you lack a requirement, describe related experience accurately instead of adding an unfamiliar skill to your resume.
Put the most relevant examples near the top of your resume. Keep job titles, dates, qualifications, and contact details consistent across your application materials. Use the employer’s terminology when it describes your experience accurately; a copied list of keywords does not explain your contribution. A short, specific bullet is easier to review than a paragraph that mixes several unrelated duties.
Check the working arrangement
Confirm the location, schedule, employment type, and any eligibility requirements on the employer’s site. A remote label does not establish worldwide eligibility or flexible hours. If compensation, benefits, equipment, travel, or contract duration are missing from this listing, write down your questions for the recruiter. Do not assume details from another opening at the same company apply to this role.
Review before submitting
Follow the employer’s requested file format and application instructions. Open your exported resume and check that its text is selectable, its sections read in order, and its links work. Proofread names, dates, and contact information. Share confidential work samples only when you have permission, and use an anonymized example when necessary.
Submit through a trusted employer or recruiting destination. If the original listing has disappeared, search the employer’s careers page for the title rather than assuming the vacancy remains open. Save the application date, job reference, and the version of your resume you sent. Those details help you prepare for a later conversation and avoid submitting conflicting information through several job boards.
Review your resume for writing and readability feedback, or browse other openings if this opportunity is unavailable.