Agentic AI QA Specialist

A Leading Technology Innovation Group is seeking a cutting-edge Agentic AI QA Specialist to lead validation across autonomous AI lifecycles.

The Agentic AI QA Specialist must combine Model Context Protocol (MCP) tool execution verification with advanced AI observability, tracing, and quality rubric evaluation.

Key Responsibilities

• Validate the complete agentic AI lifecycle spanning user intent, prompting, reasoning, routing, and tool invocation.
• Execute rigorous testing of Model Context Protocol (MCP) tools, verifying schema payloads, orchestration, retries, and error handling.
• Utilize modern AI observability and tracing frameworks (such as LangFuse and Phoenix) to isolate anomalies and trace agent actions.
• Measure and benchmark critical AI quality metrics: hallucination detection, correctness, groundedness, determinism, and latency.
• Formulate and execute prompt testing strategies across multi-turn, multi-hop, ambiguous, negative, and guardrail scenarios.
• Verify agent memory retention, dynamic context preservation, and multi-step reasoning capabilities.
• Build golden evaluation datasets, scoring rubrics, and automated AI quality evaluation pipelines.
• Automate test execution suites using enterprise API automation frameworks like Postman, Playwright, or Katalon.

Requirements and Qualifications

• Bachelor’s degree in Computer Science, Information Technology, Software Engineering, or a related technical discipline.
• 4+ Years of experience in software quality assurance with dedicated focus on Agentic AI testing and validation.
• Proven technical understanding of MCP tool testing, API payload validation, and agent orchestration.
• Hands-on practical experience using AI observability tools (LangFuse, Phoenix, or equivalent) for agent behavior debugging.
• Demonstrated expertise evaluating LLM metrics, prompt testing methodologies, and RAG retrieval quality.