Senior AI QA Automation Engineer at encora10encora10 · Costa Rica

Workplace: Remote

Posted: 2026-10-12

Skills: Python, CI/CD, TestingJob Title: Senior AI QA Automation Engineer Key Skills: Python, AI Testing, Test Automation, Playwright, Selenium, API Testing, Azure AI Foundry Evaluations, RAG Evaluation, Prompt Testing, CI/CD, Monitoring Experience: 5+ YOE. Location: Costa Rica, Peru, Colombia, Bolivia. Mode: Remote We at Coforge are hiring Senior AI QA Automation Engineer (006fu00000bWsOCAA0-4) with the following skill set. Key Responsibilities Build automated evaluation frameworks for AI agents, RAG solutions, prompts, tools, workflows, and model outputs, including benchmark datasets, golden responses, regression suites, evaluation metrics, and release thresholds. Test AI-specific behaviors including hallucinations, grounding, citation accuracy, prompt injection, tool selection, and multi-agent coordination. Develop agent-powered QA solutions for test-case generation, API and workflow validation, and intelligent UAT, while building UI, API, integration, performance, and end-to-end automation. Implement CI/CD quality gates and continuously monitor production AI quality, drift, reliability, and recurring failure patterns. Required Skills & Qualifications 5+ years of experience in Software Quality Engineering, Test Automation, or Reliability Engineering. Strong Python skills with experience building reusable automated testing frameworks. Hands-on experience with Playwright and/or Selenium. Strong experience with API, UI, integration, regression, end-to-end, and CI/CD-based test automation. Experience with Azure AI Foundry Evaluations, RAG evaluation, and prompt testing. Experience defining evaluation metrics, benchmark datasets, regression suites, and measurable quality thresholds. Knowledge of AI quality concepts such as hallucination testing, grounding, citation accuracy, prompt injection, and agent behavior validation. Experience implementing CI/CD quality gates and production monitoring. Ability to translate business requirements, workflows, and AI behaviors into measurable quality criteria.…Sign in to see the original postingBrowse more developer jobs on RoleBurst