Senior AI Quality & Evaluation Engineer at SolvdSolvd · USA

Workplace: Remote

Type: fulltime

Posted: 2026-11-13

Skills: Python, AWSSolvd Inc. is a rapidly growing AI-native consulting and technology services firm delivering enterprise transformation across cloud, data, software engineering, and artificial intelligence. We work with industry-leading organizations to design, build, and operationalize technology solutions that drive measurable business outcomes. Following the acquisition of Tooploox, a premier AI and product development company, Solvd now offers true end-to-end delivery—from strategic advisory and solution design to custom AI development and enterprise-scale implementation. Our capability centers combine deep technical expertise, proven delivery methodologies, and sector-specific knowledge to address complex business challenges quickly and effectively. We are looking for a Senior AI Quality & Evaluation Engineer to own the independent proof that retrieval works and that permissions hold. You’ll build and run the golden question suite, negative entitlement tests, and drift alarms that serve as acceptance gates — so the engineers who built the system are not grading their own work. This is a 50% allocation role across weeks 3–10, drawn from a Quality Engineering or GRC background, reporting to the POD Lead and working day-to-day with the retrieval engineer. WHAT YOU’LL DO - Build the golden question suite from top contact drivers and needs workshops — question, expected source, expected passage, and acceptance thresholds for relevance, grounding, and citation accuracy, versioned alongside the content it tests. - Build the negative entitlement test set — queries issued under unentitled identities that must return no restricted passage, with any failure raised as a severity-one alarm. - Build freshness and retirement tests — source-modified-to-index-available lag per item against the agreed threshold, and verification that retired items disappear on the next sync. - Automate the harness to run on schedule and after every content, configuration, or model change — publishing scores to t…Sign in to see the original postingBrowse more developer jobs on RoleBurst