Dell Technologies · KA, India
Skills: Python, CI/CD
AI Evaluation Engineer
The Security Resiliency Organization builds the AI-native capabilities that are replacing manual security processes across Dell, and every one of those capabilities has to earn its release through evidence, not demonstration. This team owns the quality bar for that shift: defining what "good enough" means for each workflow and building the harnesses that prove it, before and after release.
Join us as an AI Evaluation Engineer on our Security Resiliency team in Bangalore, India to do the best work of your career and make a profound social impact.
What you’ll achieve
You will own the release gate for AI-native security capabilities, deciding what evidence is required before a workflow can go live. You will build evaluation harnesses that measure model and agent behaviour objectively, not by sampling outputs and forming an impression. You will turn evaluation into a repeatable, measurable analytics practice so that every workflow after the first costs less to evaluate than the one before it.
You will:
Define and record acceptance criteria for each capability with the functional owner before evaluation begins, and specify the evidence, ground truth and coverage required to support the claim. Build and maintain evaluation harnesses in Python covering dataset construction, test orchestration, scoring and reporting, and integrate them as automated quality gates in CI/CD. Design measurable analytics for models and agentic tools – quantifiable metrics for accuracy, consistency across repeated runs, confidence-versus-accuracy calibration, latency and cost – using tooling such as LangFuse for trace-level evaluation and embedding-based similarity scoring for output comparison. Own the release gate: apply the criteria to the evidence, record the result, and block releases that do not clear the threshold, with thresholds set proportionate to the consequence of an incorrect agent action. Run ongoing regression evaluation after release to detect behavioural drift f…
Sign in to see the original posting
Browse more developer jobs on RoleBurst