Rubrik: Senior AI Evals Engineer — Bangalore

About Rubrik

Rubrik provides data-security and resilience technology for organisations facing cyberattacks, malicious activity and operational disruption. Its recruitment material discusses protecting enterprise information, including data held in business software and cloud environments. The described portfolio includes security and protection for SaaS applications, with attention to access, backup and recovery. Its business connects the technical handling of information with an organisation's ability to keep critical data protected and to recover when normal operations are disrupted.

The position and its setting

Rubrik lists Senior AI Evals Engineer in Bangalore, under reference 8273834. The details below separate the work described by the employer from essential conditions, preferred experience and practical preparation advice.

• Department: Information Technology & Services
• Employer listing date: 15 October 2026

Responsibilities and work described

• This role bridges AI development and reliable deployment — ensuring our agents are safe, ethical, effective, and meet high-quality performance standards
• A significant focus of the role is on Responsible AI and Safety Evals, including adversarial testing, red-teaming, bias detection, and toxicity measurement
• The engineer will design scalable evaluation pipelines, build synthetic test environments, and work closely with AI/ML Engineers, Researchers, and policy teams to ensure agents are production-ready and compliant with regulatory and ethical standards
• The role is 70% Automation and 30% Manual Testing, and sits within the Applied AI: Test Engineering team as a key technical authority
• Design: and build agent evaluation frameworks — create end-to-end infrastructure for evaluating LLM-based agents across correctness, latency, cost, safety, and task completion rate
• Building: synthetic testing environments and simulation strategies to stress-test AI agents under diverse real-world conditions, with specific focus on agents built on Google ADK and related Google AI/ML services (e.g., Vertex AI, Gemini models)
• Building: golden datasets and test harnesses — curate domain-specific evaluation sets for Rubrik's product verticals (data security, SaaS protection, cyber recovery, compliance)
• Establishment: eval-as-a-service — build and enhance centralized evaluation platform that any IT team members can self-serve to test their agents against standardized and custom benchmarks
• Automate regression and drift detection — implement CI/CD-integrated eval pipelines that catch regressions on every model or prompt change before production deployment
• Define eval taxonomies and rubrics — establish structured scoring criteria for agentic workflows (tool use accuracy, multi-step reasoning, hallucination rate, policy adherence, error recovery)
• Instrument production observability — build real-time monitoring for agent quality in production: success rates, user corrections, fallback triggers, escalation patterns
• Benchmark and select models — run structured model comparisons (cost, quality, latency trade-offs) to inform model selection decisions across agent use cases
• Analysis: and interpret evaluation results, providing clear, actionable quality and safety reports to stakeholders and development teams, with specific focus on risk metrics, trend analysis, and mitigation recommendations
• Define comprehensive end-to-end QA strategies — functional, integration, regression, and user acceptance testing (UAT) — tailored for conversational and goal-oriented AI agents with tool-use, memory, and multi-turn reasoning capabilities
• Development: and maintain detailed test artifacts (test plans, test cases, test scenarios, automated test suites) covering agent features, tool invocation, memory persistence, and reasoning accuracy across evolving model versions
• Identify, document, prioritize, and track defects, performance regressions, and alignment failures using Jira, collaborating closely with Applied AI Engineers to analyze root causes and validate fixes

Essential requirements and candidate background

• Deep hands-on experience with LLM evaluation — familiarity with frameworks and libraries such as LangSmith, DeepEval,LLM as a Judge, Promptfoo, RAGAS, Hugging Face evaluation tooling, custom eval harnesses
• Understanding: agentic AI patterns — tool use, function calling, multi-step planning, ReAct/CoT reasoning chains, retrieval-augmented generation (RAG)
• Production experience with eval pipelines — building automated, CI-integrated evaluation infrastructure (not just notebooks)
• Proficiency: Python — scripting and tooling skills; experience with pytest, async patterns, and data processing at scale
• Familiarity: Google Cloud Platform (GCP) or any Cloud Service relevant to AI/ML (e.g., Vertex AI) and experience integrating testing into MLOps workflows
• Statistical literacy — ability to design experiments, interpret eval metrics with appropriate confidence, and avoid common pitfalls (overfitting to benchmarks, metric gaming)
• Enterprise product sensibility — understanding of how AI quality translates to customer trust in a B2B/enterprise context (security, compliance, reliability SLAs)
• Communication — ability to present eval results and quality trade-offs to engineering leadership, articulate risks and safety findings to non-technical audiences, and drive alignment on ship/no-ship decisions
• Experience: CI/CD pipelines and Git-based workflows, including automated testing, static analysis, deployment automation, and environment configuration for our DevTestOps

Preferred experience and additional strengths

• Deep Learning/NLP Knowledge: Fundamental understanding of how LLMs work, including concepts like RAG, prompting, and the technical challenges related to alignment
• Data Analysis: Experience with data analysis and visualization tools (e.g., Pandas, SQL) for analyzing evaluation and safety results
• Policy Background: Familiarity with AI ethics principles and industry standards for responsible AI development
• Garbage-in/Garbage-out (GIGO) Tools: Any tool or framework that specializes in testing the model's response to highly ambiguous, misleading, or malformed inputs to test for robustness and safety failures
• Experience testing complex APIs, especially for tool-integration and function-calling scenarios in agentic workflows
• Experience: prompt injection (Must) and data poisoning skills is a plus
• Certifications in AI/ML, cloud platforms (GCP, AWS), AWS Machine Learning Specialty, Google Professional ML Engineer, DeepLearning.AI specializations and software testing methodologies
• Experience evaluating agents in cybersecurity, data protection, or infrastructure domains
• Familiarity: Rubrik's product portfolio or similar enterprise data management platforms
• Hands-on with human-in-the-loop eval workflows (annotation platforms, inter-rater reliability)
• Experience: model fine-tuning evaluation (pre/post quality comparison)
• Background in building internal developer platforms or tooling teams

Location, schedule and contract conditions

• Advertised location: Bangalore

Conditions requiring the complete notice

Some passages contain complex or combined conditions that have not been reduced to short eligibility statements here. Read the complete linked notice for those passages, including any alternatives, exceptions or restrictions. The concise facts above should not be treated as the full wording of every condition.

Pay and application dates

A verified salary figure is not established in this article. An online estimate for another Senior AI Evals Engineer vacancy is not an offer for reference 8273834; use the employer’s own pay disclosure and recruitment communication.

A closing date is not confirmed in this article. Check the current employer page before submitting: an undated posting can change or be withdrawn.

How to apply to Rubrik

Applications follow the official employer page linked below.

Open the official Rubrik posting below and match the title and reference 8273834.

Compare your qualifications and experience with the essential conditions for the Bangalore position, then read the preferred criteria separately.

Prepare an accurate CV and the supporting items requested in that employer’s application. Provide only information and documents you are authorised to share.

Use the employer’s application option, complete the required fields and review the submission. Retain the acknowledgement when one is provided. Job Alerts World does not receive the application.

Practical preparation: Python

Python work is easier to assess through a concrete example than a list of libraries. Explain the problem, the input data, the parts you implemented and how you checked the output. Include a repository or a short technical note when you can share it.

Relevant skills

• Python
• SQL
• CI/CD
• AWS
• GCP
• APIs
• AI/ML
• LLMs
• Java
• TypeScript
• React
• Azure
• Google Cloud