Senior Applied Scientist

Job summary

The OCI AI Evaluation Science team builds the evidence behind model-selection, product-readiness, and launch decisions. We evaluate frontier foundation models and AI systems across capabilities such as reasoning, coding and agentic coding, retrieval-augmented generation, AI agents, NL2SQL, multimodal understanding, multilingual performance, and responsible AI.

Qualifications

• model access, data, or infrastructure are still evolving.
• You will develop novel benchmarks and evaluation methodologies that are publishable at top tier AI conferences.
• resolve blockers, manage dependencies, and establish clear handoffs and ownership.
• Turn successful evaluation work into reusable protocols, documented workflows, and shared infrastructure that improve the speed and consistency of future evaluations.
• Review technical work, share expertise, and mentor junior scientists or engineers in experimental design, evaluation methodology, coding, and interpretation of results.
• Own delivery quality and timelines, communicate risks early, and maintain clear, auditable documentation of experimental configurations, data versions, results, and decisions.

Responsibilities

• As a Senior Applied Scientist on the team, you will independently own complex evaluation work from problem definition, benchmark development, and final recommendation to executive leadership.
• You will write high-quality code, work with large and imperfect datasets, develop and calibrate automated evaluators, and turn one-off analyses into reproducible evaluation protocols and reusable infrastructure.
• You will examine more than aggregate benchmark scores, considering factors such as statistical validity, data provenance, contamination, robustness, cost, latency, reliability, safety, and operational constraints.
• Independently own end-to-end evaluations of foundation models, AI agents, and enterprise AI systems, from initial question and experiment design through analysis, reporting, and stakeholder review.
• Translate customer, product, and business needs into testable hypotheses, evaluation criteria, datasets, metrics, baselines, and acceptance thresholds.
• Publish original research in top-tier peer-reviewed conferences and journals, and translate relevant evaluation advances into reusable methods, technical reports, or production capabilities for OCI.
• Write production-quality evaluation code; build reproducible pipelines, test suites, automated checks, and integrations with shared evaluation platforms.
• Evaluate model and system behavior across quality, cost, latency, reliability, safety, robustness, and domain fit rather than relying only on aggregate scores.
• Conduct statistical analysis, error analysis, ablations, and qualitative failure-mode investigations to explain model behavior and identify meaningful differences between systems.
• Develop and validate automated evaluators, including LLM-as-a-judge and VLM-as-a-judge methods; calibrate them against human judgments and quantify their reliability, bias, and limitations.
• Design human-evaluation and annotation workflows, including rubrics, gold datasets, sampling plans, quality controls, and vendor or Human-in-the-Loop validation.
• Assess dataset quality, provenance, representativeness, contamination risk, licensing constraints, privacy, and other factors that could invalidate an evaluation or limit use of its results.

Skills

Degrees

Industry

Company size