AI Evaluation Engineer Job at Motion Recruitment, Raritan, NJ

  • Motion Recruitment
  • Raritan, NJ

Job Description

Our client, a global pharmaceutical, biotechnology, and medical technology company, is actively looking for an experienced AI Evaluation Engineer to join their team! This role is 100% remote, however they are only considering candidates currently on the East Coast.

***This is a 6 month initial contract with the opportunity to extend based on business needs***

This Engineer will build and run the evaluations for a GenAI regulatory science co-pilot used by regulatory submission teams. The system works like a virtual Health Authority reviewer. It extracts and assesses regulatory risks, maps submission content to relevant guidance and precedent, checks that key messages are supported by the underlying clinical data, and anticipates likely Health Authority questions. Because its outputs directly inform regulatory submission strategy, they must be accurate, grounded in cited sources, and traceable. You will work with regulatory affairs subject-matter experts (SMEs) and the product engineering team.

Responsibilities

  • Build gold sets with SMEs: annotate regulatory and clinical documents with the expected outputs, write annotation guidelines, run adjudication, and measure inter-rater agreement.
  • Evaluate extraction: score risk extraction for precision, recall, and completeness, and compare successive engine versions against a baseline.
  • Evaluate retrieval and ranking: measure retrieval recall, ranking quality, and question coverage when the system retrieves regulatory guidance, precedent, and prior Health Authority interactions.
  • Evaluate grounding and traceability: build code-based checks that every finding traces back to a valid source and citation, and detect fabricated guidance, fabricated precedent, and unsupported regulatory claims.
  • Evaluate reasoning with LLM judges: design and validate LLM-as-judge rubrics for regulatory relevance, evidence sufficiency, risk assessment, and explanation quality, calibrated against SME judgment.
  • Evaluate each stage of the multi-stage pipeline (evidence structuring, question coverage, retrieval, ranking, mapping, grounding), plus an end-to-end review with SMEs.
  • Turn failure analysis into clear readouts for the product team, and package the evaluations as regression suites that run in CI/CD.

Required Skills & Experience

  • 5+ years of hands-on work in AI/ML evaluation, NLP, or applied data science, including recent delivery on LLM, RAG, or agent systems.
  • Experience building evaluation frameworks and gold datasets for information extraction and retrieval/ranking systems (please be ready to share examples).
  • Strong grounding in evaluation metrics and statistics: precision/recall/F1, Recall@K, MRR, nDCG, inter-rater reliability (e.g., Cohen’s kappa or Krippendorff’s alpha), and confidence intervals for small evaluation sets.
  • Experience designing and validating LLM-as-judge systems against human expert ratings.
  • Experience running annotation projects with domain experts, including guidelines, adjudication, and quality control.
  • Strong Python, with experience using evaluation harnesses (e.g., Ragas, DeepEval, promptfoo, Inspect, or in-house equivalents), LLM APIs, and Git/CI/CD.

Preferred Experience

  • Regulatory affairs, regulatory intelligence, or submissions experience: FDA/EMA guidance, CTD structure (e.g., Modules 2.5 and 2.7), Health Authority questions, and labeling.
  • Experience evaluating AI in regulated (GxP) environments.
  • Familiarity with clinical trial outputs (CSRs, TLFs, statistical outputs) and drug development.
  • Publications or open-source work in AI evaluation, information retrieval, or biomedical NLP.

Job Tags

Full time, Contract work, Remote work

Similar Jobs

Kettering Health Network

Chief Medical Information Officer (CMIO) Job at Kettering Health Network

 ...leadership of clinical technology initiatives across a $3B integrated healthcare system. As the bridge between frontline clinicians and...  ...(MBA), Health Administration (MHA), or Biomedical Informatics.Work ExperienceMinimum of 5-7 years of progressive leadership... 

State Personnel Board

Presiding Administrative Law Judge Job at State Personnel Board

 ...Job Description and Duties Under the general direction of the Chief Administrative Law Judge of the State Personnel Board, the incumbent will plan, coordinate, and direct the work of SPB Administrative Law Judges in conducting hearings and other proceedings within... 

Tazco Soil Company

Agricultural Equipment Operators Job at Tazco Soil Company

 ...feed lines, manifolds, tanks and related fertilizer equipment; clean residue; inspect piping, valves, fittings...  ...spring fertilizer season. Maintain and repair agricultural commodity and equipment storage buildings. Operate tractors with snow blower attachments, trucks with... 

Sila Services

Commercial HVAC Service Technician Job at Sila Services

 ...Job Description Job Description Commercial HVAC Service Technician Astacio Plumbing, Heating, Cooling & Electrical | Norwalk, CT Bring Your Commercial HVAC Expertise to a Team That Values Professionalism Commercial customers depend on their heating and cooling... 

Tier 1 Consulting, Inc.

Revit Coordinator Job at Tier 1 Consulting, Inc.

 ...Job Description Job Description Architecture and design firm seeking an experienced Project Coordinator with 5 years of AEC experience...  ...is accurate, current, and accessible. This is not an entry-level position. The firm is looking for someone who has been in the...