First Monday
Jobs
Jobs open to youOnly jobs that can hire where you liveA CV for each jobWritten from your record, every line checkedMemoryIt remembers, so you don’t repeat yourselfInterview helpResearch, prep and hints only you seeWatch how it works0:28
FAQDocsUse in AI, Free
Your countryNorth AmericaEuropeAsia-PacificLatin AmericaNot on the list? Tell us where you liveRequest early accessEarly access
First Monday

Remote jobs that can actually hire you.

One email to confirm your address, and another when your group is invited. Free during the beta. By requesting access you agree to the Terms & Conditions and the Privacy Policy.

Beta formacOSWindows

Jobs

Jobs open to you in CanadaRemote jobs in the USRemote jobs in PolandRemote jobs in the UKRemote jobs in CanadaRemote jobs in Mexico

Product

Jobs open to youA CV for each jobMemoryInterview helpHow it works

Company

FAQContact

Legal

Privacy PolicyTerms & Conditions
Jobs
Jobs open to you in CanadaRemote jobs in the USRemote jobs in PolandRemote jobs in the UKRemote jobs in CanadaRemote jobs in Mexico
Product
Jobs open to youA CV for each jobMemoryInterview helpHow it works
Company
FAQContact
Legal
Privacy PolicyTerms & Conditions
Remote jobs by country
United StatesPolandUnited KingdomCanadaMexicoBrazilGermanySpainPortugalIrelandNetherlandsIndiaSwedenBelgiumAustriaDenmarkFinlandSwitzerlandNorwayAustraliaNew Zealand

Remote jobs by country

United StatesPolandUnited KingdomCanadaMexicoBrazilGermanySpainPortugalIrelandNetherlandsIndiaSwedenBelgiumAustriaDenmarkFinlandSwitzerlandNorwayAustraliaNew Zealand
© 2026 First MondayNo tracking cookies. Anonymous page statistics only.
First Monday
Jobs/Data and AI/Tech Lead Data Scientist, AI Evaluation & Monitoring

Tech Lead Data Scientist, AI Evaluation & Monitoring

Geisinger

Remote · Pennsylvania, USData and AIPosted April 29, 2026
Probably open to people in the United StatesApply for this job Write my CV for this job Early access
Pay
Not stated
Required
Python, SQL, Experimental design, Causal inference, fairness and equity evaluation for ML systems
Nice to have
Healthcare, clinical, or regulated-industry experience
Experience
6+ years
Benefits
Health insurance, Dental and vision
Posted
April 29, 2026
Company
Geisinger

Who it can hire

We read the post for each of the 21 countries we cover.

  • Probably open to people in the US · Only some regions

    The post says: “Work from home (Pennsylvania).”

  • Not open to people in the other 20 countries we cover, as far as the post says.

The job post

Location:

Work from home (Pennsylvania)

Shift:

Days (United States of America)

Scheduled Weekly Hours:

40

Worker Type:

Regular

Exemption Status:

Yes

Job Summary:

The Tech Lead Data Scientist, AI Evaluation & Monitoring is the principal technical expert for how Geisinger evaluates, monitors, and optimizes AI systems in production. This is a hands-on technical leadership role. The Tech Lead sets the technical direction for AI evaluation across a large and growing portfolio, provides technical leadership to a team of data analysts who execute evaluation work, and partners directly with AI program teams to raise the quality of how AI is validated, monitored, and improved over time.

The role exists because AI at Geisinger has scaled past the point where oversight can be a document-review exercise. We need a technical leader who can guide program teams toward better-designed evaluations up front, instrument meaningful production monitoring, and continually advance the methods we use, from LLM-as-Judge frameworks to simulation-based testing to pragmatic experiment design that actually scales in healthcare.

Job Duties:

What You Will Own:

  • The technical evaluation methodology applied to AI programs across the enterprise, pre-production validation, production monitoring, and ongoing optimization
  • Hands-on guidance to program teams as they design validation studies, equity audits, monitoring plans, and escalation playbooks for their AI systems
  • Instrumentation of production monitoring: translating program-specific failure modes into concrete, measurable metrics
  • The evaluation toolkit: LLM-as-Judge frameworks, golden sets, simulation harnesses, experimental study designs, drift detection, subgroup fairness analysis
  • Reusable evaluation playbooks and templates that let each new program move faster than the last
  • Technical direction, design review, and mentorship for a team of data analysts supporting the evaluation function

What You Will Not Own:

  • People management, HR administration, or formal performance evaluations for the analyst team (those sit with the analysts' line manager; the Tech Lead provides technical input)
  • Program-level product strategy or go/no-go decisions
  • Final clinical validation judgment on whether a given AI is safe for a given clinical use
  • The software infrastructure behind the evaluation and monitoring tooling (built by the AI Platform team — the Tech Lead defines what's measured and how; Platform builds the backend)

Shape of the Work:

This is a role that lives at three altitudes at once:

With program teams (hands-on advisory). Partner with program owners early, before evaluations are designed, to shape study approach, sample size, stratification, gold-standard definition, and decision thresholds. Translate ambiguous failure modes into concrete, defensible evaluation designs. Coach teams through the technical work so that what arrives at governance review is rigorous, not performative.

With the evaluation toolkit (hands-on build). Design and operate the reusable assets that let evaluation scale: LLM-as-Judge rubrics and calibration methods, golden sets, simulation harnesses, A/B and shadow-mode study templates, subgroup fairness analyses, and drift monitors. Keep a pragmatic eye on what actually works in a clinical environment versus what works in a paper.

With the analyst team (technical leadership). Set technical direction, assign work across active evaluations, review analysis code and study designs, and raise the technical bar. Mentor analysts on methodology, statistical rigor, and the domain knowledge that makes evaluation credible. Grow them from execution into independent evaluation design.

Methods You'll Use:

  • Experimental and quasi-experimental design for production AI systems
  • LLM and generative AI evaluation: golden sets, judge-based evaluation, hallucination and grounding checks
  • Fairness and equity evaluation across patient and stakeholder subgroups
  • Production monitoring design: drift detection, performance decay, adoption, and outcome metrics
  • Causal inference methods appropriate to healthcare settings where full RCTs are often impractical
  • Simulation and adversarial testing for pre-production stress testing
  • Python, SQL, modern ML and evaluation tooling, cloud-native data platforms

Work is typically performed in an office or remote environment. Accountable for satisfying all job specific obligations and complying with all organization policies and procedures. The specific statements in this profile are not intended to be all-inclusive. They represent typical elements considered necessary to successfully perform the job.

*Relevant experience may be a combination of related work experience and degree obtained (Master's Degree = 2 years; PHD = 4 years ).

Position Details:

Required Skills & Qualifications:

  • 6+ years in data science, statistics, ML engineering, or applied quantitative research, with demonstrated experience as the senior technical voice on cross-functional projects

  • Strong foundation in experimental design and causal inference — and judgment about which method fits which situation

  • Hands-on experience designing and running model evaluation studies in real production settings

  • Experience evaluating LLM or generative AI systems, or comparable experience evaluating complex ML systems where ground truth is messy

  • Proven ability to translate ambiguous failure modes into concrete, defensible evaluation designs and monitoring metrics

  • Strong fluency in Python and SQL; working comfort with modern ML tooling and cloud-native data environments

  • Experience with fairness and equity evaluation for ML systems

  • Track record of providing technical leadership and mentorship without formal people-management authority

  • Clear written communication — the role produces evaluation memos and specifications that non-technical decision-makers rely on

  • Healthcare, clinical, or regulated-industry experience strongly preferred

  • MS or PhD in a quantitative field preferred; equivalent experience accepted

Education:

Bachelor's Degree-Related Field of Study (Required)

Experience:

Minimum of 6 years-Relevant experience* (Required)

Certification(s) and License(s):

Skills:

Analyzing, processing and building AI/ML solutions from Clinical and Operational data sources, such as Epic Clarity, HL7, DICOM, or ECG data, Clinical Databases, Communication, Critical Thinking, Data Analysis, Data Presentations, Group Collaboration, Leadership, Machine Learning Methods, Programming Languages, Structured Query Language (SQL)

OUR PURPOSE & VALUES: Everything we do is about caring for our patients, our members, our students, our Geisinger family and our communities.

  • KINDNESS: We strive to treat everyone as we would hope to be treated ourselves.
  • EXCELLENCE: We treasure colleagues who humbly strive for excellence.
  • LEARNING: We share our knowledge with the best and brightest to better prepare the caregivers for tomorrow.
  • INNOVATION: We constantly seek new and better ways to care for our patients, our members, our community, and the nation.
  • SAFETY: We provide a safe environment for our patients and members and the Geisinger family.

We offer healthcare benefits for full time and part time positions from day one, including vision, dental and domestic partners. Perhaps just as important, we encourage an atmosphere of collaboration, cooperation and collegiality.

We know that a diverse workforce with unique experiences and backgrounds makes our team stronger. Our patients, members and community come from a wide variety of backgrounds, and it takes a diverse workforce to make better health easier for all. We are proud to be an affirmative action, equal opportunity employer and all qualified applicants will receive consideration for employment regardless to race, color, religion, sex, sexual orientation, gender identity, national origin, disability or status as a protected veteran.

First Monday isn’t the employer or a recruiter and isn’t affiliated with Geisinger. Details may have changed since; confirm with the employer before you apply.

Probably open to people in the United StatesApply for this job Write my CV for this job Early access
Pay
Not stated
Required
Python, SQL, Experimental design, Causal inference, fairness and equity evaluation for ML systems
Nice to have
Healthcare, clinical, or regulated-industry experience
Experience
6+ years
Benefits
Health insurance, Dental and vision
Posted
April 29, 2026
Company
Geisinger

Similar jobs open to people in the United States

FAI Evaluation LeadFintentional Ai · Remote · USStrategic Project Lead – AI Post-TrainingAppen · Remote · USSQL Developer & Analyst Team Lead (Remote Opportunity)VetsEZ · Charleston, SC, United StatesTech Lead Manager (Machine Learning)Cedar · USA
First Monday

Get jobs like this every Monday.

One email a week with new remote jobs open to people in Canada. Free, and one click to unsubscribe. Privacy Policy

Or request early access to the app
Apply for this job