First Monday
Jobs
Jobs open to youOnly jobs that can hire where you liveA CV for each jobWritten from your record, every line checkedMemoryIt remembers, so you don’t repeat yourselfInterview helpResearch, prep and hints only you seeWatch how it works0:28
FAQDocsUse in AI, Free
Your countryNorth AmericaEuropeAsia-PacificLatin AmericaNot on the list? Tell us where you liveRequest early accessEarly access
First Monday

Remote jobs that can actually hire you.

One email to confirm your address, and another when your group is invited. Free during the beta. By requesting access you agree to the Terms & Conditions and the Privacy Policy.

Beta formacOSWindows

Jobs

Jobs open to you in CanadaRemote jobs in the USRemote jobs in PolandRemote jobs in the UKRemote jobs in CanadaRemote jobs in Mexico

Product

Jobs open to youA CV for each jobMemoryInterview helpHow it works

Company

FAQContact

Legal

Privacy PolicyTerms & Conditions
Jobs
Jobs open to you in CanadaRemote jobs in the USRemote jobs in PolandRemote jobs in the UKRemote jobs in CanadaRemote jobs in Mexico
Product
Jobs open to youA CV for each jobMemoryInterview helpHow it works
Company
FAQContact
Legal
Privacy PolicyTerms & Conditions
Remote jobs by country
United StatesPolandUnited KingdomCanadaMexicoBrazilGermanySpainPortugalIrelandNetherlandsIndiaSwedenBelgiumAustriaDenmarkFinlandSwitzerlandNorwayAustraliaNew Zealand

Remote jobs by country

United StatesPolandUnited KingdomCanadaMexicoBrazilGermanySpainPortugalIrelandNetherlandsIndiaSwedenBelgiumAustriaDenmarkFinlandSwitzerlandNorwayAustraliaNew Zealand
© 2026 First MondayNo tracking cookies. Anonymous page statistics only.
First Monday
Jobs/Data and AI/AI Quality & Evaluation Lead

AI Quality & Evaluation Lead

Whisk

PolandData and AIPosted October 5, 2026
Open to people in PolandApply for this job Write my CV for this job Early access
Pay
Not stated
Required
Notebooks, Spreadsheets
Posted
October 5, 2026
Company
Whisk

Who it can hire

We read the post for each of the 21 countries we cover.

  • Open to people in Poland

    The employer lists this job as remote there: “Poland”.

  • Not open to people in the other 20 countries we cover, as far as the post says.

The job post

About Samsung Food (Whisk)

At Samsung Food (you might know us as Whisk), we are pioneering the future of everyday care and health — helping individuals and families around the world live better through connected, proactive, and personalized experiences. Our mission is to connect food, health, and home across the Samsung ecosystem, turning data from your devices into personalized guidance that supports you and your loved ones.

We work at the intersection of AI, nutrition, behavior change, and digital health. The Samsung Food app was included in Google Play’s Best of 2020 Everyday Essentials list, has been regularly featured on the Apple App Store, and was nominated for a 2021 Webby Award. We’ve also launched Vision AI, which recognises food and meals using our global Food Genome, and developed deep integrations with Samsung Health and SmartThings.

We are a remote-first team of more than 100 people in over 30 countries, united by our commitment to innovation and impact. As part of Samsung Electronics, we combine the agility of a startup with the scale of the world’s largest consumer electronics company. Join us and help build technology that improves daily life for tens of millions of people across phones, wearables, TVs, and kitchen appliances in the years to come.

Learn more about how we’ve shaped a high-performing global team over the past 14 years.

Summary

Overview

We're building an AI coaching experience inside Samsung Health. Every piece of coaching a user sees is generated, not written. Our platform partner owns the evaluation runtime and functional evals (did the pipeline execute, did the tool call succeed). Nobody owns the layer above that: is the output actually good — right framing, right tone, right structure, does it deliver the coaching logic, does it feel like it's talking to me.

This engagement exists to build that layer: author the failure taxonomy, codify it into a binary rubric, validate an LLM judge against your own human grades, and hand the whole thing to an internal owner. We have a v0.1 rubric and golden scenarios from our product lead, and a functional rubric owned by engineering. We don't have the experience rubric — that's what this engagement produces.

Deliverables by Phase (Overall project ~13 weeks)

1. Failure taxonomy and golden set (within 4 weeks from the contract start date)

At least 150 traces hand-graded with open-coded notes. At least 40 of them come from sparse-data synthetic profiles. The taxonomy has 5–10 categories, each with a frequency count, and the top three failures are shown with their share of all failures. Saturation is evidenced: the last 20 traces produced no new category. At least 50 traces are frozen as a holdout. Head of Product signs off the taxonomy.

2. Rubric v1 and calibration (within 4 weeks from phase 1 completion)

One binary pass/fail criterion per taxonomy category, each with a definition, a pass example and a fail example. Written annotation guidelines. At least 30 traces are double-coded with a second grader, and every disagreement is logged and resolved in writing. A guideline revision log records each change and its reason. Head of Product signs off the rubric.

3. Validated judge and weekly readout (within 4 weeks from phase 2 completion)

Judge prompts for every criterion. A validation report giving true positive rate and true negative rate per criterion on both the dev set and the holdout. A weekly readout on helpfulness, relevance and tone, run at least twice: once by the consultant, and once by the internal owner with the consultant shadowing. A documented revalidation routine, triggered by every model or prompt change and run quarterly regardless.

4. Playbook and handover test (within 1 week from phase 3 completion)

A written playbook covering the whole method: grading, updating the taxonomy, revising the rubric, revalidating the judge and running the readout. The handover test passes when the internal owner reruns the judge validation unaided, gets within ±5 points, and runs one weekly readout with no consultant involvement.

What This Role Is Not

- Not building eval infrastructure, harnesses, or dashboards (owned elsewhere)

- Not functional/module-level evals or accuracy metrics

- Not clinical safety validation

- Not high-volume annotation — grading a few hundred traces to build the taxonomy; volume work goes to vendors/the judge once the rubric exists

What We're Looking For

You've run this loop end-to-end at least once on conversational or generated-text output: error analysis on real traces → a failure taxonomy you built yourself → binary criteria and annotation guidelines → an LLM judge validated against your own labels.

Background: conversation design, AI product quality, model behaviour/policy, content design, human data operations, UX research with strong qualitative coding, or RLHF. Applied linguistics and product management backgrounds also fit.

You'll be strong in this role if you:

- Can describe a failure mode you personally discovered by reading output — one nobody had named before you

- Are comfortable being the arbiter and documenting what you overruled and why

- Understand the rubric is discovered through grading, not written upfront

- Can explain why a judge that agrees with you 94% of the time might still be useless

- Write annotation guidelines that don't need interpreting

- Are comfortable working in notebooks and spreadsheets (no production code required)

Not required: domain expertise in nutrition, weight management, or behaviour change — we have that in-house.

First Monday isn’t the employer or a recruiter and isn’t affiliated with Whisk. Details may have changed since; confirm with the employer before you apply.

Open to people in PolandApply for this job Write my CV for this job Early access
Pay
Not stated
Required
Notebooks, Spreadsheets
Posted
October 5, 2026
Company
Whisk

Similar jobs open to people in Poland

Senior ML EngineerSword · Remote · EuropePrincipal Applied AI Engineer – Entity AgentsZoomInfo Technologies · Anywhere in the WorldSenior AI EngineerWPP Enterprise Solutions · PolandEND-TO-END SALES OFFICERHuman Intelligence · Worldwide
First Monday

Get jobs like this every Monday.

One email a week with new remote jobs open to people in Canada. Free, and one click to unsubscribe. Privacy Policy

Or request early access to the app
Apply for this job