First Monday
Jobs
Jobs open to youOnly jobs that can hire where you liveA CV for each jobWritten from your record, every line checkedMemoryIt remembers, so you don’t repeat yourselfInterview helpResearch, prep and hints only you seeWatch how it works0:28
FAQDocsUse in AI, Free
Your countryNorth AmericaEuropeAsia-PacificLatin AmericaNot on the list? Tell us where you liveRequest early accessEarly access
First Monday

Remote jobs that can actually hire you.

Beta formacOSWindows

Jobs

Jobs open to you in CanadaRemote jobs in the USRemote jobs in PolandRemote jobs in the UKRemote jobs in CanadaRemote jobs in Mexico

Product

Jobs open to youA CV for each jobMemoryInterview helpHow it works

Company

FAQContact

Legal

Privacy PolicyTerms & Conditions
Jobs
Jobs open to you in CanadaRemote jobs in the USRemote jobs in PolandRemote jobs in the UKRemote jobs in CanadaRemote jobs in Mexico
Product
Jobs open to youA CV for each jobMemoryInterview helpHow it works
Company
FAQContact
Legal
Privacy PolicyTerms & Conditions
Remote jobs by country
United StatesPolandUnited KingdomCanadaMexicoBrazilGermanySpainPortugalIrelandNetherlandsIndiaSwedenBelgiumAustriaDenmarkFinlandSwitzerlandNorwayAustraliaNew Zealand

Remote jobs by country

United StatesPolandUnited KingdomCanadaMexicoBrazilGermanySpainPortugalIrelandNetherlandsIndiaSwedenBelgiumAustriaDenmarkFinlandSwitzerlandNorwayAustraliaNew Zealand
© 2026 First MondayNo tracking cookies. Anonymous page statistics only.
First Monday
Jobs/Engineering/Senior AI Quality & Evaluation Engineer

Senior AI Quality & Evaluation Engineer

Solvd

USAEngineeringPosted November 13, 2026
Open to people in the United StatesApply for this job Write my CV for this job Early access
Pay
Not stated
Required
Python, Bedrock, OpenSearch, CloudWatch
Nice to have
Contact Center, Knowledge management, AWS Certified AI Practitioner, Machine Learning Engineer Associate
Reports to
POD Lead
Experience
3+ years
Posted
November 13, 2026
Company
Solvd

Who it can hire

We read the post for each of the 21 countries we cover.

  • Open to people in the US

    The employer lists this job as remote there: “USA”.

  • Not open to people in the other 20 countries we cover, as far as the post says.

The job post

Solvd Inc. is a rapidly growing AI-native consulting and technology services firm delivering enterprise transformation across cloud, data, software engineering, and artificial intelligence. We work with industry-leading organizations to design, build, and operationalize technology solutions that drive measurable business outcomes.

Following the acquisition of Tooploox, a premier AI and product development company, Solvd now offers true end-to-end delivery—from strategic advisory and solution design to custom AI development and enterprise-scale implementation. Our capability centers combine deep technical expertise, proven delivery methodologies, and sector-specific knowledge to address complex business challenges quickly and effectively.

We are looking for a Senior AI Quality & Evaluation Engineer to own the independent proof that retrieval works and that permissions hold. You’ll build and run the golden question suite, negative entitlement tests, and drift alarms that serve as acceptance gates — so the engineers who built the system are not grading their own work.

This is a 50% allocation role across weeks 3–10, drawn from a Quality Engineering or GRC background, reporting to the POD Lead and working day-to-day with the retrieval engineer.

What you’ll do

  • Build the golden question suite from top contact drivers and needs workshops — question, expected source, expected passage, and acceptance thresholds for relevance, grounding, and citation accuracy, versioned alongside the content it tests.

  • Build the negative entitlement test set — queries issued under unentitled identities that must return no restricted passage, with any failure raised as a severity-one alarm.

  • Build freshness and retirement tests — source-modified-to-index-available lag per item against the agreed threshold, and verification that retired items disappear on the next sync.

  • Automate the harness to run on schedule and after every content, configuration, or model change — publishing scores to the quality dashboard with trend lines and regression alarms.

  • Define the QA gate content analysts apply to remediated content.

  • Run the suite before every pilot change in weeks 7–9 and report misses with root cause — content, chunking, ranking, or permissions — to the right owner.

  • Write the evaluation runbook and train client stewards and platform owners to add questions, re-run the suite, and interpret regressions.

Required skills & experience

  • 3+ years in software quality engineering or test automation with Python, including building test harnesses that run in CI on a schedule.

  • Hands-on evaluation of a search or RAG system in production — writing ground truth sets, scoring relevance and grounding, and tracking drift over time.

  • Experience designing security-focused negative tests — authorization, entitlement, data exposure — and reporting them at the severity a client security team expects.

  • Ability to work in AWS — invoking Bedrock and OpenSearch APIs, reading CloudWatch, deploying a scheduled job with infrastructure code.

  • Clear written reporting; the suite’s output is read by client stewards, not only engineers.

Preferred qualifications

  • AWS Certified AI Practitioner or Machine Learning Engineer Associate.

  • Experience with RAG evaluation libraries and LLM-as-judge scoring with human calibration.

  • Contact center or knowledge management domain exposure.

  • Familiarity with SOC 2 or ISO 27001 evidence practices.

When you join Solvd, you'll…

  • Shape real-world AI-driven projects across key industries, working with clients from startup innovation to enterprise transformation.

  • Be part of a global team with equal opportunities for collaboration across continents and cultures.

  • Thrive in an inclusive environment that prioritizes continuous learning, innovation, and ethical AI standards.

Ready to make an impact?

If you're excited to build things that matter, champion responsible AI, and grow with some of the industry’s sharpest minds. Apply today and let’s innovate together.

Solvd is an equal opportunity employer.

First Monday isn’t the employer or a recruiter and isn’t affiliated with Solvd. Details may have changed since; confirm with the employer before you apply.

Open to people in the United StatesApply for this job Write my CV for this job Early access
Pay
Not stated
Required
Python, Bedrock, OpenSearch, CloudWatch
Nice to have
Contact Center, Knowledge management, AWS Certified AI Practitioner, Machine Learning Engineer Associate
Reports to
POD Lead
Experience
3+ years
Posted
November 13, 2026
Company
Solvd

Similar jobs open to people in the United States

Senior Full-Stack Product Engineer, AI ProductsSolv Health · Remote · USSenior Forward Deployed Engineer (AI Enablement)Reddit · Remote · USSenior Software Engineer – AI EnablementWorkiva · Remote · USSenior Solutions Architect, Agentic AINVIDIA · Remote · US
First Monday

Get jobs like this every Monday.

One email a week with new remote jobs open to people in Canada. Free, and one click to unsubscribe. Privacy Policy

Or request early access to the app
Apply for this job