First Monday
Jobs
Jobs open to youOnly jobs that can hire where you liveA CV for each jobWritten from your record, every line checkedMemoryIt remembers, so you don’t repeat yourselfInterview helpResearch, prep and hints only you seeWatch how it works0:28
FAQDocsUse in AI, Free
Your countryNorth AmericaEuropeAsia-PacificLatin AmericaNot on the list? Tell us where you liveRequest early accessEarly access
First Monday

Remote jobs that can actually hire you.

Beta formacOSWindows

Jobs

Jobs open to you in CanadaRemote jobs in the USRemote jobs in PolandRemote jobs in the UKRemote jobs in CanadaRemote jobs in Mexico

Product

Jobs open to youA CV for each jobMemoryInterview helpHow it works

Company

FAQContact

Legal

Privacy PolicyTerms & Conditions
Jobs
Jobs open to you in CanadaRemote jobs in the USRemote jobs in PolandRemote jobs in the UKRemote jobs in CanadaRemote jobs in Mexico
Product
Jobs open to youA CV for each jobMemoryInterview helpHow it works
Company
FAQContact
Legal
Privacy PolicyTerms & Conditions
Remote jobs by country
United StatesPolandUnited KingdomCanadaMexicoBrazilGermanySpainPortugalIrelandNetherlandsIndiaSwedenBelgiumAustriaDenmarkFinlandSwitzerlandNorwayAustraliaNew Zealand

Remote jobs by country

United StatesPolandUnited KingdomCanadaMexicoBrazilGermanySpainPortugalIrelandNetherlandsIndiaSwedenBelgiumAustriaDenmarkFinlandSwitzerlandNorwayAustraliaNew Zealand
© 2026 First MondayNo tracking cookies. Anonymous page statistics only.
First Monday
Jobs/Engineering/QA Engineer (AI + backend)

QA Engineer (AI + backend)

apreel

PolandEngineeringPosted October 16, 2026Recruitment agency
Open to people in PolandApply for this job Write my CV for this job Early access
Pay
Up to 160 PLN an hour
Required
Python, pytest, Pact, Postman, Swagger/OpenAPI, Git, TestRail, LLM evaluation, Prompt engineering, Security-aware API testing
Nice to have
LangGraph, Langfuse, LangSmith, Phoenix, Appium
Experience
3+ years
Starts
Asap
Posted
October 16, 2026
Company
apreel

Who it can hire

We read the post for each of the 21 countries we cover.

  • Open to people in Poland

    The employer lists this job as remote there: “Poland”.

  • Not open to people in the other 20 countries we cover, as far as the post says.

The job post

Engagement: We are partnering with a US-based health-tech company on the takeover of a production AI-powered mobile coaching platform. Real users, real production, real releases — not a greenfield project.

Role summary: Own quality on the two hardest surfaces of a production AI coaching platform:

the LLM agents that answer differently every run, and the API layer everything else depends on.

As AI & Backend QA Engineer you own the quality of what the model says and what the APIs return. On the AI side that means building the evaluation practice: golden sets, quality metrics, and regression detection that catch a degraded answer before a user does. On the backend side it means API test suites and consumer-driven contract testing with Pact — agreed with the client's US engineering team as a standing requirement for this role, not an optional technique.

The defining skill is localization: when an answer is wrong, you can tell whether the problem sits in the model, in the service code, or in the input data — and you bring the evidence, not a guess.

What you'll be testing:

  • AI core — a Python service with LangGraph agents producing streaming, non-deterministic coaching responses; this

  • role's primary surface

  • Backend APIs — a NestJS REST gateway behind Auth0, plus the services behind it

  • Mobile app — React Native, shipping to iOS and Android app stores; AI features surface directly in the app that

  • members use

  • Data layer — MongoDB, Redis, and Postgres behind the services

  • Process — Agile/Scrum, Jira for work tracking and defects, TestRail for test management

  • You join the Polish QA function alongside our Manual QA Engineer and QA Automation Engineer, and work in a shared workflow with the client's US-based engineering and QA teams.

What you'll do:

  • Build and run LLM evaluation. Curate datasets and golden sets, define response quality metrics, and catch quality regressions between prompt and model versions before they reach production

  • Define acceptance criteria for non-deterministic output. Assertions on properties and contracts — structure, constraints, invariants — rather than exact wording

  • Treat prompts as tested artifacts. Change a prompt, measure the effect of that change; probe systematically for hallucinations and prompt injection

  • Own API testing. REST endpoints, JSON payloads, auth token flows, error handling — driven through Postman and Swagger/OpenAPI specs

  • Implement contract testing with Pact. Consumer-driven contracts between the mobile app, the gateway, and the backend services, verified on every change

  • Test with security in mind. Authorisation boundaries, token misuse, malformed input — the API-level abuse cases nobody writes a happy-path test for

  • Localise failures across the stack. Logs, traces, and database queries to pin a defect to the model, the service code, or the input data

  • Help wire evaluations into CI/CD. Evaluation runs in the pipeline and quality gates on pull requests, so a bad prompt change cannot merge quietly

  • Keep the record straight. Results mapped to test cases in TestRail, defects logged and driven to closure in Jira, findings written up so the team can act on them

Must-have skills:

  • 3+ years in technical QA or test automation, including API testing on production systems

  • Python — writing and maintaining test code, pytest, fixtures

  • Strong HTTP and API fundamentals — REST, JSON, status codes, headers, auth tokens

  • Contract testing with Pact — consumer-driven contracts in practice; this is a hard requirement for the role, agreed with the client's engineering team

  • Security-aware API testing — authorisation boundaries, token handling, malformed and hostile input

  • Comfort with non-determinism — defining acceptance criteria when there is no single correct output: assertions on properties and contracts rather than exact wording

  • LLM evaluation — datasets and golden sets, response quality metrics, catching quality regressions between prompt and model versions

  • Prompt engineering — the ability to change a prompt and measure the effect of that change; awareness of hallucinations and prompt injection

  • Backend savvy — HTTP and REST, JSON, tokens, logs, database queries; you can tell whether a problem sits in the model, in the service code, or in the input data

  • Postman and Swagger/OpenAPI as daily tools

  • Git in a team workflow — branches, pull requests, reviews

  • TestRail for test-case management and run tracking

  • Mobile product context (Android/iOS) — comfortable testing features that ship in a mobile app, including apprelease timing and store builds

  • Strong English — you work in a shared workflow with US colleagues

  • Must work hours that overlap US Mountain Time afternoons (~14:00–18:00 PL time)

Nice-to-haves:

  • Langfuse or equivalent (LangSmith, Phoenix) — working with traces and attaching evaluations to them

  • LangGraph — testing an agent as a state graph, both a single node and the full flow

  • CI/CD — running evaluations in a pipeline and quality gates on pull requests

  • Python with Appium — mobile automation exposure alongside the API work

  • Testing other AI or LLM-driven features in production

  • Experience taking over quality ownership of someone else's product

Offer:

  • Location: 100% remote

  • Engagement: B2B contract

  • Duration: Initial 3-month contract, expected to extend

  • Rate: upt to 160 PLN/h

  • Start: asap

First Monday isn’t the employer or a recruiter and isn’t affiliated with apreel. Details may have changed since; confirm with the employer before you apply.

Open to people in PolandApply for this job Write my CV for this job Early access
Pay
Up to 160 PLN an hour
Required
Python, pytest, Pact, Postman, Swagger/OpenAPI, Git, TestRail, LLM evaluation, Prompt engineering, Security-aware API testing
Nice to have
LangGraph, Langfuse, LangSmith, Phoenix, Appium
Experience
3+ years
Starts
Asap
Posted
October 16, 2026
Company
apreel

Similar jobs open to people in Poland

AI Engineer (AgentQ)Virtusa · PolandBack-end Engineer Integration and AI (PHP/Python)Nextcloud · Remote · WorldwideSenior Backend Engineer (Platform)Acclaim · GeorgiaSenior Backend Engineer (Platform)Acclaim · Georgia
First Monday

Get jobs like this every Monday.

One email a week with new remote jobs open to people in Canada. Free, and one click to unsubscribe. Privacy Policy

Or request early access to the app
Apply for this job