First Monday
Jobs
Jobs open to youOnly jobs that can hire where you liveA CV for each jobWritten from your record, every line checkedMemoryIt remembers, so you don’t repeat yourselfInterview helpResearch, prep and hints only you seeWatch how it works0:28
FAQDocsUse in AI, Free
Your countryNorth AmericaEuropeAsia-PacificLatin AmericaNot on the list? Tell us where you liveRequest early accessEarly access
First Monday

Remote jobs that can actually hire you.

Beta formacOSWindows

Jobs

Jobs open to you in CanadaRemote jobs in the USRemote jobs in PolandRemote jobs in the UKRemote jobs in CanadaRemote jobs in Mexico

Product

Jobs open to youA CV for each jobMemoryInterview helpHow it works

Company

FAQContact

Legal

Privacy PolicyTerms & Conditions
Jobs
Jobs open to you in CanadaRemote jobs in the USRemote jobs in PolandRemote jobs in the UKRemote jobs in CanadaRemote jobs in Mexico
Product
Jobs open to youA CV for each jobMemoryInterview helpHow it works
Company
FAQContact
Legal
Privacy PolicyTerms & Conditions
Remote jobs by country
United StatesPolandUnited KingdomCanadaMexicoBrazilGermanySpainPortugalIrelandNetherlandsIndiaSwedenBelgiumAustriaDenmarkFinlandSwitzerlandNorwayAustraliaNew Zealand

Remote jobs by country

United StatesPolandUnited KingdomCanadaMexicoBrazilGermanySpainPortugalIrelandNetherlandsIndiaSwedenBelgiumAustriaDenmarkFinlandSwitzerlandNorwayAustraliaNew Zealand
© 2026 First MondayNo tracking cookies. Anonymous page statistics only.
First Monday
Jobs/Engineering/Test Automation Engineer

Test Automation Engineer

Billennium

PolandEngineeringPosted November 16, 2026Recruitment agency
Open to people in PolandApply for this job Write my CV for this job Early access
Pay
Not stated
Required
Python, TypeScript, pytest, httpx, Playwright, GitLab CI, Docker, Playwright
Nice to have
API Gateway, DynamoDB, S3, CloudWatch, Bedrock AgentCore
Experience
5+ years
Benefits
Udemy for Business, Health insurance, Multisport card, veterinary package
Posted
November 16, 2026
Company
Billennium

Who it can hire

We read the post for each of the 21 countries we cover.

  • Open to people in Poland

    The employer lists this job as remote there: “Poland”.

  • Not open to people in the other 20 countries we cover, as far as the post says.

The job post

Key Responsibilities:

  • Test Strategy: Define the test strategy and test pyramid with developers and platform engineers, and keep it up to date as the platform grows.

  • Test Repository Ownership: Set up and own the automation repository: structure, conventions, shared fixtures, code review and reporting. Keep it simple enough that developers can add tests themselves.

  • API Test Automation: Build functional, schema, negative and regression suites for the REST API. Cover long-running asynchronous flows: agent deployments, image builds, provisioning of dev environments and WebSocket log streaming.

  • UI Test Automation: Build a stable Playwright E2E suite for the critical user journeys: workspaces, agent deployment, marketplace, evaluation, monitoring and admin.

  • Agent Integration & Smoke Testing: Deploy the supported agent templates through the platform. Check that endpoints respond, auth is enforced, logs and traces arrive, MCP tool calls succeed and memory persists.

  • CI/CD & Test Environments: Run the suites in CI as merge-request gates, nightly regression runs and post-deploy smoke tests. Automate the setup and cleanup of short-lived test workspaces and test data in AWS.

  • Collaboration and Troubleshooting: Work closely with developers, AI engineers, and platform teams to identify automation opportunities and improve overall software quality. Conduct root cause analysis for identified issues and collaborate with development teams for resolution.

Qualifications:

  • 5+ years in software quality engineering. You have defined a test strategy for a web platform or SaaS product. You have also set up and owned a test automation repository (not only contributed tests to one), including its structure, conventions, CI integration, code reviews and long-term maintenance, in Python or TypeScript.

  • Strong hands-on experience automating REST API tests (e.g., pytest + httpx, Playwright API testing), including: schema validation against OpenAPI (contract testing is a plus), asynchronous flows (polling, state machines, queues, WebSockets), auth and access control (OAuth2/OIDC, JWT, role- and tenant-based access, negative cases).

  • Proven experience with Playwright on React/SPA applications. You have kept a large suite stable and kept flaky tests measured and under control.

  • Experience building quality gates in GitLab CI (or a similar tool) with Docker. Experience testing cloud-hosted applications and debugging through logs, metrics and traces.

  • You understand how LLM-based agents behave (tools, memory, non-determinism). You use AI assistants (e.g., Claude Code, GitHub Copilot) every day in your own work: writing tests, analysing failures and documenting. You review AI-generated code critically.

  • Strong collaboration and reporting skills; ability to work in cross-functional and international teams. Strong analytical and problem-solving skills with attention to detail.

Nice to have:

  • AWS experience (Lambda, API Gateway, DynamoDB, S3, CloudWatch, Bedrock AgentCore).

  • Experience with Strands or other agent frameworks (e.g., LangGraph, CrewAI).

  • Familiarity with LLM evaluation tools (e.g., DeepEval, Promptfoo, Ragas).

  • Experience with MCP / A2A servers and protocols.

  • Knowledge of vector-based retrieval systems.

  • Experience in monitoring & observability.

  • Experience with full-stack application testing.

  • Security testing: working knowledge of the OWASP Top 10 (web and API) and hands-on penetration testing experience (e.g., Burp Suite, OWASP ZAP)

  • Knowledge of GxP or regulated environments is a plus.

  • Strong analytical mindset and attention to detail.

What we offer:

  • Comprehensive benefits - enjoy Udemy for Business, private medical care, Multisport card, veterinary package, language lessons, and shopping vouchers.

  • Career growth - access opportunities for professional development and learning, including perks related to our official partnerships with global IT giants: Microsoft, AWS, Snowflake, Salesforce & more.

  • Global collaboration - work with a diverse, international team.

  • Innovative environment - be part of a forward-thinking and growth-oriented workplace.

  • Engaging community – Work with passionate professionals and participate in team-building events, hackathons, and CSR initiatives to make an impact beyond work.

  • Team-building events including our company tradition (annual company event in Mazury).

  • A pleasant surprise to start your journey with us in the form of a welcome pack.

First Monday isn’t the employer or a recruiter and isn’t affiliated with Billennium. Details may have changed since; confirm with the employer before you apply.

Open to people in PolandApply for this job Write my CV for this job Early access
Pay
Not stated
Required
Python, TypeScript, pytest, httpx, Playwright, GitLab CI, Docker, Playwright
Nice to have
API Gateway, DynamoDB, S3, CloudWatch, Bedrock AgentCore
Experience
5+ years
Benefits
Udemy for Business, Health insurance, Multisport card, veterinary package
Posted
November 16, 2026
Company
Billennium

Similar jobs open to people in Poland

Automation QA Engineer (AQA)IBA Poland · PolandAutomation QA EngineerUnisystech Consulting · PolandBridgewise Senior AQA Engineer – UI AutomationCommIT · PolandSenior QA Engineer (AI & Automation)BRAVER IT / ITeamly · Poland
First Monday

Get jobs like this every Monday.

One email a week with new remote jobs open to people in Canada. Free, and one click to unsubscribe. Privacy Policy

Or request early access to the app
Apply for this job