First Monday
Jobs
Jobs open to youOnly jobs that can hire where you liveA CV for each jobWritten from your record, every line checkedMemoryIt remembers, so you don’t repeat yourselfInterview helpResearch, prep and hints only you seeWatch how it works0:28
FAQDocsUse in AI, Free
Your countryNorth AmericaEuropeAsia-PacificLatin AmericaNot on the list? Tell us where you liveRequest early accessEarly access
First Monday

Remote jobs that can actually hire you.

Beta formacOSWindows

Jobs

Jobs open to you in CanadaRemote jobs in the USRemote jobs in PolandRemote jobs in the UKRemote jobs in CanadaRemote jobs in Mexico

Product

Jobs open to youA CV for each jobMemoryInterview helpHow it works

Company

FAQContact

Legal

Privacy PolicyTerms & Conditions
Jobs
Jobs open to you in CanadaRemote jobs in the USRemote jobs in PolandRemote jobs in the UKRemote jobs in CanadaRemote jobs in Mexico
Product
Jobs open to youA CV for each jobMemoryInterview helpHow it works
Company
FAQContact
Legal
Privacy PolicyTerms & Conditions
Remote jobs by country
United StatesPolandUnited KingdomCanadaMexicoBrazilGermanySpainPortugalIrelandNetherlandsIndiaSwedenBelgiumAustriaDenmarkFinlandSwitzerlandNorwayAustraliaNew Zealand

Remote jobs by country

United StatesPolandUnited KingdomCanadaMexicoBrazilGermanySpainPortugalIrelandNetherlandsIndiaSwedenBelgiumAustriaDenmarkFinlandSwitzerlandNorwayAustraliaNew Zealand
© 2026 First MondayNo tracking cookies. Anonymous page statistics only.
First Monday
Jobs/Data and AI/Senior Data Scientist AI Evaluation

Senior Data Scientist AI Evaluation

Alpaca

Remote · AmericasData and AIPosted October 7, 2026
Open to people in 4 countriesApply for this job Write my CV for this job Early access
Pay
Stated in the post
Required
Python, SQL
Experience
6–10 years
Equity
Stock options
Benefits
Health insurance, Home office stipend, Monthly Stipend: USD $150 per month…
Posted
October 7, 2026
Company
Alpaca

Who it can hire

We read the post for each of the 21 countries we cover.

  • Open to people in the US, Canada, Mexico and Brazil

    The employer lists this job as remote there: “Remote - Americas”.

  • Not open to people in the other 17 countries we cover, as far as the post says.

The job post

Who We Are:

Alpaca is a US-headquartered, global leader in agent-first brokerage infrastructure for stocks, ETFs, options, crypto, fixed income, 24/5 trading, and more.

Amongst our subsidiaries, Alpaca is a licensed financial services company, serving hundreds of financial institutions across 40 countries with our institutional-grade APIs. This includes broker-dealers, investment advisors, wealth managers, hedge funds, and crypto exchanges, totalling over 10 million brokerage accounts.

Our global team is a diverse group of experienced engineers, traders, and brokerage professionals who are working to achieve our mission of opening financial services to everyone on the planet. We're deeply committed to open-source contributions and fostering a vibrant community, continuously enhancing our award-winning, developer-friendly API and the robust infrastructure behind it.

Alpaca is proudly backed by $400 million in funding from top-tier global investors including Portage Ventures, Spark Capital, Tribe Capital, Social Leverage, Horizons Ventures, Opera Tech Ventures, SBI Group, Derayah Financial, Unbound, Peak XV, Elefund, and Y Combinator.

Our Team Members:

We're a dynamic team of 400+ globally distributed members who thrive working from our favorite places around the world, with teammates spanning the USA, Canada, Japan, Hungary, Nigeria, Brazil, the UK, and beyond!

We're searching for passionate individuals eager to contribute to Alpaca's rapid growth. If you align with our core values—Stay Curious, Have Empathy, and Be Accountable—and are ready to make a significant impact, we encourage you to apply.

Your Role: We're looking for a Senior Data Scientist, AI Evaluation to design how Alpaca measures whether our models and agents are actually right. You'll be a senior individual contributor who turns ambiguous quality questions into ground truth, scoring methods, and eval loops that the company can trust—and uses those results to make the systems better. You'll build on an established data foundation, so the focus is raising quality and speeding up safe rollout. You'll own the quality bar, independent of the teams that build and optimize those systems.

This role is for someone who cares as much about whether an answer is correct as about whether a model can generate one. You'll partner with Product, Engineering, Analytics Engineering, and business stakeholders to define what "good" looks like, build the evaluations that test it, and close the loop so evals drive iteration. If you have a strong quantitative background, have shipped rigorous, measurable work (evaluation, experimentation, or model validation), and want ownership over a greenfield eval practice at a fast-growing brokerage-infrastructure company, this is the role.

What You'll Do

  • Design AI evaluations: Define ground truth, metrics, and scoring methods for models and agents.
  • Build repeatable eval loops: Track quality over time and catch regressions before release.
  • Partner on infrastructure: Work with engineering and analytics engineering to operationalize eval harnesses.
  • Drive iteration: Translate eval results into actionable recommendations for system improvements.
  • Establish quality standards: Set evaluation guidelines, documentation, and review practices.
  • Mentor and align: Foster evaluation best practices and build a culture of measurable AI quality across the team.

What We're Looking For

  • Track record of quantitative measurement rigor (e.g., LLM/model evaluation, metric validation, or experimentation).
  • Strong statistical and ML foundation—you treat evaluations as experiments (sample sizing, confidence intervals, significance, handling non-determinism) and validate automated graders against human ground truth.
  • Proficiency in Python and SQL, with experience evaluating models in production environments.
  • Strong judgment in defining quality metrics and ground truth for ambiguous outputs.
  • Excellent communication and cross-functional collaboration skills to align technical teams and leadership.
  • Strong problem-solving ability in fast-paced, greenfield environments.
  • 6–10 years in quantitative data science or ML, with focused experience in measurement or evaluation. A quantitative degree is a plus; equivalent industry experience is equally welcome.

Nice to Have

  • Hands-on LLM/agent evaluation in production, including eval harnesses, LLM-as-judge calibration, and CI regression gates.
  • Experience evaluating text-to-SQL, analytics agents, or other systems where correctness is verifiable against data.
  • Background in fintech, brokerage, or other domains where a wrong answer has real business or risk consequences.
  • Fluency with AI tools in research and engineering workflows.

How We Take Care of You:

  • Competitive Salary & Stock Options
  • Health Benefits
  • New Hire Home-Office Setup: One-time USD $500
  • Monthly Stipend: USD $150 per month via a Brex Card

Alpaca is proud to be an equal opportunity workplace dedicated to pursuing and hiring a diverse workforce.

Recruitment Privacy Policy

First Monday isn’t the employer or a recruiter and isn’t affiliated with Alpaca. Details may have changed since; confirm with the employer before you apply.

Open to people in 4 countriesApply for this job Write my CV for this job Early access
Pay
Stated in the post
Required
Python, SQL
Experience
6–10 years
Equity
Stock options
Benefits
Health insurance, Home office stipend, Monthly Stipend: USD $150 per month…
Posted
October 7, 2026
Company
Alpaca

Similar jobs open to people in the United States

Senior Applied AI Research ScientistSardine · North America · United States · CanadaSenior Data Scientist, Full StackSentiLink · United StatesResearch Engineer – Data InfrastructureElevenLabs · United Kingdom · United States · Poland…Staff Data Scientist, Full StackSentiLink · United States
First Monday

Get jobs like this every Monday.

Get jobs like this every Monday One email a week. Unsubscribe anytime.Or request early access to the app
Apply for this job