Skip to content

Research

ML Evaluation Engineer (m/f/d)

Exact and traceable is our promise. You build the evaluations that show whether we keep it, for every model and every use case.

All roles
Open · start from Q4 2027
Team
Research
Employment
Full-time
Location
Munich · hybrid
Start
from Q4 2027
Languages
German and English

What you will do

  • Build and maintain evaluation datasets from real use cases
  • Develop metrics for correctness, faithfulness to sources, permissions and cost
  • Build automatic evaluations into development, for every change
  • Compare models from different providers and derive recommendations
  • Analyse failure modes and fix them with engineering

What you bring

  • You think in measurements: you say openly what a number does not show, and you change your mind when the data says so
  • A degree in computer science, statistics, mathematics or a related field
  • Experience evaluating ML or LLM systems
  • Excellent Python and solid foundations in statistics
  • A critical eye for benchmarks and measurement error
  • You work with AI tools as a matter of course

Bonus points

  • Experience with LLM-as-a-judge, human annotation or retrieval metrics
  • Publications or a thesis on evaluation

Hiring process

How we hire for this role

Six steps. In steps 4 and 5 you show how you work: with AI, and in the practice of your role.

  1. Step 1Online form

    Application

    A link to LinkedIn or your CV and a few sentences on why demi. No upload, no long cover letter.

    What we look for

    • What you have built, sold or moved
    • Why this role, why now
  2. Step 2Remote · with a founderYour time: 45 min

    Personal fit interview

    A conversation with one of the founders. Both sides get to know each other: your background, our mission, how we work.

    What we look for

    • Curiosity and drive
    • Ownership
    • Does the way we work suit you?
  3. Step 3Remote or on-site

    Skills interview

    We go deep: your expertise, your experience and how you structure problems and justify decisions.

    What we look for

    • Depth in your field
    • Clear communication
    • Sound reasoning
  4. Step 4Task and live walkthroughFor every role

    AI challenge

    You automate a real piece of work with the current AI tools of your choice and show us how you went about it. In sales just as in engineering.

    • Automate

      You build a flow that does the work repeatably: with agents, scripts or workflows.

    • Latest tools

      You use current models and agent tools and know which one is good for what.

    • Linux and terminal

      You are at home in the shell: files, scripts, environments, automation.

    • Self-reliant

      You make progress on your own: research, try, check and ask targeted questions.

    • Verify

      You show how you check AI results before you trust them.

  5. Step 5Live and as a task

    Role challenge

    Research: Think like a scientist, deliver like an engineer.

    1. APaper discussion

      We discuss a recent paper from your field: what holds, what does not, what we would build from it.

    2. BEvaluation case

      You design how to measure an agent's answers: dataset, metrics, failure modes.

    3. CPrototype with AI

      You turn an idea into a small prototype with AI tools and check the result critically.

    Your task

    Is the answer really right?

    You design a grading scheme for twenty agent answers, implement it with AI and show where the automatic grading departs from human judgement.

    What we look for

    • Craft
    • Speed with quality
    • Judgement
  6. Step 6On-site · Munich

    On-site in Munich and offer

    You visit us at the Munich Technology Centre and meet the founders and your future team. If it fits for everyone, we make you an offer with an attractive compensation package.

    What we look for

    • Meet the team and the office
    • Answer open questions
    • Your offer

What we offer

  • Tools and tokens

    The best AI tools and the tokens to go with them. Trying a new model needs no approval.

  • Ownership from day one

    You join early and own products, customers and decisions.

  • Grow into leadership

    Strong performers grow into leadership roles as the company scales.

  • Straight with the founders

    Full transparency on strategy, go-to-market and product decisions.

  • Share in the success

    Attractive pay with the opportunity to share in demi's long-term success.

  • Office at the MTZ

    A modern office in the Munich Technology Centre near the Olympic Park, with top equipment.

Application

Apply.

ML Evaluation Engineer (m/f/d)

One link and a few sentences. No upload, no long cover letter.

For example your LinkedIn profile or a shared folder with your CV.

Three to five sentences are enough.

We use your details only for your application and delete them no later than six months after the process ends, unless you agree to the talent pool. Privacy policy

Your configurator

No use cases yet

Your configurator is still empty

Add a use case from any card or case page and see what it brings per year, with the calculation. Free, no sign-up, no commitment: try it out and change any number.

Browse use cases