Machine Learning Engineer - Evals

About this role

You'll own evaluation of Honcho end to end, the instrument that tells a research-driven team whether their identity representations are actually getting better. Plastic Labs builds the memory and identity layer for the agentic world, and the product works.

It's a complex multi-agent harness with real black-box behavior, so the job is defining what "better" means for representations that change over time, without ground truth, and building the machinery that measures it.

If you love constructing measurements from nothing, reading traces to find what's actually broken, and shipping the fix yourself, you'll feel at home here.

What You'll Do

Design the evals. Define the scores and decide what "better" means for an entity representation that changes over time, and keep that definition current as the product and methods move.

Build the pipelines and harnesses. Data in, labels, versions, reruns, judges: the machinery that lets the team ask a new question this week and get an answer this week.

Run them and harvest insights. Read the traces and results, find what's actually broken, not what's easy to measure, and propose fixes that produce higher-fidelity representations.

Build simulation agents at scale. The ceiling on iteration speed is how many entities can be modeled and measured at once. You raise it.

Own the loop end to end. The question, the pipeline, the rerun, the writeup. Nothing gets scoped and handed off, and nothing waits on someone else's sprint.

Work experience

• Must have: Designed,​ built,​ and ran evals end to end (LLMS)

Education

• Degree in computer science,​ mathematics,​ statistics,​ or a related quantitative field

Skills

• Must have: Production-​quality Python (tests,​ maintenance,​ release cadence)
• 3+ years of experience 3+ years designing and owning LLM evaluation
• PyTorch,​ Hugging Face,​ Weights &​ Biases.​ Data pipeline tooling (Hydra,​ Kafka,​ SQL) a plus

Nice-to-have

• Work on memory,​ identity,​ personalization,​ agent evaluation,​ or agentic optimization.​
• Shipped research:​ first or co-​author at NeurIPS,​ ICML,​ ICLR,​ or equivalent,​ OR self-​published open-​source work of comparable quality

Join us to be at the forefront of AI innovation! Bring your passion for machine learning/AI-based analysis to develop impactful models that drive strategic decisions across industries while working with the latest technologies in a collaborative environment that values your expertise!

Pay: $220,000.00 - $300,000.00 per year

Application Question(s):

• Are you able to work in-person at our NYC office full-time?
• Describe your experience designing and building a production ML evaluation system from scratch — particularly when you had no ground truth or existing benchmark to rely on.
• What's your trackable research experience? Share a first-author paper or significant open-source project and explain your specific contribution.
• Can the candidate be on-site? If not, is the candidate willing to relocate?
• What is their salary expectation?

Education:

• Bachelor's (Required)

Experience:

• designing and owning LLM evaluation: 3 years (Required)

Location:

• New York, NY (Required)

Work Location: In person