Favur Evals – evals of our agent harness, explore and control replays

Name: Favur Evals – evals of our agent harness, explore and control replays
Availability: InStock
Author: favurdev

by favurdev·Jul 17, 2026·2 points·0 comments

Visit Project View on HN

AI Analysis

●●SolidBig BrainNiche Gem

Faithful end-to-end replays let you watch agents hand off work like a movie, not just see final scores.

Strengths

•Coordinator logic sits in Python, not an LLM, avoiding prompt-injection drift during long runs.
•Eight-dimensional scoring matrix captures nuance beyond simple pass/fail benchmarks.
•Interactive replays show tool calls, file writes, and review cycles with exact timestamps.

Weaknesses

•No public leaderboard or comparison against open baselines like SWE-agent or OpenDevin.
•Eval suite relies on synthetic SOWs; real-world repo performance remains unproven.

Post Description

Maker here. Favur is a multi-agent harness written in Python: a team of 14 specialized agents — planner, architect, coder, tester, reviewer, builder among them — coordinated by Favur itself, not by an LLM. It takes a written statement of work and produces completed code, with no hand-holding or constant guidance along the way. Favur Evals scores those runs across models, using the same standardized SOW every time.

The fastest way to get it: https://favur.dev/drive/top_run — an interactive replay of whatever run currently tops the board. Every run is captured end-to-end, so you can watch the agents plan, hand work to each other, write the tests before the code exists, go through review, and ship — at your own pace, exactly as it happened. (Faithful playback of the record, not a live run.) Every run on the board has its replay linked from its details page.

Scoring: each run gets a composite across eight engineering subjects — code quality, test quality, cost efficiency, velocity, tool discipline, effort efficiency, process discipline, deliverables — computed from the run's own artifacts (lint, complexity, its pytest results, tool telemetry). Click any score and it expands into its formula.

Favur is a multi-model harness — different models can take different seats in the same run — and what we keep finding is that no single model leads every part of the job. Currently the top composite is an all-Meta Muse Spark run, while the strongest test suites, the best value-per-dollar, and the cleanest zero-failure execution belong to three other configs. The per-seat behavior fingerprints (cache utilization, reasoning depth, tool cadence) are my favorite way to compare models.

Caveats: every model runs inside our scaffolding, so treat the scores as relative rankings inside this harness, and some configs only have a run or two so far. Favur itself is closed-source and invite-only — please sign up for email updates to get news and future access.

Happy to answer anything about the harness, the scoring, or the replays.

https://evals.favur.dev https://favur.dev