Multi-agent workflow with Qwen 27B models outperforms Fable 5 on coding benchmark

A Memdoor multi-agent workflow using five Qwen3.8-27B models achieved 9.8 out of 10 on the first 10 problems of LiveCodeBench Hard, outperforming Fable 5 running solo (8.0) and nearly matching the original researchers' harness (9.6). The system succeeded in 49 out of 50 runs across five attempts per problem, with each agent starting from a fresh context and communicating only through disk files. The workflow assigned five distinct roles—planner, two idea proposers, solver, and manager—and relied on shell script execution to verify solutions rather than model self-assessment.

Key points

• Achieved 9.8/10 average score on LiveCodeBench Hard first 10 problems (50 runs total, 49 successful)
• Five Qwen3.8-27B agents in specialized roles (planner, proposer A, proposer B, solver, manager) coordinated only via file system
• Cost $35.79 for 50 runs with 15.3M output tokens; median execution time 40 minutes, longest 4 hours
• Reproduced original GVS5H research prompts and iteration procedure, though evaluation limited to 10 problems vs. full 100-problem benchmark
• Manager occasionally generated false counterexamples that overwrote correct solutions; proposed fixes include brute-force validation and preserving last passing code

How to use it

• The team-solve workflow runs with memdoor workflow run team-solve --wait and returns exit code 0 when all samples pass
• Define your problem in task.md with test pairs (tests/N.in, tests/N.out); the workflow generates solution.py reading from standard input
• Each agent role is defined in a YAML file under steps/ with DAG dependencies via requires:; modify prompts to customize roles
• Works with OpenRouter, Anthropic, OpenAI, Gemini, and local runtimes (Ollama, llama.cpp, vLLM, LM Studio); model selection via /model setting

Has anyone tested this approach on the full 100-problem LiveCodeBench Hard set?