Atelier runs a crew of AI agents in parallel, each doing real work in its own git worktree. The genuinely hard part of building it wasn't the AI. It was everything that comes from running concurrent work against shared, stateful things, a counter, a git repository, a collection of results. Here are three bugs I hit, with the real code, because each one passes every test you'd think to write, right up until it doesn't.
1. The semaphore that miscounts when a waiter wakes
I cap how many agents run at once with a counting semaphore. The obvious implementation keeps an active count: increment on acquire, decrement on release, and when you release, wake a waiter. That obvious version has a race, and it's subtle enough that I left the explanation in the code:
async use<T>(job: () => Promise<T>): Promise<T> {
// A freshly-arriving caller takes a slot iff one is free RIGHT NOW (synchronously,
// before any await yields). Otherwise it parks. It must NOT increment after waking:
// the slot was handed to it on release. Incrementing on both acquire paths is the
// decrement/wake race: release does `active--` then resolves a waiter whose `active++`
// continuation is a queued microtask; any other pending use() prologue that runs in
// that window reads the stale, lowered `active` and slips past the gate.
if (this.active < this.size) this.active++;
else await new Promise<void>((r) => this.waiters.push(r));
try {
return await job();
} finally {
const next = this.waiters.shift();
if (next) next(); // hand THIS slot straight to the next waiter; no -- / ++
else this.active--; // no one waiting: actually free the slot
}
}
The fix is slot-transfer: on release, if someone is waiting, hand them your slot directly without decrementing and re-incrementing. The count never dips, so there's no window where a third caller reads a stale low number and slips through. A parked caller is, by construction, already accounted for. The buggy version would occasionally run size + 1 jobs, which, when each job is an agent shelling out to your machine, is not a rounding error.
2. Git refuses to be parallel
The whole point of the system is parallelism: many agents, many worktrees, at once. But git itself takes a repository-level lock. Two concurrent git worktree adds, or two merges, contend on the index and lock files and one of them fails. So the operations that touch the repo's shared state are funneled through a mutex, which is just a semaphore of one:
// FIFO mutex: a Semaphore of one, named for intent. Every git worktree-add and merge
// is serialized through this: git's repo-level lock is the failure mode it prevents.
export class Mutex extends Semaphore {
constructor() { super(1); }
}
The rule that came out of this: parallelize the agents, serialize the git. The expensive, slow work (an agent thinking and editing) runs concurrently. The fast, contended work (touching the repo) runs one at a time. Getting that split wrong, trying to parallelize the git, gives you intermittent failures that look like git being flaky when it's actually you.
3. The try/catch that catches nothing
My scheduler runs tasks honoring their dependencies, with a concurrency cap. When a task fails, I don't want the whole run to die, so instead of throwing, it records the error in a results map and skips that task's dependents:
runOne(task).then(
(value) => { results.set(id, value); active--; pump(); },
(err) => { results.set(id, err instanceof Error ? err : new Error(String(err))); failed.add(id); active--; pump(); },
);
// ...the promise resolves with the Map; it never rejects on a task failure.
That's a reasonable design, and it sets a trap for the caller. The natural way to call this is try { const results = await runScheduled(...) } catch (e) { ... }, and that catch is dead code. A task failure never throws, it's a value sitting in the map. So a real failure, even a hard budget-abort, sails straight past the catch and the run looks like it succeeded. The only way to know something failed is to scan the returned map for Error instances. The bug isn't in the scheduler; it's in every caller who assumed failure means an exception.
None of these are AI bugs
That's the thing I'd want someone building agent infrastructure to take away. "AI agents" is the headline, but the moment you run several of them at once against shared resources, the bugs you fight are decades-old concurrency bugs in new clothes. A semaphore that double-counts across a microtask boundary. A resource that doesn't parallelize. An error channel your reflexive try/catch doesn't watch. The model was, honestly, the predictable part. Knowing to go looking for these was worth more than any amount of prompt engineering.