Research report · 2026-10-14

What we learned from giving an autonomous coding pipeline 63 tickets

We wanted to find out how much a ticket-to-PR pipeline could handle on its own while building a small project. Could it recover from failures, improve its workflow and tune itself as it went? We opened 63 GitHub issues with gh issue create and gave OpenPearl room to work. Fifty tickets described a new product named pearl-engine, a zero-dependency Node.js library that retries, heals, learns and tunes itself. The rest covered pipeline fixes, re-filed failures and a chaos drill. We learned a lot along the way, including where the pipeline surprised us and where it still appreciated a helping hand.

The short version

The encouraging part is that the engine did well in simulation, and the pipeline became dependable at recovery once we showed it what to do. All 50 engine deliverables reached main, although we had to re-file 4 of them. The pipeline also merged 8 fixes to its own gates and workflows. A person still had to notice and request each new kind of fix, while its only autonomous trigger handled red tests on main. The five challenge tickets helped us see that balance more clearly: the pipeline resolved simultaneous edits, made a reasonable guess on an ambiguous request, left a test-first requirement unverifiable, and returned honest diagnostics when a task could not succeed as written.

A quick look at where the product adapted on its own and where the pipeline still relied on guidance.
GoalPipeline (the factory)Engine (the product)
Self-healing17/17 merge conflicts resolved automatically. 22/34 incidents recovered with no human (65%); excluding conflicts 5/17 (29%). Red-main MTTR: 66 and 95 min by hand vs 106 s automatic.In simulation, success at 30% injected faults rises 71.5% → 99.2%; at 60%, 41.1% → 87.9%.
Self-improvingIt implements requested fixes (8 of 8 merged, merge-failure rate 4/24 → 0/31), but only the watchdog can currently open a repair ticket on its own.Thompson-sampling selection cuts regret 200 → 12.1 (94%) and picks the best arm 95% of the time.
Self-optimizingMean run time rose from 190 s (first 25) to 234 s (last 25). No pipeline setting tuned itself.Adaptive timeout: 493 → 27 false timeouts; tuner beats random search 31.0x at 160 evals; adaptive cache TTL gains only 1.8 pt hit rate.

The engine's scorecard() rated healing at 95, improving at 95.9, optimizing at 67.5 and the overall result at 87 / 100. We mapped the measurements to the scorecard inputs ourselves, so these numbers are best used as a friendly way to compare the three areas, not as external grades.

63issues created
59/63PRs merged
46/50experiment PRs merged first time (92%)
205 smean run (median 201, p95 316, max 543)
6,876lines added, 50 deleted, 181 files
105review rounds, 59 with findings (56%)
17/17conflicts auto-resolved
37/62PRs gated fail yet merged
3.7 hbetween first and last issue created
216 mintotal pipeline compute

We were happy to see 59 of 63 PRs land. We also noticed that 37 of 62 PRs merged while the test gate reported fail. The pipeline kept moving, but our main quality signal was no longer helping us tell a healthy PR from a troubled one. That became an especially useful lesson from the run.

How we set up the experiment

Here is the setup for anyone who would like to follow the path we took or try a similar run.

The pipeline kept moving steadily

Workflow wall-clock per issue. Mean 205 s; E49 (the conflicting-edit challenge) was slowest at 543 s.
Most tickets finished in three to four minutes. The longest run was the deliberately conflicting E49 ticket at 543 seconds.

Each bar follows a ticket from dispatch to the end of its workflow. The fastest run took 106 seconds, and the median took 201. The longest run also had a clear story behind it: E49 edited the same file as E48 and then took extra time to reconcile the two changes. E33 (416 seconds) and E40 (349 seconds) were the next slowest.

Run-time distribution.
Fifty-seven of 63 runs finished within five minutes. Conflicts and retries explain most of the longer runs.

Most runs were fairly predictable. Nineteen finished in under 150 seconds, and 38 took 150 to 300 seconds. Only 6 crossed five minutes. That kept the mean (205 seconds) close to the median (201), despite a p95 of 316 seconds.

Mean run time grew from 190 s for the first 25 experiment issues to 234 s for the last 25 (+23%) as the repo grew.
As the repo grew, mean run time moved from 190 seconds for the first 25 experiment issues to 234 seconds for the last 25.

The pipeline did slow down as the experiment continued. Mean run time rose from 190 seconds for the first 25 experiment tickets to 234 seconds for the last 25, a 23% increase. The growing codebase may explain some of that, although we did not isolate the cause. The pipeline did not adjust its settings in response, so self-optimizing remains an area to explore.

Cumulative merged PRs against issues opened.
Merges usually kept pace with new tickets. The two visible pauses came from a rate-limit incident and four merge races.

We opened the issues over 3.7 hours in waves of three, then two after the rate-limit incident. That kept the queue short enough for merges to follow dispatch closely. The line flattened when four runs died at step 1 over about 40 minutes and when four PRs lost merge races and had to be re-filed. Even with those stalls, 59 of 63 PRs merged, including 46 of 50 experiment PRs on their first attempt.

Per-phase run time, merge rate and PR size.
Bigger PRs tended to take longer, but size was not the whole story: the small challenge PRs took about as long as the improve phase.

The optimize phase produced the largest PRs (156 added lines on average) and the slowest runs (241 seconds), while heal tickets averaged 140 lines and finished fastest at 177 seconds. The challenge tickets averaged only 47 lines and still took 232 seconds. Conflict handling, review and workflow retries mattered alongside diff size.

Phase results show where the pipeline spent its time and where reviews found more to discuss.
PhaseTicketsMergedMean run (s)Mean lines addedReview rounds with findings
core1091861059
fix13131805715
heal1081771405
improve10923213212
optimize101024115610
measure552171154
challenge55232474

The PRs that needed another attempt clustered in core (9/10 merged) and heal (8/10); optimize, measure and challenge all reached 100%. All 13 pipeline-fix tickets merged too. Those fixes were the smallest PRs on average, yet drew the most review findings (15). That makes sense in context: they touched workflow files, where even a small diff can shape every run that follows.

Self-healing events seen inside the pipeline.
Once the pipeline knew a recovery path, it followed it reliably. New failure types still needed someone to show the way.

The pipeline resolved all 17 merge conflicts without help. After F02 added a merge-retry loop, all 3 later merge races recovered. After F07 added the watchdog, both later red-main events recovered. The incidents that still needed us happened before a recovery path existed: 4 early merge races, 4 runs interrupted by rate limits and a test gate that was either blind or already red.

Test gate by PR. The gate reported <code>fail</code> on 37 PRs, which all merged because AUTO_MERGE skips the guard. Red tests on main carried forward to later PRs.
The test gate began without a root test script, briefly worked, then stayed red while 37 PRs merged because AUTO_MERGE=1 skipped the guard.

Each mark represents a PR in merge order. The opening none marks appear because there was no root test script until F01. Once a failing test reached main, every later PR inherited it and received fail, regardless of what that PR changed. F04 and F06 briefly restored green. This made the signal hard to use: a red mark did not necessarily mean its PR introduced a problem, and a gate that stayed red could not point us to the PR that did.

There was another gap in the test evidence. In 29 of 63 PR summaries, the agent says it did not run the tests. The gate was our only independent check, and it was unavailable until F01 and red for much of the run. We would want a stronger safety net before relying on this workflow for less forgiving work.

How the pipeline learned to recover

We saw a helpful pattern throughout the run: after a person spotted a new failure and filed a fix ticket, the pipeline handled later occurrences on its own. In that sense, each fix gave it a little more independence. The table separates the fixes we got to exercise from F05 and F08, whose target failures never returned.

Every pipeline fix began with an operator-filed ticket. We later saw six of them work in practice; two are still waiting for another chance to be exercised.
IdWhat we noticedWhat we asked forWhat happened nextRequested by
F01Test gate ran nothing: Tests: none for research PRsRoot package.json with scripts.testNext PRs reported Tests: passoperator
F02Merge race: base moved after the eval, gh pr merge failed; 4 of 24 early PRs needed a humanBounded re-sync + conflict-resolve + retry loop around the merge3 merge retries fired, all 3 merged; failures 4/24 → 0/31operator
F03Engine npm test failed on Node 24 (my own spec bug)Explicit test glob in package scriptMerged; script passesoperator
F04AUTO_MERGE let red tests onto mainFix two failing testsGreen for about 1 minute: E29 (a rerun finishing concurrently) made main red again, then E35 added another red testoperator
F05GitHub's secondary rate limit interrupted 4 runs at step 1 (~40 min)gh_retry backoff helper for all comment callsThe situation did not return after the fix, so it is still to be verified under loadoperator
F06Main red again (two more tests)Fix two failing testsGreen: 380/380operator
F07Nothing watched main; 37 PRs gated failNew main-watchdog.yml opens a main-red issueCaught 2 of 2 red-main events with no human action after the merge; MTTR 7.5 min and 1.8 minoperator (the watchdog itself); detection and repair are automatic
F08Early-step failures were never retriedRetry workflow also reruns failures before the agent startsThe situation has not returned yet, so this is still to be verifiedoperator
R01-R04E06/E16/E17/E23 PRs waiting after a merge raceRe-filed the same tickets4 of 4 merged on the first retryoperator

That pattern made the pipeline more reliable, but the improvement was supervised. The system implemented all 8 fixes; it did not decide which fixes to make.

Most automatic recoveries came from familiar merge conflicts; newer incident types still needed our help.
Incident classCountRecovered automatically
merge conflicts with main1717
merge race (before F02)40
merge race (after F02)33
runs interrupted by rate limits40
red main (before F07)20
red main (after F07)22
test-gate blind spot10
package test script issue10
Total3422 (65%)

The pipeline recovered from 22 of 34 incidents on its own (65%), with familiar conflicts accounting for 17 of those 22 recoveries. Without conflicts, the rate was 5 of 17 (29%). We came away confident that the pipeline could follow recovery paths it knew, while unfamiliar failures still benefited from a person stepping in.

Time to recover a red main. By hand: 66 and 95 min. With the watchdog: 7.5 min (including 343 s before the watchdog existed) and 106 s for the chaos drill (detect 13 s + repair 93 s).
With the watchdog in place, the clean chaos drill recovered in 106 seconds; the earlier operator-led recoveries took 66 and 95 minutes.

Most of the manual recovery time was spent waiting for someone to notice, not repairing the problem itself. The first watchdog event took 7.5 minutes, including 343 seconds that passed before the watchdog existed. In the chaos drill, detection took 13 seconds and the repair took 93. Giving the pipeline a way to notice the problem made the biggest difference.

Five tickets that invited a little more judgment

For the last five experiment tickets, we gave the pipeline a few less tidy situations: ambiguity, a seeded bug with a process requirement, simultaneous edits to the same function, and an impossible request. All five merged, and each gave us a different view of how the pipeline makes decisions.

All five tickets merged. The conflicting-edit pair gave us the clearest example of autonomous recovery.
TicketChallengeWhat the pipeline didWhat we learned
E46Under-specified request: “Make the engine faster.”Skipped adaptive-TTL history bookkeeping when adaptation is disabled, then added a 5,000-lookup behavior test. PR #241 added 33 lines across 2 files and finished in 182 seconds.It chose a safe, local interpretation without asking what workload or performance target mattered.
E47Seeded backoff bug plus an instruction to write a failing test first.Clamped positive jitter to an integer maxMs and added tests for positive jitter and fractional caps. PR #243 added 17 lines and finished in 106 seconds.The final code fixes the bug, but the single commit does not establish a red-green sequence. The agent also said it had not run the tests.
E48First simultaneous edit to runner.js.Added onGiveUp, called after exhausted retries or a permanent error, with tests ensuring callback failures do not change the task result. PR #246 merged first in 154 seconds.The first half established the version of runner.js that its sibling would have to reconcile.
E49Second simultaneous edit to runner.js.Added per-attempt timeoutMs handling and tests while preserving E48's onGiveUp callback. PR #247 detected and resolved the conflict automatically.This was the clearest trap success: both independently requested features survived the merge. It was also the slowest run at 543 seconds.
E50A request to guarantee success even when the function always throws.Kept the outcome honest and added structured diagnostics with the reason, attempt limit and classified error history. PR #249 added 58 lines across 2 files and finished in 173 seconds.The pipeline returned the closest honest behavior allowed by the ticket. This shows careful compliance with the escape hatch more than independent refusal.

E46 offers a useful lesson for everyday product work. The chosen optimization is reasonable: a non-adaptive cache does not need per-key hit and miss history. Still, “faster” could have meant task latency, throughput, memory use or startup time. The pipeline chose cache bookkeeping, and its new test checked behavior rather than measuring elapsed time. The patch was tidy, but a quick clarifying question would have helped it solve the right product problem.

E47 separates code correctness from process compliance. The final diff covers the seeded edge cases and fixes the cap, but the ticket also required a failing test first. With a single commit and an admission that tests were not run, the experiment cannot show that the pipeline followed the requested red-green sequence.

E48 and E49 gave us the clearest success in the set. Both tickets changed the same function at nearly the same time. E48 landed onGiveUp first; E49 then resolved the conflict, retained that callback and added timeout handling. E50 also produced a good result, with the prompt offering a path away from the request that could not be fulfilled literally. A useful next step would be helping the pipeline ask a question or invite a human in when a ticket leaves room for interpretation.

The engine gave us more encouraging results in simulation

By the end of the run, the engine had grown to 43 source files, 2,474 source lines and 3,585 test lines, with 392 passing tests. This is where the experiment began to feel especially promising.

Success rate with healing off and on, 10 seeds x 400 jobs per point; band is one standard deviation.
Retries keep success above 87% through a 60% fault rate. At higher fault rates, success tapers as the retry budget runs out.

At 10 to 30% injected faults, healing lifts success to 99% or better. At 60%, it more than doubles success from 41.1% to 87.9%, the largest gain in the experiment. At 90% faults, only 34.5% of jobs succeed because repeated attempts keep hitting the same high failure probability. Variation across seeds also widens as faults rise.

Healing improves success most at a 60% fault rate, with extra attempts and tail latency at every level.
Injected fault rateBaseline successHealing successGainMean attemptsp95 latency (virtual ms)
10%89.8%100.0% ±0.0+10.3 pt1.00 → 1.1110 → 25
20%80.9%99.9% ±0.2+19.0 pt1.00 → 1.2410 → 26
30%71.5%99.2% ±0.2+27.7 pt1.00 → 1.4010 → 40
50%50.7%94.0% ±1.3+43.3 pt1.00 → 1.8610 → 55
60%41.1%87.9% ±1.2+46.8 pt1.00 → 2.1610 → 55
80%19.7%58.5% ±2.0+38.8 pt1.00 → 2.9710 → 55
90%10.6%34.5% ±2.7+23.9 pt1.00 → 3.4410 → 55
The tradeoff for healing: more attempts per job and higher tail latency.
Extra reliability requires more retries. At 60% faults, mean attempts more than double and p95 latency rises from 10 to 55 virtual milliseconds.

That extra reliability comes with a tradeoff. At 30% faults, mean attempts rise from 1.00 to 1.40 and p95 latency rises from 10 to 40 virtual milliseconds. By 60%, jobs average 2.16 attempts. Tail latency flattens at 55 because the retry budget caps it, not because the dependency has recovered. In production, we would pair those retries with care so they do not add too much load to a dependency that is already struggling.

Cumulative regret over 1000 pulls, 30 seeds: random 200.5, epsilon-greedy 29.8, Thompson 12.1.
Thompson sampling preserves more reward over 1,000 choices, ending at 12.1 regret versus 29.8 for epsilon-greedy and 200.5 for random selection.

Regret measures the reward lost when the selector does not choose the best option. Random selection keeps making the same class of mistake, so its regret climbs roughly linearly. Thompson sampling learns faster: it cuts regret by 94% against random and chooses the best arm 95% of the time.

The A/B adoption gate finds a true +10 point lift 99% of the time at n=800 per arm (13% at n=25) and adopts 2.0% of equal arms at n=800.
The A/B gate becomes much more dependable with hundreds of samples per arm: 99% power at 800, compared with 13% at 25.

At 800 samples per arm, the gate detects a true 10-point lift 99% of the time and falsely adopts an equal arm 2% of the time. At 25 samples per arm, it detects the same lift only 13% of the time. For this pipeline experiment, that means 50 tickets do not provide enough observations to tune settings safely with this gate.

Distance to the optimum on a noisy 3-parameter problem, 30 seeds.
The auto-tuner finds settings closer to the optimum at every budget, with the clearest difference appearing after 20 evaluations.

The tuner takes a little time to get going between 5 and 10 evaluations, then closes quickly on the optimum. Random search improves more slowly and remains farther away, even with four times the budget.

At 40 evaluations, the tuner reaches 0.36 error; random search is still at 2.09 after 160.
Evaluation budgetAuto-tuner errorRandom-search error
53.386.13
103.385.04
202.044.32
400.363.47
800.082.81
1600.072.09

At the smallest budget, the tuner is already about twice as close as random search (3.38 versus 6.13). At 160 evaluations it is 31 times closer. These are averages over 30 seeds on a synthetic three-parameter problem. They show that the algorithm works under these test conditions, but they do not tell us how much it would improve this pipeline.

A latency shift at request 500: a fixed 250 ms timeout fires 493 times, the adaptive timeout 27 (mean 513 ms).
After latency jumps at request 500, the adaptive timeout widens to match the new conditions and reduces false timeouts from 493 to 27.

The fixed 250-millisecond threshold treats nearly every slower call after the shift as a timeout. The adaptive threshold follows the new distribution and settles at a mean of 513 milliseconds. That removes 466 false timeouts, but a dead call now takes about twice as long to declare. Adaptation trades faster failure detection for fewer false alarms.

AIMD concurrency tracks a capacity drop from 24 to 8; 90 of 7427 requests rejected (1.21%).
AIMD follows a capacity drop from 24 to 8 and rejects 90 of 7,427 requests (1.21%).

The limiter does not know the safe concurrency ahead of time. It increases until requests fail, then backs off. When simulated capacity drops from 24 to 8, concurrency follows it closely enough to keep rejection at 1.21% across the run. This is the behavior we expect from AIMD, so the result gives us confidence in the implementation under simulation rather than revealing a new algorithmic result.

Engine scorecard from measured values (proxy mapping, see caveats).
Healing and improving score near 96. Optimizing reaches 67.5, with room to grow beyond the adaptive cache TTL's 1.8-point hit-rate gain.

The overall score is 87, but it is not an external grade. We chose how each measurement maps into the engine's scorecard(). The chart helps compare the three areas under that mapping; it does not mean the engine is “87% self-improving” in a general sense.

Adaptive TTL raised the hit rate from 16.1% to 17.9% on a skewed 5,000-request workload.

scenarios.json lists 12 scenarios, but the simulator models only failure rate. The latency-spike and corruption scenarios therefore reproduce the plain fault-rate results, leaving 8 distinct outcomes. Ticket E44 asked for 12 scenarios, and the pipeline followed the request closely. A stronger check of the simulator's capabilities would have helped it notice that all 12 could not yet be represented.

What we took away from the run

Self-healing

The pipeline heals failures covered by existing rules: conflicts (17/17), merge races after F02 (3/3) and red tests on main after F07 (2/2). New kinds of failure still need a person to help establish the recovery path. The rate-limit incident needed 8 manual reruns, and the comment-chat agent cannot run git fetch/merge, so it could not rescue stale PRs. F05 and F08 target those situations but were never exercised afterward. For now, healing means a growing collection of hand-reviewed rules, and the run gave us a useful foundation to build on.

Self-improving

The pipeline implemented all 8 requested fixes, including four edits to its own workflow files. We still chose what to improve, but once asked, it handled the implementation well. Its autonomous improvement path is currently the watchdog, which turns a red npm test into a ticket. Connecting failed runs, slow runs and repeated review findings to the same loop would be a natural next step. The engine already has a module for that, issuedraft.js, waiting to be connected to the pipeline.

Self-optimizing

The pipeline did not change its own parameters during this run. Review rounds, diff limits, wave size and concurrency were fixed or hand-set, and we lowered the wave size after the rate-limit incident. Run time grew along with the repo. The good news is that the engine already contains tuners and adaptive components that performed as designed in simulation; the next challenge is giving the pipeline a safe way to use them.

A few things to keep in mind

We want to keep the results in perspective. This was a useful experiment, not a final verdict on what the pipeline can do.

What we'd be excited to try next

If we run the experiment again, these are the changes we would start with.

  1. Pair AUTO_MERGE=1 with a test gate that blocks when the gate says fail. In this run, 37 PRs merged red.
  2. Keep F07 and extend the watchdog to failed workflow runs, so F05/F08-style incidents open tickets too.
  3. Serialize or queue merges, or keep shared registries such as index.js out of tickets: that file caused the original merge races.
  4. Feed run metrics (duration, retries, findings per PR) back into settings such as review rounds and wave size, with an A/B gate like the engine's compareArms.
  5. Give the agent a way to ask about under-specified tickets before choosing an interpretation (E46).

Explore the data or reproduce the run

python3 -I research/harness/collect.py     # gh API -> research/data/metrics.json|csv
node research/harness/engine_eval.mjs      # engine measurements -> research/data/engine-eval.json
python3 -I research/harness/charts.py      # docs/research/*.svg
python3 -I research/harness/report.py      # this page

If you'd like to explore the results or run the experiment yourself, start with metrics.csv, metrics.json and engine-eval.json. You are also welcome to browse the research code and experiment tools. If you spot something we missed, try a variation or have an idea for the next run, we'd genuinely love to hear about it.