How we make AI coding more cost efficient without sacrificing task quality

Stripping tool output to save tokens actually made GitHub Copilot tasks more expensive and take longer to finish. 👇

When building AI coding agents, developers often optimize for local metrics, like making individual tool responses smaller. But if a compressed response omits a crucial detail, the agent gets stuck, re-runs commands, or re-reads logs to recover context. Those extra recovery turns inflate the total token count and drag out task execution.

GitHub realized tokens per tool call is the wrong metric to optimize. Instead, they re-engineered the Copilot harness to cut model work across the entire task without hiding information the agent needs.

First, they built a selective output compressor. Early prototypes aggressively truncated everything, but agents kept reopening original outputs like `git diff` to fetch missing details. GitHub adjusted the strategy: source code, diffs, and raw file reads are left untouched, search results are reorganized losslessly, and only repetitive build, test, and lint noise gets compressed. They also added a recovery path so agents could fetch raw logs if needed—tracking how rarely agents used it proved the compressor wasn't stripping vital context.

Second, they removed formatting leftovers. Copilot's file-reading tool prefixed every line with line numbers—a leftover from older editing tools. Modern agents match surrounding code context rather than line numbers. Removing those unused prefixes sent file contents to the model cleanly, freeing up context window space.

Third, they shrank system prompts. Using a meta-prompting loop, GitHub cut task-tool instructions by roughly half. An early prompt rewrite accidentally broke parallel processing by forcing independent sub-agents to run sequentially. GitHub caught the bug in online testing, wrote a regression test, and replaced complex explicit rules with a single sentence letting the model decide when to run tasks concurrently: "Independent agents can run in parallel; consider side effects."

Fourth, they eliminated unnecessary model turns. When background shell commands or sub-agents finished, the harness used to wake the model just to announce completion, forcing the agent to spend another turn requesting the output. The harness now batches completed background tasks and delivers the output directly inside the wake-up notification, skipping the extra retrieval call entirely.

The payoff across Copilot tools was immediate: * Removing line-number prefixes cut model inference costs by 3% to 5%. * Prompt compression removed ~1,300 tokens per turn, dropping active hourly costs by 2.9%. * Direct background result delivery reduced token usage by 2.3%. * Combined with earlier instruction tuning, overall code review costs dropped by roughly 20% without hurting task quality.