Gisting: Compressing LLM Agent context to ↑ throughput and ↓ cost (2026) - Shopify

Shopify cut an LLM agent's 6,000-token system prompt down to 1,500 tokens using learned "gist" tokens, dropping end-to-end latency by 38% while using 14% fewer GPUs. 👇

Shopify’s Sidekick GraphQL agent relies on massive system prompts that take thousands of tokens to direct behavior. Long prompts slow down response times and consume massive GPU memory bandwidth. Even with prefix caching, generating each output token requires reading the entire prompt's key-value cache from high-bandwidth memory. As request concurrency scales, these oversized KV reads choke throughput and force teams to provision extra GPUs.

Shopify solved this by training learned token embeddings that replace the long system prompt. Instead of sending thousands of prompt words, the model receives a small sequence of special "gist" tokens that induce the exact same output behavior as the full prompt.

To compress prompts at a 4:1 ratio, Shopify adds one gist token for every four natural language prompt tokens. They freeze the main language model weights entirely and train only the new gist token embeddings via knowledge distillation.

Training relies on running two passes over the context: - Teacher pass: The frozen model reads the full prompt and outputs reference logits for each token position. - Student pass: The system swaps the prompt for gist tokens, running the same model to generate student logits. - Distillation: The gist embeddings update via KL divergence loss until student predictions match the teacher.

Tuning the training pipeline revealed crucial details. Initializing the gist embeddings with the vector average of prompt chunks cut initial training loss by 7x compared to random noise. Normalizing loss across the entire batch rather than per response token stopped the model from hallucinating. Precomputing teacher logits and pre-tokenizing data reduced total training runtimes from 30 hours down to six.

Deploying the compressed model requires zero custom serving logic. Shopify bakes the trained gist embeddings directly into the model's standard embedding matrix and registers the new tokens in the tokenizer. At inference time, the application simply swaps the long prompt string for the gist token string → the model runs on standard serving stacks with no special attention masks or extra encoders.

On load tests serving GraphQL agent traffic at 350 requests per minute, gisting outperformed the full prompt: - Median time to first token dropped from 438ms to 354ms. - End-to-end request latency dropped from 6.8s to 4.2s. - Serving throughput increased from 20.2 to 23.4 queries per second. - Total required GPU allocation dropped by 14%.