Activation harvesting

Get a model's internal residual-stream activations back alongside a normal completion — the values flowing between the model's layers, for interpretability and probing work (SAEs, linear probes, logit lens, …).

It's the same POST https://infra.acsresearch.org/v1/completions endpoint and the same API key as ordinary completions — you just add one field. Activation requests run on a separate engine per model, kept apart from ordinary completions. One consequence for seeded work: a seed reproduces a draw within an engine, so don't use a plain completion as a control for an activation-engine request — see Baselines on the steering page for the clean recipe.

Turn it on

Add "output_residual_stream": true to the request body:

curl -s "$ACS_API_BASE/completions" \
  -H "Authorization: Bearer $ACS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-8b",
    "prompt": "The capital of France is",
    "max_tokens": 1,
    "temperature": 0,
    "output_residual_stream": true
  }'

true captures the residual stream at every decoder layer. If you only need a few layers, pass a list of layer indices instead — the subset is applied on the engine, so only those layers cross the wire (much smaller responses):

"output_residual_stream": [15, 20]   // capture only layers 15 and 20

n_layers in the returned shape is then the number of layers you asked for (in ascending order), not the model's full count. Pass true for all layers.

What comes back

The normal completion JSON, plus a top-level activations object:

{
  "choices": [ /* the normal completion */ ],
  "usage":   { /* the normal usage */ },
  "activations": {
    "residual_stream": {
      "data": "<base64 string>",       // zstd-compressed, int16-encoded bf16 bytes
      "dtype": "int16",                // storage dtype (bf16 reinterpreted as int16)
      "original_dtype": "torch.bfloat16",
      "shape": [n_layers, n_tokens, hidden],   // n_layers = full count for `true`, else how many you requested
      "compression": "zstd"
    }
  }
}

shape[0] is the model's full layer count (e.g. 32 for llama-8b, 126 for llama-405b, 60 for trinity-truebase). Slice the layers you want after decoding.

Which token positions are captured

shape[1] covers every token that went through a forward pass:

n_tokens = prompt tokens + generated tokens − 1

A 6-token prompt with max_tokens: 5 returns 10 positions: rows 0–5 are the prompt, rows 6–9 are the generated tokens that were fed back through the model. The final sampled token is not included — it's drawn from the last forward pass but never processed itself, so it has no residual stream. With max_tokens: 1 (the probing default, as in the example above) that reduces to exactly the prompt positions.

So a single request captures most of the generation trajectory too, not just the prompt — useful when you want activations while the model writes, at the cost of a larger payload (the prompt-length cap below applies to the prompt).

Decode it

dtype: "int16" holds the bfloat16 bit pattern stored as int16. Reinterpret those bits back to bfloat16 with .view(...), as the code below does — a numeric .astype() would convert the integer values and give you garbage:

import base64, numpy as np, ml_dtypes, zstandard as zstd

def decode_residual_stream(resp):
    d = resp["activations"]["residual_stream"]
    raw = base64.b64decode(d["data"])
    if d.get("compression") == "zstd":
        raw = zstd.ZstdDecompressor().decompress(raw)
    # reinterpret the 16 bits as bfloat16 with .view (an .astype would corrupt them)
    arr = np.frombuffer(raw, dtype=np.uint16).view(ml_dtypes.bfloat16)
    return arr.reshape(d["shape"])          # (n_layers, n_tokens, hidden)

x = decode_residual_stream(resp)            # e.g. shape (32, 6, 4096) for llama-8b
layer16 = x[16]                             # slice the layer(s) you want, client-side

Always assert the shape

A capture hook that fails is skipped server-side with only a log warning, so a partial capture can come back looking like valid data. Assert before you trust it:

x = decode_residual_stream(resp)
assert x.ndim == 3, x.shape
assert x.shape[0] == n_layers_for_model, f"partial capture: {x.shape[0]} layers"
assert np.isfinite(x.astype(np.float32)).all(), "non-finite — broken capture"

Which models support this?

All three models we serve support harvesting (and steering): llama-8b, llama-405b, and trinity-truebase.

Limits & practicalities

Prompt-length cap. Because capture returns every layer, the response can be large — a big model at 128 tokens is hundreds of MB. Each model advertises a max_activation_prompt_tokens cap under its capabilities in /v1/models; a longer capture prompt gets a 400 activation_prompt_too_long. Caps are tight on the largest models (~4 MB per token). This inline endpoint is meant for probing. It comfortably handles up to ~10k examples; for corpus-scale harvesting, use Bulk harvest — a batch job that writes shards you download by URL.
Cold boot. All activation engines scale to zero when idle. llama-8b restores from a GPU snapshot, so its first request after an idle gap takes ~30–40 s; everything after that is interactive. The big engines (llama-405b, trinity-truebase) rebuild from scratch — several minutes on the 8×H200 models. The wrapper holds that request open and sends keepalive bytes while the engine starts, so it normally completes in one shot — set a client timeout of at least 15 minutes rather than retrying. If the engine can't come up (e.g. the 14-minute safety deadline passes), you get a retryable modal_cold_boot error — note it can arrive inside a 200 response body once keepalive bytes have committed the status, so check the body for error, don't rely on the HTTP code alone. See Cold-boot waiting.
Usage & quota. Activation requests are attributed to your key and may count against a per-key monthly activation quota (a heavier operation than a plain completion); over quota returns 429 activation_quota_exceeded.