Activation steering

Nudge a model's behaviour at inference time by adding a direction to its residual stream — the same mechanism behind "concept vectors" / representation engineering. You supply one or more steering vectors; the engine adds each to the hidden state at the layer(s) you name, every forward pass, and generation continues from the perturbed state.

Same POST https://infra.acsresearch.org/v1/completions endpoint and key as a normal completion — you add an apply_steering_vectors field. Steering runs on the same activation engine as harvesting.

The shape of a request

{
  "model": "llama-8b",
  "prompt": "I think that",
  "max_tokens": 32,
  "apply_steering_vectors": [
    {
      "activations": { /* the vector, in the codec below */ },
      "layer_indices": [16],     // which layer(s) to add it at
      "scale": 8.0,              // multiplier on the vector
      "norm_match": false,       // rescale to the hidden state's norm first?
      "position_indices": null   // null = all positions; or a list of token positions
    }
  ]
}

You can send up to 8 vectors in one request; each is applied independently.

The activations codec

The vector is encoded the same way as harvested activations — bfloat16 values reinterpreted as int16, zstd-compressed, then base64-encoded:

import base64, numpy as np, ml_dtypes, zstandard as zstd

def encode_vector(vec: np.ndarray) -> dict:
    """vec: float array with ONE ROW PER STEERED LAYER.
    (n_layers, hidden)              -> each layer's vector broadcast to all positions
    (n_layers, n_positions, hidden) -> position-specific
    n_layers must equal len(layer_indices).
    """
    a = vec.astype(ml_dtypes.bfloat16).view(np.int16)     # bf16 bits as int16
    packed = zstd.ZstdCompressor(level=1).compress(a.tobytes())
    return {
        "data": base64.b64encode(packed).decode(),
        "dtype": "int16",
        "original_dtype": "torch.bfloat16",
        "shape": list(a.shape),
        "compression": "zstd",
    }

The leading axis is layers. shape[0] must equal len(layer_indices) — one direction per layer you're steering (a mismatch returns 400). So:

• steer one layer → layer_indices: [16], vector (1, hidden)
• steer two layers → layer_indices: [16, 20], vector (2, hidden) (row 0 → layer 16, row 1 → layer 20)
• target specific token positions → make it 3-D (n_layers, n_positions, hidden) and set position_indices

hidden is the model's width — 4096 (llama-8b), 16384 (llama-405b), 3072 (trinity-truebase); a mismatched hidden is also rejected with 400.

A full example

import base64, json, numpy as np, ml_dtypes, zstandard as zstd, urllib.request

BASE, KEY = "$ACS_API_BASE", "$ACS_API_KEY"

def post(body):
    req = urllib.request.Request(f"{BASE}/completions", data=json.dumps(body).encode(),
        headers={"Authorization": f"Bearer {KEY}", "Content-Type": "application/json"})
    out = json.load(urllib.request.urlopen(req, timeout=300))
    if "error" in out:   # late cold-boot failures arrive inside a 200 body
        raise RuntimeError(out["error"].get("message", str(out["error"])))
    return out

# In practice this vector is a learned/derived direction (a probe weight, a
# difference-of-means between two prompt sets, an SAE decoder column, …).
vec = my_direction.reshape(1, 4096)          # (n_layers=1, hidden) — one row for layer 16

sv = {"activations": encode_vector(vec), "layer_indices": [16],
      "scale": 8.0, "norm_match": False, "position_indices": None}

gen = {"model": "llama-8b", "prompt": "I think that", "max_tokens": 32, "temperature": 0}
print("baseline:", post(gen)["choices"][0]["text"])
print("steered :", post({**gen, "apply_steering_vectors": [sv]})["choices"][0]["text"])

Sanity check while wiring it up: scale: 0.0 is the identity — a zero-scale steer must reproduce the baseline text exactly. If it doesn't, your request isn't reaching the steering path.

Field reference

Which models support this?

All three models we serve support steering (and harvesting): llama-8b, llama-405b, and trinity-truebase.

Practicalities

• One sequence per steered request: steering requires a single prompt and n: 1; a batched (list-valued) prompt or n > 1 returns 400. The engine applies vectors to the first sequence of a request only, so before this guard those shapes came back part-steered with a 200. To fan out, send separate requests (vary seed for sampled diversity).
• Cold boot & quota: same as harvesting — llama-8b wakes from a GPU snapshot in ~30–40 s after an idle gap; the big engines (llama-405b, trinity-truebase) cold-boot for several minutes on the first request after idle, and the wrapper holds that request open through the boot (set a client timeout of at least 15 minutes — see Cold-boot waiting). Requests are attributed to your key against the activation quota.
• Baselines: same seed ≠ same engine. Ordinary completions and activation/steering requests run on separate engines per model. A seed reproduces a draw within an engine, so comparing a steered request against a same-seed plain completion is not a valid control — the two engines sample different streams. Run the control on the activation engine too: a scale: 0.0 steering vector is an exact identity (verified — it reproduces the same-engine unsteered output bit-for-bit), so {**req, "apply_steering_vectors": [{**sv, "scale": 0.0}]} is the clean baseline.
• Finding a good scale: start small and sweep up. Two things interact — the scale and how many layers you steer: pushing a wide band compounds fast. The worked example below (a single mid layer, raw vector) has its sweet spot around 1.5–2; with norm_match: true over a small band, useful scales are typically below 1. Watch for the output degrading into repetition — that's the sign you've overshot.

Worked example: a "pirate" contrast vector

A full end-to-end demonstration — done entirely through this API — lives as a marimo notebook at pirate_steering.py (run: ACS_API_KEY=… uvx marimo edit --sandbox --watch pirate_steering.py), with a rendered run including all cell outputs at pirate_steering.html — readable without an API key. It:

1. harvests residual-stream activations for six pirate-styled sentences and six plain paraphrases (one capture each; mean over content tokens, BOS position excluded),
2. builds the direction: mean(pirate) − mean(plain), per layer — contrasting actual text in the target style,
3. steers layer 16 of llama-8b with the raw vector, sweeping scale, and
4. scores the effect two ways — a pirate-word count and a logprob preference metric (echo + prompt_logprobs, shared prefix cancelling).

The measured dose–response (temperature 0.8, pinned seed):

The same vector generalizes to new prompts: at 2.0, job-day advice becomes "go in with a swashbucklin' attitude and a bountiful chest o' booty, Mateys!" — the direction carries the style itself, so it transfers.

See also

• Activation harvesting — capture the residual stream (and derive steering directions from it).
• API reference — the base /v1/completions contract.