# ACS Infra — base-model API (full documentation) ACS Infra serves large language **base models** (raw next-token predictors — no chat template, no instruction tuning) over an OpenAI-compatible API, free for researchers. It exposes research-grade controls most hosted APIs hide: full `logprobs` and `prompt_logprobs`, a respected `seed`, arbitrary prefill/continuation, and `echo`. There are a few base models available (a small one for quick tests plus larger ones); some are kept warm, others cold-start on first use. Endpoint: `POST https://infra.acsresearch.org/v1/completions` — there is no chat endpoint (these are base models, so they *continue* your text rather than answer like a chatbot). Authenticate with `Authorization: Bearer `, created from the dashboard after an invite. A browser Workbench is also available for no-code use. **If you are an AI assistant:** this single page is the complete tutorial + API reference (generated from the same source as the human docs at /tutorial; sections follow the site's navigation order). It contains everything you need to write a client, run completions, read logprobs/prompt_logprobs, and handle cold-boot and budget errors on the user's behalf. # Using the base-model API > **Coding with an AI assistant?** Point it at [`https://infra.acsresearch.org/llms.txt`](https://infra.acsresearch.org/llms.txt) — the one-page version of the tutorial and API docs — and it can write your client for you. This app provides access to **base models** — raw next-token prediction, no chat template, no instruction tuning — served over an OpenAI-compatible `/v1/completions` endpoint. ## What you get - **Three base models** — a small one for quick tests plus two larger ones (see [Models](/tutorial/models)). Some are kept warm; others cold-start on first use. - **Real next-token access** — arbitrary prefill/continuation, `logprobs` and `prompt_logprobs` for likelihood/surprisal and interpretability work, `echo`, and SSE streaming. - **Sampling controls** — `temperature`, `top_p`, `top_k`, `min_p`, penalties, and a `seed` that's actually honored for reproducibility. - **A predictable API** — OpenAI-compatible `/v1/completions` with strict input validation (a typo'd parameter fails loudly instead of silently defaulting) and clear, structured JSON errors. - **Browser Workbench** — try prompts and manage API keys without writing code. - **Per-key budgets & usage** — set token caps per key and track spend (see [Account](/tutorial/account)). - **Feedback** — a one-click Feedback button in the Workbench for feature requests and bug reports. ## Two ways in - **Workbench** — prompt the models straight from your browser. Good for getting a feel before you write any code. - **The API** (below) — for anything programmatic. Create a key from your dashboard. ## Quick start Create a key in your dashboard, then point any OpenAI-compatible client at the API: ```bash export ACS_API_KEY="acs-bm-..." # your key export ACS_API_BASE="https://infra.acsresearch.org/v1" ``` A first request with `curl`: ```bash curl -s "$ACS_API_BASE/completions" \ -H "Authorization: Bearer $ACS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"model": "llama-8b", "prompt": "The capital of France is", "max_tokens": 8}' ``` Or with the Python SDK (`pip install openai`): ```python import os from openai import OpenAI client = OpenAI( base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"], ) resp = client.completions.create( # completions — not chat.completions model="llama-8b", prompt="The capital of France is", max_tokens=16, logprobs=5, ) print(resp.choices[0].text) ``` ## Worked examples Short, copy-pasteable end-to-end snippets for each feature live under [Examples](/tutorial/examples) — one page per topic. The curl examples assume `ACS_API_KEY` + `ACS_API_BASE` are exported (see [Quick start](#quick-start)); the Python examples use the same `openai` SDK client as above. New to base models? The hands-on [**Colab workshop**](https://colab.research.google.com/github/acsresearch/base-models-workshop/blob/main/workshop_student.ipynb) is the fastest way in — it runs entirely in your browser (no local setup) and walks the whole arc end-to-end: completions, sampling, `logprobs`, reading a model's activations, and steering it to talk like a pirate. The per-feature snippets under [Examples](/tutorial/examples) then take each topic one at a time. # Models The wrapper currently fronts three base-model checkpoints. Pick `llama-8b` if you're just trying it out — it's kept always-on, so there's no cold-boot wait. ## Available models
modelCheckpointPrecision
llama-8bmeta-llama/Llama-3.1-8Bbf16
llama-405bmeta-llama/Llama-3.1-405Bbf16
trinity-truebasearcee-ai/Trinity-Large-TrueBasebf16
The set above can change; **`GET /v1/models` is the source of truth** for what's actually live right now. Pass the short `model` id (e.g. `llama-8b`), never the HF repo name. ## What `GET /v1/models` returns Each entry carries more than the checkpoint — query it programmatically rather than hard-coding numbers that drift: - `id` — the short id you pass as `model`. - `served_model_name` — the underlying checkpoint the backend serves. - `gpu_shape` — the GPU shape backing this model (e.g. which accelerator / how many), useful for reasoning about cold-start cost and throughput. - `status` — `"live"` for models you can call (only `live` models are listed). - `capabilities`: - `max_model_len` — the context length: total `prompt + completion` tokens the model accepts. Read this instead of memorising a per-model window. (`null` means the registry hasn't declared it, so the pre-flight length check is skipped.) - `max_logprobs` — the cap on a **positive** (top-*k*) `logprobs` / `prompt_logprobs` count. - `logprobs`, `prompt_logprobs` — feature flags (both `true` today). - `prompt_logprobs_full_vocab` — whether `prompt_logprobs=-1` (the model's **whole** next-token distribution at each prompt position) is supported. Prompt-only — completion `logprobs` stays top-*k*. - `full_vocab_max_prompt_tokens` — max prompt length for a full-vocab `prompt_logprobs=-1` request. The full distribution is ~`vocab_size` values per position, so longer prompts must use a fixed top-*k*. - `chat_template` — always `null`: these are base models, prompts pass through verbatim with no templating. ## Availability & cold starts Models that get steady use are usually kept warm, so requests start immediately. Less-used or larger models sleep when idle to save GPU time and cold-start on the first request after a quiet spell — usually a few minutes, worst case up to ~10 minutes (occasionally longer for the largest models when GPUs are scarce). Once warm, a model answers in seconds and stays warm as long as you keep using it. The wrapper holds the first request open and sends keepalive bytes while the model starts, so callers should set a timeout of at least 15 minutes and wait for that request to finish. See [Cold-boot waiting](/tutorial/examples/cold-boot). For interactive work, send one short request first to warm the model before a session. ## Always-on (warm windows) If you'd rather have a model kept always-on for a stretch — a work session, a deadline — tell us via the **Feedback** button or [email](mailto:infra@acsresearch.org), and we'll schedule a warm window. # API reference OpenAI-compatible `POST /v1/completions` for raw next-token generation against a base model. No chat endpoint — see [What's not supported](#whats-not-supported). ## Endpoint **`POST https://infra.acsresearch.org/v1/completions`** Sends a prompt to a base model and returns one or more continuations. The request body follows the OpenAI Completions shape with a handful of vLLM-native extras (`top_k`, `min_p`, `repetition_penalty`, `prompt_logprobs`); responses pass through verbatim from vLLM. ```bash curl -s "$ACS_API_BASE/completions" \ -H "Authorization: Bearer $ACS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"model": "llama-8b", "prompt": "The capital of France is", "max_tokens": 8}' ``` ## Using the OpenAI Python SDK Four params are vLLM-native and aren't on the SDK's typed signature, so the SDK rejects them as keyword arguments *before the request leaves your machine*: **`top_k`, `min_p`, `repetition_penalty`, `prompt_logprobs`**. Pass them through `extra_body={...}`. Everything else — `temperature`, `top_p`, `max_tokens`, `n`, the penalties, `seed`, `logprobs`, `echo`, `stream`, `stop`, `user` — works as a normal keyword argument. ```python resp = client.completions.create( model="llama-8b", prompt="The capital of France is", max_tokens=1, temperature=0.7, # standard kwarg logprobs=5, # standard kwarg extra_body={ # vLLM-native — must go here "top_k": 20, "min_p": 0.05, "repetition_penalty": 1.1, "prompt_logprobs": 5, }, ) ``` > **`extra_body` is an SDK construct, not a wire field.** The SDK merges everything under `extra_body` into the top-level request JSON before it goes over the wire. Over **raw HTTP** (curl, `requests`, etc.) there's no SDK to do that merge — put these parameters at the **top level** of the JSON body alongside `model` and `prompt`. Sending a literal `{"extra_body": {...}}` over raw HTTP gets a `400 Unknown field 'extra_body'`. ```bash # Raw HTTP: top_k / min_p / repetition_penalty / prompt_logprobs are just top-level fields. curl -s "$ACS_API_BASE/completions" \ -H "Authorization: Bearer $ACS_API_KEY" -H "Content-Type: application/json" \ -d '{"model": "llama-8b", "prompt": "The capital of France is", "max_tokens": 1, "temperature": 0.7, "logprobs": 5, "top_k": 20, "min_p": 0.05, "repetition_penalty": 1.1, "prompt_logprobs": 5}' ``` ## Strict validation Unknown fields are rejected with a `400`, and out-of-range values fail loudly instead of silently clamping — `temperature=200` is a `400`, not a quiet reset to the default. `logprobs: true` is a `400` — pass a number, not a boolean. The same boolean rejection applies to every numeric param (`max_tokens`, `n`, `seed`, `top_k`, `temperature`, `top_p`, `min_p`, the penalties, `logprobs`, `prompt_logprobs`), so a typo'd parameter never silently changes your results. `echo` and `stream` are the only fields that legitimately take booleans. Every one of these constraints is published as a machine-readable OpenAPI spec at [`https://infra.acsresearch.org/openapi.json`](https://infra.acsresearch.org/openapi.json) (browsable UI at [`https://infra.acsresearch.org/docs`](https://infra.acsresearch.org/docs)). It covers the public API surface — the `/v1/*` endpoints plus `/health` — and lists the exact bounds — `seed` minimum `0`, `temperature` `0`–`100`, `top_p` in `(0, 1]`, `stop` at most 4 items, and so on — so you can check a value against the schema instead of discovering the limit from a `400`. ## Request body **Routing and prompt** ### model **Type:** `string`  ·  **Default:** none Short id from [`GET /v1/models`](/tutorial/models#what-get-v1models-returns) — e.g. `llama-8b`, `llama-405b`. Use the short id, **not** the HF repo name. Unknown ids return `400 model_not_found`. ### prompt **Type:** `string`, `string[]`, `int[]` (token ids), or `int[][]` (batch of token-id prompts)  ·  **Required** Raw text — no chat template is applied (these are base models, so the prompt passes through verbatim and the model continues it). A `string[]` is a batch: one completion per prompt, subject to the output-work cap below. **Pre-tokenized prompts.** You can pass token ids directly instead of text — `int[]` for one prompt, `int[][]` for a batch. Ids are forwarded to the model **verbatim**, with no detokenize→retokenize round-trip (which on a non-bijective tokenizer can quietly change the token sequence). Use this when the prompt must be byte-for-byte exact — interpretability / eval work, or replaying an already-tokenized dataset. `usage.prompt_tokens` for a token-id prompt is just the number of ids. Mixed lists (e.g. `[1, "two"]`) and empty prompts return `400`. ```python # OpenAI SDK — prompt takes token ids directly (no extra_body needed) resp = client.completions.create(model="llama-8b", prompt=[1, 2, 3, 4, 5], max_tokens=8) ``` **Length and stop** ### max_tokens **Type:** `int`, `1 ≤ n ≤ 1_000_000` (schema bound)  ·  **Default:** auto-injected for a single prompt with `n=1`; required otherwise Maximum tokens to generate **per completion**. A single prompt with `n=1` may omit it — the wrapper injects `min(32_000, remaining model context)` rather than letting vLLM fill an unbounded context. Batched prompts or `n > 1` **must** set it explicitly because the total worst-case output work is capped: > **`prompt_count × n × max_tokens ≤ 32_000`**  —  exceed this and the request returns `400 invalid_request`. > > `prompt_count` is the number of prompts the request carries — **not** a token count: `1` for a single string (or a single pre-tokenized `list[int]` prompt), the list length for a batch (`list[str]` or `list[list[int]]`). `max_tokens=0` is **not** supported. To score existing text rather than generate, use `prompt_logprobs` with `max_tokens=1` and `echo=true`; you get the prompt-position logprobs and ignore the one generated token. See the [prompt_logprobs](/tutorial/examples/prompt-logprobs) and [echo](/tutorial/examples/echo) examples. The wrapper may further reduce `max_tokens` to fit a per-key budget; when that happens the response carries an `X-Acs-Max-Tokens-Clamped` header. ### n **Type:** `int`, `1 ≤ n ≤ 16`  ·  **Default:** `1` Number of independently-sampled completions per prompt. Counts against the `prompt_count × n × max_tokens` cap above. ### stop **Type:** `string` or `string[]` (up to 4 entries)  ·  **Default:** none Strings that, when produced, stop sampling. Matched stop content is **not** included in the response. Lists longer than 4 return `400 invalid_request`. **Sampling** ### temperature **Type:** `float`, `0.0 ≤ x ≤ 100`  ·  **Default:** `1.0` Sampling temperature. `0.0` is greedy decoding (top-1 at every step) and is fully deterministic regardless of `seed`. Higher values flatten the distribution toward uniform; the effect scales as ~1/T, so it's already ~90% of the way to uniform by `10` and negligible beyond. (The Workbench slider stops at `5` — the common range — but you can pass higher here.) Pair a high temperature with `top_p` / `top_k` / `min_p` to keep the flattened sampling inside a plausible set. ### top_p **Type:** `float`, `0.0 < x ≤ 1.0`  ·  **Default:** `1.0` Nucleus sampling threshold — keep the smallest set of tokens whose cumulative probability reaches `top_p`. `1.0` disables. Must be strictly positive (`0.0` returns `400`). ### top_k **Type:** `int`, `-1` (disabled) or `n ≥ 1`  ·  **Default:** `-1` Keep only the top-*k* tokens at each step. `-1` disables. > **vLLM-native:** with the OpenAI Python SDK, pass via `extra_body={"top_k": ...}`. ### min_p **Type:** `float`, `0.0 ≤ x ≤ 1.0`  ·  **Default:** `0.0` Minimum probability — relative to the top token — a token must have to be sampled. `0.0` disables. > **vLLM-native:** with the OpenAI Python SDK, pass via `extra_body={"min_p": ...}`. **Penalties** ### presence_penalty **Type:** `float`, `-2.0 ≤ x ≤ 2.0`  ·  **Default:** `0.0` Penalty applied once a token has appeared anywhere in the generation so far. Positive values push the model away from repeating any token; negative encourage it. ### frequency_penalty **Type:** `float`, `-2.0 ≤ x ≤ 2.0`  ·  **Default:** `0.0` Like `presence_penalty`, but scaled by how often the token has already appeared. ### repetition_penalty **Type:** `float`, `0.0 < x ≤ 2.0`  ·  **Default:** `1.0` Multiplicative repetition penalty (vLLM-native, distinct from the additive presence / frequency penalties). `1.0` disables; values above `1.0` discourage repeats; values below `1.0` encourage them. Must be strictly positive. > **vLLM-native:** with the OpenAI Python SDK, pass via `extra_body={"repetition_penalty": ...}`. **Determinism** ### seed **Type:** `int`, `n ≥ 0`  ·  **Default:** none (random) PRNG seed for sampling. Honored — a fixed seed with identical params on the same underlying checkpoint gives a reproducible draw. Greedy decoding (`temperature=0`) is deterministic without `seed`. **Token-level inspection** ### logprobs **Type:** `int`, `0 ≤ n ≤ 100`  ·  **Default:** none Top-*k* logprobs **per generated token**. `0` / unset disables. The per-model cap is exposed in `GET /v1/models` as `capabilities.max_logprobs` — read that rather than hard-coding `100`. See the [logprobs example](/tutorial/examples/logprobs). ### prompt_logprobs **Type:** `int`, `n = -1` or `0 ≤ n ≤ 100`  ·  **Default:** none Top-*k* logprobs at each **prompt** position — the model's predictions over the prompt, not the actual prompt tokens unless they happened to be in the top *k*. Useful for likelihood / surprisal / interpretability work. See the [prompt_logprobs example](/tutorial/examples/prompt-logprobs). Pass **`-1` for the full vocabulary** — the model's entire next-token distribution at every prompt position, not just the top *k*. This is prompt-only (completion `logprobs` can't be `-1`) and, because the payload is ~`vocab_size` values *per token* (tens of MB), prompt length is capped — see `capabilities.full_vocab_max_prompt_tokens` in `GET /v1/models`, currently **1024 tokens** for every model; a longer full-vocab prompt returns `400 full_vocab_prompt_too_long`. For longer prompts, use a fixed top-*k*. Send `Accept-Encoding: gzip` (most HTTP clients do by default) — the wrapper compresses the response, which shrinks these logprobs bodies several-fold. Expect a large download either way, and note the response streams: the first bytes can take a while on long prompts while the model server serializes the body. > **vLLM-native:** with the OpenAI Python SDK, pass via `extra_body={"prompt_logprobs": ...}`. ### echo **Type:** `bool`  ·  **Default:** `false` If `true`, the prompt is prepended to `choices[i].text` (no separator). See the [echo example](/tutorial/examples/echo). **Streaming** ### stream **Type:** `bool`  ·  **Default:** `false` If `true`, the response is `text/event-stream` (SSE) with one chunk per token, terminated by `data: [DONE]`. See the [stream example](/tutorial/examples/stream). ### stream_options **Type:** `object`  ·  **Default:** none Optional streaming flags. Currently honored: - `include_usage: bool` — emit a terminal chunk with `choices: []` and a populated `usage` object **before** `[DONE]`. Without this, streamed responses skip the usage frame. **Bookkeeping** ### user **Type:** `string`  ·  **Default:** none Free-form tag echoed back for your own bookkeeping (matches the OpenAI shape). Telemetry only — does not affect routing, scheduling, or pricing. ## What's not supported - **No chat endpoint.** `POST /v1/chat/completions` returns `400 chat_completions_unsupported` — these are base models, no chat template. The `openai` SDK defaults to chat, so call `client.completions.create(...)` (not `client.chat.completions.create(...)`). ## Returns A `text_completion` object (JSON for non-streaming; SSE frames for `stream: true`): ```json { "id": "cmpl-91ea94c6ed0e5b75", "object": "text_completion", "model": "meta-llama/Llama-3.1-8B", "choices": [ { "index": 0, "text": " a", "logprobs": { "text_offset": [0], "tokens": [" a"], "token_logprobs": [-1.7437], "top_logprobs": [ {" a": -1.7437, " Paris": -1.9937, " one": -2.5562, " the": -2.5562, " also": -3.1187} ] }, "finish_reason": "length" } ], "usage": {"prompt_tokens": 6, "completion_tokens": 1, "total_tokens": 7} } ``` **Top-level** - `id` — unique per response. Distinct from the `X-Request-Id` response header (which is the wrapper's trace id — quote that one in bug reports). - `object` — always `"text_completion"`. - `model` — the **checkpoint** the backend served (e.g. `"meta-llama/Llama-3.1-8B"`), not the short id you passed. Use this for reproducibility; the short id is a routing alias and can re-point. - `choices` — array of length `prompt_count × n`, ordered first by prompt then by fan-out index. - `usage` — `prompt_tokens`, `completion_tokens`, `total_tokens`. Streaming responses include `usage` **only** if you set `stream_options.include_usage: true`. **Per choice** - `text` — the generated continuation (or prompt + continuation, if `echo: true`). - `index` — the choice's position in the response. - `finish_reason` — `"stop"` (a stop string matched), `"length"` (hit `max_tokens` or the model's context limit), or `null` mid-stream. - `stop_reason` — the matched stop string when `finish_reason == "stop"`, else `null`. vLLM-specific; omitted from the abbreviated example above but present on the wire. - `logprobs` — `null` unless you set `logprobs`. When present: - `tokens[]` — generated token strings. - `token_logprobs[]` — logprob of each generated token. - `top_logprobs[]` — array of `{token: logprob}` dicts, one per position; each dict's size is the `logprobs` value you asked for. - `text_offset[]` — byte offsets of each token in `text`. When you set `prompt_logprobs`, vLLM surfaces it under `choices[i].prompt_logprobs` — see the [prompt_logprobs example](/tutorial/examples/prompt-logprobs) for the full shape. > **Responses are passed through verbatim from vLLM.** The wrapper validates request bodies strictly but does not re-shape responses, so `logprobs`, `prompt_logprobs`, and token ids surface exactly as the engine produced them. ## Limits - Each key has a **monthly token budget** (`prompt + completion` tokens, reset on the 1st, UTC). Exceed it and requests return `429 budget_exceeded`. [Email](mailto:infra@acsresearch.org) to raise it. - Up to **16 active requests per key**; another **64 may wait server-side**. Beyond that, requests return retryable `429 queue_full` rather than consuming unbounded memory. - Each key may start up to **600 requests/minute**. Sustained floods above that return `429 rate_limited`. - A single request may ask for at most **32,000 worst-case output tokens**, calculated as `prompt_count × n × max_tokens` (`prompt_count` = prompts in the request: 1 for a single prompt, the list length for a batch). Batched prompts and `n > 1` must set `max_tokens` explicitly. ## Errors Every error body follows the OpenAI shape: `{"error": {"code": "...", "message": "...", "type": "..."}}` with extra fields tagged where useful. Read `error.code` for programmatic handling.
HTTPCodeMeaningWhat to do
400invalid_requestUnknown field, bad type, or out-of-range sampling param (e.g. temperature>2, top_p>1, logprobs>100)The message names the offending field; fix and retry
400invalid_requestprompt_count × n × max_tokens > 32,000, or a fan-out request omitted max_tokensReduce the prompt batch, n, or max_tokens
400context_length_exceededprompt_tokens + max_tokens > max_model_lenReduce the prompt or max_tokens; check /v1/models for the per-model limit
400chat_completions_unsupportedYou hit /v1/chat/completionsUse /v1/completions — these are base models, no chat template
400model_not_foundUnknown model idUse a short id from /v1/models (not the HF repo name)
400bad_jsonRequest body wasn't valid JSONFix the JSON
401invalid_api_keyKey missing, wrong, paused, or revokedCheck ACS_API_KEY in your account settings
429budget_exceededMonthly / daily / input / output token budget hitWait for the reset, or ask for more. See Budget-cap recovery
429queue_fullThis key already has 8 active + 64 queued requestsHonor Retry-After, reduce client fan-out, and use a client-side Semaphore(8)
429rate_limitedThis key exceeded 600 request starts/minuteHonor Retry-After and reduce sustained request rate
502upstream_unreachableWrapper couldn't reach the model server (DNS / connection / read timeout) after retriesRetry shortly; persistent failures are an outage — report it
502vllm_oomUpstream model server ran out of GPU memoryRetry with a smaller prompt / max_tokens / lower n
502vllm_context_lengthUpstream enforced its context-length limit (rare — the wrapper usually catches this as context_length_exceeded first)Reduce prompt / max_tokens
502vllm_engine_deadUpstream vLLM engine crashedRetry; if it persists the model is down — check status or report
502upstream_server_errorOther upstream 5xx after retriesRetry; check error.upstream_status for the original code
200*modal_cold_bootThe model didn't become ready before the 14-minute safety deadline. The 200 status was already sent (keepalive bytes had committed it), so the error arrives in the response body, not as an HTTP error code.Treat the JSON/SSE error payload as a failed request; report repeated allocation failures. Normal cold boots complete in the original request. See Cold-boot waiting
503circuit_openBackend is in a circuit-breaker open state after repeated failuresUse retry_after_seconds; if the model is critical, contact us
## Request headers - `Authorization: Bearer ` — required. - `Content-Type: application/json` — for the JSON body. - `X-Acs-Workload: interactive | batch` — **please set this on your requests**: `batch` for bulk / offline / eval jobs, `interactive` for live, latency-sensitive calls. It's telemetry only — it does **not** change scheduling or priority, and unrecognised values are ignored — but it's the main signal we use to understand usage and plan capacity / keep-warm policy, so setting it accurately genuinely helps us run the service well. See [Batch rollouts](/tutorial/examples/batch-rollouts). ## Response headers Reliability hints on the response itself: - `X-Request-Id` — unique trace id for the request; quote it in bug reports so we can find it in our logs. - `X-Acs-Upstream-Model` / `X-Acs-Upstream-Gpu` — which backend served (or failed) this request. Cite this in bug reports. - `X-Acs-Upstream-Error-Kind` — set on upstream-error 5xx responses (mirrors `error.code`). - `X-Acs-Max-Tokens-Clamped` — present when the wrapper reduced your `max_tokens` to fit a budget; format `requested=N,applied=M,reason=budget`. - `Retry-After` — RFC-7231 header on retryable `429` and `503` responses. ## Privacy API requests to `/v1/completions` are logged as **metadata only** — key prefix, email, IP, endpoint, model, token counts, status, latency — which we use to run the service and prevent abuse. We don't log your prompts or completions. Note this might change: we may start logging API requests for abuse-prevention reasons. We will not train on them. (Keep your own copies if you need them for reproducibility.) The one exception is the Workbench: prompts you save there are stored server-side, so your sessions persist across browsers. Don't keep anything there you wouldn't want stored. ## Getting help Email [infra@acsresearch.org](mailto:infra@acsresearch.org), or use the in-app **Feedback** button if you're signed in. To help us trace a specific request, include the rough time you made it, your key prefix or name (found in your dashboard — *not* the secret), and the error body. # Activation harvesting Get a model's **internal residual-stream activations** back alongside a normal completion — the values flowing between the model's layers, for interpretability and probing work (SAEs, linear probes, logit lens, …). It's the same `POST https://infra.acsresearch.org/v1/completions` endpoint and the same API key as ordinary completions — you just add one field. Activation requests run on a **separate engine** per model, kept apart from ordinary completions. One consequence for seeded work: a seed reproduces a draw *within* an engine, so don't use a plain completion as a control for an activation-engine request — see [Baselines on the steering page](/tutorial/activation-steering) for the clean recipe. ## Turn it on Add **`"output_residual_stream": true`** to the request body: ```bash curl -s "$ACS_API_BASE/completions" \ -H "Authorization: Bearer $ACS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "llama-8b", "prompt": "The capital of France is", "max_tokens": 1, "temperature": 0, "output_residual_stream": true }' ``` `true` captures the residual stream at **every** decoder layer. If you only need a few layers, pass a **list of layer indices** instead — the subset is applied on the engine, so only those layers cross the wire (much smaller responses): ```jsonc "output_residual_stream": [15, 20] // capture only layers 15 and 20 ``` `n_layers` in the returned `shape` is then the number of layers you asked for (in ascending order), not the model's full count. Pass `true` for all layers. ## What comes back The normal completion JSON, plus a top-level **`activations`** object: ```jsonc { "choices": [ /* the normal completion */ ], "usage": { /* the normal usage */ }, "activations": { "residual_stream": { "data": "", // zstd-compressed, int16-encoded bf16 bytes "dtype": "int16", // storage dtype (bf16 reinterpreted as int16) "original_dtype": "torch.bfloat16", "shape": [n_layers, n_tokens, hidden], // n_layers = full count for `true`, else how many you requested "compression": "zstd" } } } ``` `shape[0]` is the model's **full** layer count (e.g. 32 for llama-8b, 126 for llama-405b, 60 for trinity-truebase). Slice the layers you want after decoding. ## Which token positions are captured `shape[1]` covers every token that went through a forward pass: ``` n_tokens = prompt tokens + generated tokens − 1 ``` A 6-token prompt with `max_tokens: 5` returns 10 positions: rows 0–5 are the prompt, rows 6–9 are the generated tokens that were fed back through the model. The **final sampled token is not included** — it's drawn from the last forward pass but never processed itself, so it has no residual stream. With `max_tokens: 1` (the probing default, as in the example above) that reduces to exactly the prompt positions. So a single request captures most of the generation trajectory too, not just the prompt — useful when you want activations *while the model writes*, at the cost of a larger payload (the prompt-length cap below applies to the prompt). ## Decode it `dtype: "int16"` holds the bfloat16 **bit pattern** stored as int16. Reinterpret those bits back to bfloat16 with `.view(...)`, as the code below does — a numeric `.astype()` would convert the integer values and give you garbage: ```python import base64, numpy as np, ml_dtypes, zstandard as zstd def decode_residual_stream(resp): d = resp["activations"]["residual_stream"] raw = base64.b64decode(d["data"]) if d.get("compression") == "zstd": raw = zstd.ZstdDecompressor().decompress(raw) # reinterpret the 16 bits as bfloat16 with .view (an .astype would corrupt them) arr = np.frombuffer(raw, dtype=np.uint16).view(ml_dtypes.bfloat16) return arr.reshape(d["shape"]) # (n_layers, n_tokens, hidden) x = decode_residual_stream(resp) # e.g. shape (32, 6, 4096) for llama-8b layer16 = x[16] # slice the layer(s) you want, client-side ``` ## Always assert the shape A capture hook that fails is skipped **server-side with only a log warning**, so a partial capture can come back *looking* like valid data. Assert before you trust it: ```python x = decode_residual_stream(resp) assert x.ndim == 3, x.shape assert x.shape[0] == n_layers_for_model, f"partial capture: {x.shape[0]} layers" assert np.isfinite(x.astype(np.float32)).all(), "non-finite — broken capture" ``` ## Which models support this? All three models we serve support harvesting (and steering): **llama-8b**, **llama-405b**, and **trinity-truebase**. ## Limits & practicalities - **Prompt-length cap.** Because capture returns *every* layer, the response can be large — a big model at 128 tokens is hundreds of MB. Each model advertises a **`max_activation_prompt_tokens`** cap under its `capabilities` in `/v1/models`; a longer capture prompt gets a `400 activation_prompt_too_long`. Caps are tight on the largest models (~4 MB per token). **This inline endpoint is meant for probing.** It comfortably handles up to ~10k examples; for corpus-scale harvesting, use [Bulk harvest](/tutorial/bulk-harvest) — a batch job that writes shards you download by URL. - **Cold boot.** All activation engines scale to zero when idle. llama-8b restores from a GPU snapshot, so its first request after an idle gap takes ~30–40 s; everything after that is interactive. The big engines (llama-405b, trinity-truebase) rebuild from scratch — several minutes on the 8×H200 models. The wrapper holds that request open and sends keepalive bytes while the engine starts, so it normally completes in one shot — set a client timeout of at least 15 minutes rather than retrying. If the engine can't come up (e.g. the 14-minute safety deadline passes), you get a retryable `modal_cold_boot` error — note it can arrive inside a `200` response body once keepalive bytes have committed the status, so check the body for `error`, don't rely on the HTTP code alone. See [Cold-boot waiting](/tutorial/examples/cold-boot). - **Usage & quota.** Activation requests are attributed to your key and may count against a per-key monthly activation quota (a heavier operation than a plain completion); over quota returns `429 activation_quota_exceeded`. ## See also - [Bulk harvest](/tutorial/bulk-harvest) — the batch pipeline for corpus-scale harvesting (submit a job, download shards). - [Activation steering](/tutorial/activation-steering) — inject your own direction into the residual stream. - [API reference](/tutorial/api) — the base `/v1/completions` contract. # Bulk harvest Harvest **residual-stream activations over a whole prompt corpus** as downloadable files — the batch counterpart to [inline harvesting](/tutorial/activation-harvesting), for when you want thousands to millions of prompts (SAE training, large probing datasets) rather than a few activations in an HTTP response. You submit a job, poll it, and download `safetensors` shards by URL. Same API key as everything else. The GPU work runs on a dedicated batch app per model; nothing you do here slows the completions engines. ## Submit a job `POST https://infra.acsresearch.org/v1/harvest` with your prompts: ```bash curl -s "$ACS_API_BASE/harvest" \ -H "Authorization: Bearer $ACS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "llama-8b", "prompts": ["The quick brown fox jumps over the lazy dog.", "Interpretability research reads the residual stream."], "layers": [8, 16, 24] }' ``` ```jsonc { "job_id": "071fa547-…", "status": "running", "model": "llama-8b", "n_prompts": 2 } ``` (Responses below are abridged — a real one carries a few more bookkeeping keys like `run_id` and `created_at`.) Fields beyond `model` and `prompts` (a typo'd field name returns a `400` rather than silently using a default on a paid GPU job): - **`layers`** — which decoder-block outputs to keep, e.g. `[8, 16, 24]`. Omit it for the default quartile subset (blocks at ~25/50/75% depth — `[8, 16, 24]` on llama-8b); pass the string `"all"` (lowercase, exact) to keep every block. Unlike the inline endpoint, the harvester subsets **server-side**, so the 3-layer default on a 32-layer model writes ~10× less data than `"all"`. Same convention as inline: layer `k` is the output of block `k`. - **`shard_size`** — prompts per output file (default 32). - **`batch_size`** — forward-pass batch inside the job. Leave it unset: the default vLLM-Lens backend batches continuously on its own; the value only shapes the HF reference path (`HARVEST_USE_VLLM=0`). ## Poll it `GET https://infra.acsresearch.org/v1/harvest/` with the same key. `status` moves from `running` to `done` (or `failed`, with a short error message): ```bash curl -s "$ACS_API_BASE/harvest/071fa547-…" -H "Authorization: Bearer $ACS_API_KEY" ``` ```jsonc { "job_id": "071fa547-…", "status": "done", "model": "llama-8b", "n_prompts": 2, "result": { "n_shards": 1, "layer_indices": [8, 16, 24], "manifest_url": "https://…/manifest.json?…", // presigned — no key needed "shard_urls": ["https://…/shard_00000.safetensors?…"], "stats_url": "https://…/stats.safetensors?…", "timings": { "forward_s": 1.59, "upload_s": 4.05, "tokens": 21, /* … */ } }, "urls_expire_at": "2026-07-24T20:31:37+00:00" } ``` A small llama-8b job goes submit → `done` in about a minute, most of it engine start-up. Poll every 20–30 s; there's no webhook. ## Download and read the shards The URLs are presigned and work in a plain browser, `curl`, or `wget` — no API key. They expire at `urls_expire_at` (7 days after completion; the response adds `"urls_expired": true` once they have). Each shard holds two tensors per prompt — the activations and the **input token ids**, so you can line activations up with tokens without re-tokenizing: ```python import os, requests from safetensors.torch import load_file API_BASE = os.environ["ACS_API_BASE"] # e.g. https://…/v1 KEY = os.environ["ACS_API_KEY"] job = requests.get(f"{API_BASE}/harvest/{job_id}", headers={"Authorization": f"Bearer {KEY}"}).json() open("shard0.safetensors", "wb").write(requests.get(job["result"]["shard_urls"][0]).content) shard = load_file("shard0.safetensors") shard["prompt_0"] # bf16, [n_layers_kept, n_tokens, hidden] — e.g. [3, 11, 4096] shard["tokens_0"] # int32, [n_tokens] — same order as the tensor's token axis ``` `manifest_url` points to a JSON index of every shard (which prompts are in which file, shapes, the layer convention spelled out). `stats_url` is a small safetensors file with per-layer token `mean` and `std` over the whole run — the normalization statistics SAE training wants, with each prompt's BOS token left out of the statistics because its outlier norms would skew them. ## Limits & practicalities - **One running job per key (the default).** A second submit while one is running returns `429 harvest_concurrency_exceeded` — poll the first to completion. Your key may also carry a monthly job quota (`429 harvest_quota_exceeded`). - **Corpus caps.** Up to 4,096 prompts and ~2M estimated tokens per job, and request bodies up to 50 MB (`413` beyond that). For more, split into several jobs. - **Prompts longer than the model's context window are truncated** to fit, and the token axis you get back matches what actually ran. - **Big models are slow to start.** llama-405b streams ~810 GB of weights on a cold start — the first job can spend ~50 minutes loading before any prompt runs. The load is paid once per job, so batch your corpus into one large job instead of many small ones. - **`503 harvest_unavailable`** means the harvest app for that model isn't reachable right now; the response records a job id you can show us. ## Inline or bulk? Inline (`output_residual_stream`) answers in the same HTTP response and is right for probing and steering workflows up to ~10k examples. Bulk trades latency for throughput: the capture itself runs ~13× faster than the inline path's transport ceiling, keeps only the layers you ask for, and hands you files you can re-download for a week. ## See also - [Activation harvesting](/tutorial/activation-harvesting) — the inline path. - [Activation steering](/tutorial/activation-steering) — steer with vectors you build from harvested activations. # Activation steering Nudge a model's behaviour at inference time by **adding a direction to its residual stream** — the same mechanism behind "concept vectors" / representation engineering. You supply one or more steering vectors; the engine adds each to the hidden state at the layer(s) you name, every forward pass, and generation continues from the perturbed state. Same `POST https://infra.acsresearch.org/v1/completions` endpoint and key as a normal completion — you add an **`apply_steering_vectors`** field. Steering runs on the same activation engine as [harvesting](/tutorial/activation-harvesting). ## The shape of a request ```jsonc { "model": "llama-8b", "prompt": "I think that", "max_tokens": 32, "apply_steering_vectors": [ { "activations": { /* the vector, in the codec below */ }, "layer_indices": [16], // which layer(s) to add it at "scale": 8.0, // multiplier on the vector "norm_match": false, // rescale to the hidden state's norm first? "position_indices": null // null = all positions; or a list of token positions } ] } ``` You can send up to **8** vectors in one request; each is applied independently. ### The `activations` codec The vector is encoded the **same way as harvested activations** — bfloat16 values reinterpreted as int16, zstd-compressed, then base64-encoded: ```python import base64, numpy as np, ml_dtypes, zstandard as zstd def encode_vector(vec: np.ndarray) -> dict: """vec: float array with ONE ROW PER STEERED LAYER. (n_layers, hidden) -> each layer's vector broadcast to all positions (n_layers, n_positions, hidden) -> position-specific n_layers must equal len(layer_indices). """ a = vec.astype(ml_dtypes.bfloat16).view(np.int16) # bf16 bits as int16 packed = zstd.ZstdCompressor(level=1).compress(a.tobytes()) return { "data": base64.b64encode(packed).decode(), "dtype": "int16", "original_dtype": "torch.bfloat16", "shape": list(a.shape), "compression": "zstd", } ``` **The leading axis is layers.** `shape[0]` must equal `len(layer_indices)` — one direction per layer you're steering (a mismatch returns `400`). So: - steer **one** layer → `layer_indices: [16]`, vector `(1, hidden)` - steer **two** layers → `layer_indices: [16, 20]`, vector `(2, hidden)` (row 0 → layer 16, row 1 → layer 20) - target **specific token positions** → make it 3-D `(n_layers, n_positions, hidden)` and set `position_indices` `hidden` is the model's width — **4096** (llama-8b), **16384** (llama-405b), **3072** (trinity-truebase); a mismatched `hidden` is also rejected with `400`. ## A full example ```python import base64, json, numpy as np, ml_dtypes, zstandard as zstd, urllib.request BASE, KEY = "$ACS_API_BASE", "$ACS_API_KEY" def post(body): req = urllib.request.Request(f"{BASE}/completions", data=json.dumps(body).encode(), headers={"Authorization": f"Bearer {KEY}", "Content-Type": "application/json"}) out = json.load(urllib.request.urlopen(req, timeout=300)) if "error" in out: # late cold-boot failures arrive inside a 200 body raise RuntimeError(out["error"].get("message", str(out["error"]))) return out # In practice this vector is a learned/derived direction (a probe weight, a # difference-of-means between two prompt sets, an SAE decoder column, …). vec = my_direction.reshape(1, 4096) # (n_layers=1, hidden) — one row for layer 16 sv = {"activations": encode_vector(vec), "layer_indices": [16], "scale": 8.0, "norm_match": False, "position_indices": None} gen = {"model": "llama-8b", "prompt": "I think that", "max_tokens": 32, "temperature": 0} print("baseline:", post(gen)["choices"][0]["text"]) print("steered :", post({**gen, "apply_steering_vectors": [sv]})["choices"][0]["text"]) ``` Sanity check while wiring it up: **`scale: 0.0` is the identity** — a zero-scale steer must reproduce the baseline text exactly. If it doesn't, your request isn't reaching the steering path. ## Field reference | Field | Meaning | |---|---| | `activations` | The direction(s), in the codec above — **one row per steered layer**; `shape[0]` must equal `len(layer_indices)`. | | `layer_indices` | Decoder layer(s) to steer (0-indexed). Row *i* of `activations` is added at `layer_indices[i]`. | | `scale` | Multiplier. Bigger = stronger effect; too big collapses into gibberish/repetition. | | `norm_match` | If `true`, the vector is rescaled to the hidden state's norm at each position *before* `scale` — makes a good `scale` transfer across layers/prompts. If `false`, the raw vector is used. | | `position_indices` | `null` steers every token position; a list steers only those positions (e.g. `[−1]` for the last token). Requires a 3-D `activations` shape with `shape[1] == len(position_indices)` — combined with a 2-D vector the engine would ignore it, so that returns `400`. | ## Which models support this? All three models we serve support steering (and harvesting): **llama-8b**, **llama-405b**, and **trinity-truebase**. ## Practicalities - **One sequence per steered request:** steering requires a single prompt and `n: 1`; a batched (list-valued) prompt or `n > 1` returns `400`. The engine applies vectors to the first sequence of a request only, so before this guard those shapes came back part-steered with a `200`. To fan out, send separate requests (vary `seed` for sampled diversity). - **Cold boot & quota:** same as harvesting — llama-8b wakes from a GPU snapshot in ~30–40 s after an idle gap; the big engines (llama-405b, trinity-truebase) cold-boot for several minutes on the first request after idle, and the wrapper holds that request open through the boot (set a client timeout of at least 15 minutes — see [Cold-boot waiting](/tutorial/examples/cold-boot)). Requests are attributed to your key against the activation quota. - **Baselines: same seed ≠ same engine.** Ordinary completions and activation/steering requests run on **separate engines** per model. A seed reproduces a draw *within* an engine, so comparing a steered request against a same-seed plain completion is not a valid control — the two engines sample different streams. Run the control on the activation engine too: a `scale: 0.0` steering vector is an exact identity (verified — it reproduces the same-engine unsteered output bit-for-bit), so `{**req, "apply_steering_vectors": [{**sv, "scale": 0.0}]}` is the clean baseline. - **Finding a good `scale`:** start small and sweep up. Two things interact — the `scale` **and how many layers you steer**: pushing a wide band compounds fast. The worked example below (a single mid layer, raw vector) has its sweet spot around `1.5–2`; with `norm_match: true` over a small band, useful scales are typically below `1`. Watch for the output degrading into repetition — that's the sign you've overshot. ## Worked example: a "pirate" contrast vector A full end-to-end demonstration — done entirely through this API — lives as a **marimo notebook** at [`pirate_steering.py`](/examples/pirate_steering.py) (run: `ACS_API_KEY=… uvx marimo edit --sandbox --watch pirate_steering.py`), with a rendered run **including all cell outputs** at [`pirate_steering.html`](/examples/pirate_steering.html) — readable without an API key. It: 1. **harvests** residual-stream activations for six pirate-styled sentences and six plain paraphrases (one capture each; mean over content tokens, BOS position excluded), 2. **builds** the direction: `mean(pirate) − mean(plain)`, per layer — contrasting **actual text in the target style**, 3. **steers** layer 16 of `llama-8b` with the raw vector, sweeping `scale`, and 4. **scores** the effect two ways — a pirate-word count and a logprob preference metric (`echo` + `prompt_logprobs`, shared prefix cancelling). The measured dose–response (temperature 0.8, pinned seed): | scale | result | |---|---| | `0` | ordinary completions — verified **identical** to a same-engine unsteered baseline | | `1.5` | *"**Arr, me hearties**, 'tis a fair day for ye walk the plank, **matey**!"* — coherent pirate | | `2.0` | the weather forecast: *"…I'm **hoisting the main 'n jib** and **shiver me arrgh**! we'll be a-fer shore by 3 bells!"* | | `≥3` | degenerates into repetition — the expected overshoot | The same vector **generalizes** to new prompts: at `2.0`, job-day advice becomes *"go in with a **swashbucklin'** attitude and a **bountiful chest o' booty**, **Mateys**!"* — the direction carries the style itself, so it transfers. ## See also - [Activation harvesting](/tutorial/activation-harvesting) — capture the residual stream (and derive steering directions from it). - [API reference](/tutorial/api) — the base `/v1/completions` contract. # Account Where to find your keys, your spend, and your monthly budget. ## API keys You create keys from the Dashboard once your account is approved. Each key is shown **once** at creation time — copy it then; we only ever store a hash. If you lose it, revoke the key and create a new one. Keys can be **paused** (reversibly disabled) or **revoked** (permanent). Paused keys come back with one click; revoked keys are gone. ## Budgets A request is checked against several token budgets before it runs. Any one of them can refuse it — and the `429 budget_exceeded` message **names the dimension that was hit**, so you know which cap to ask us to raise. - **Per-key monthly budget** — a token cap (`prompt + completion`) that resets on the 1st (UTC). You can set this per key when you create it; "unbounded" means no per-key sub-cap (the account-level total still applies). - **Account total (monthly)** — an aggregate cap across all your keys. Set by an admin; visible on the dashboard. A per-key cap can't exceed the account total. On top of those two totals, a key can carry optional **sub-budgets** that scope the cap more tightly. Each is opt-in — when unset, only the monthly totals above apply: - **Daily budget** — a per-key `prompt + completion` ceiling that resets every day (UTC), for pacing a key across a longer run rather than letting it burn the whole month in one batch. - **Monthly input-token budget** — a per-key cap on **prompt** tokens only. - **Monthly output-token budget** — a per-key cap on **completion** tokens only. When a request would exceed **any** of these, it returns `429 budget_exceeded` with a message naming the dimension that was hit. [Budget-cap recovery](/tutorial/examples/budget-cap) shows how to handle it in code — the short version: don't retry. Monthly, input, and output caps are sticky until the 1st; the daily cap until tomorrow (UTC). ## Where things live - **Dashboard** — keys, this-month's spend per key, create / pause / revoke buttons. - **Usage** — usage chart with per-model breakdown for the current month, plus the running totals you'll need for budget planning. - **Workbench** — in-browser prompt UI; saved prompts persist across browsers. Once you're signed in, all three pages are linked from the top nav. ## Asking for more If you need a bigger budget, an always-on warm window, or access to a model that isn't on the list — use the in-app **Feedback** button or [email infra@acsresearch.org](mailto:infra@acsresearch.org). We read everything; please include your key prefix or name so we can find your account. # Workbench The Workbench lets you prompt the base models straight from your browser — no code. Pick a model, type a raw prompt, and read the completion, with the same honored sampling controls as the API. Two power-user views sit on top of the single-pane chat, built for base-model exploration: - [**Loom**](/tutorial/workbench/loom) — branch a prompt into a *tree* of completions: generate several continuations at once, pick one and keep going, backtrack to any point, and inspect per-token logprobs. - [**Compare mode**](/tutorial/workbench/compare) — run one prompt across several models (or the same model at different settings) *side by side*. **Loom** is a first-class surface of its own — open it from the **Loom** link in the top nav. **Compare mode** lives inside any Workbench chat: flip the **Single / Compare** toggle in the composer. Both are private to your account. # Loom A **loom** is a tree view for exploring what a base model might say. Instead of one linear completion, you generate several branches at a point, pick the one you like, keep going, and backtrack whenever you want — keeping the whole "multiverse" as a navigable tree. Per-token logprobs are shown inline so you can see how confident the model was at each step. Open **Loom** from the top nav. It lists your saved looms — click **＋ New loom** to start one, or open a recent one. Each loom is its own saved object (private to you), so the tree is still there when you come back. ## 1. Set a root Type a starting prompt in the text box and click **Set root (replaces tree)**. The root is your prompt verbatim — no generation happens yet. (Re-setting the root clears the current tree, so you can start over cleanly.) ## 2. Branch Select a node, then click **Branch from here**. The controls above it: - **n** — how many completions to generate at once. Each becomes a sibling branch under the selected node. - **max tokens** and **temperature** — the usual sampling controls (base-model defaults: `temperature` 1.0). - **logprobs** — how many alternatives to record per token. This drives the heatmap (below); set it to `0` to skip. - **seed** — leave it `auto` for fresh sampling, or pin a number to make a branch reproducible. One click sends a **single** request for `n` completions; each non-empty one is added as a child of the selected node and streams in live. ## 3. Navigate the tree - The **sidebar** shows the whole tree. Click any node to select it; the centre panel shows the full text from the root down to that node. - Switch to the **Graph** view (the *List / Graph* toggle above the tree) for a graphical map of the whole "multiverse": one box per node, branching left→right, with the current path highlighted. **Drag** to pan, **scroll** (or the +/− buttons) to zoom, and **click** a node to select it — the node panel and branch controls work exactly as in the list view. Your choice of view is remembered. - **Branch from** any node to explore a new direction. To **backtrack**, just select an earlier node and branch again — the original branches stay put. - Delete a node (and everything under it) with the delete control on the node. ## 4. Edit, split, and hand-write branches Looms are meant to be driven at speed. With a node selected, the panel shows three actions (and matching hotkeys): - **Edit** (`e`) — change a node's text in place. Click **Save edit** to keep it. Editing a node re-uses its stored context, so its children continue from your edited text. (Because the text changed, the per-token heatmap for that node is dropped — logprobs only line up with the text the model actually produced.) - **Split at cursor** (`s`) — the signature loom move. Cut the selected node in two at the cursor: the node keeps the text before the cursor, and a new child holds the text after it. Any children the node already had are moved onto that new child, so every deeper branch keeps its exact context. Splitting is how you insert a fork *inside* an existing completion. Pressing **Split at cursor** (or `s`) opens the edit box so you can place the cursor exactly, then click **Split here** to cut at that point — the edited text and the split commit together in one step. - **New sibling** (`i`) — add a hand-written alternative continuation next to the selected node (same parent), then type into it. Only works on non-root nodes — to start from a different seed, make a new loom. **Save vs. new sibling — the convention:** editing **mutates the node in place** (matching the reference loom's "edit node"); it does not fork. To keep the original *and* an alternative, use **New sibling** (or generate more branches). ## Hotkeys Press **?** any time for the full legend. While your cursor is in a text box, letter keys type normally — shortcuts only fire when no field is focused. | Key | Action | | --- | --- | | `↑` / `↓` | Previous / next sibling | | `←` | Go to parent | | `→` | Go to first child | | `g` | Generate branches from the selected node | | `e` | Edit the node's text | | `s` | Split the node at the cursor | | `i` | Insert a new sibling | | `Delete` | Delete the node and its subtree | | `Esc` | Cancel an edit / close the legend | | `?` | Toggle the shortcut legend | ## Export / import Buttons sit on the top view bar: - **Export JSON** — download the whole tree as a JSON file (topology + text + logprobs), for backup or sharing. - **Export text** — download a plain-text transcript of the *current path* (root → selected node), i.e. the full continuation you're reading. - **Import JSON** — pick a previously exported JSON file to load it into a **brand-new loom**. Node IDs are regenerated on import, so importing never touches or collides with an existing loom. ## 5. Read the logprobs When a generated node is selected, its tokens are colour-coded by probability — **green = the model was confident, red = surprising**. **Click a token** to see the top-*k* alternatives it considered and their probabilities. (The root, being your own text, isn't coloured.) ## 6. Reproduce a branch Pin a **seed** before branching, and the same model + prompt + seed will reproduce that exact continuation — handy for sharing a path or debugging a specific generation. Leave the seed on `auto` to let each branch sample freshly (so `n` branches diverge). # Compare mode **Compare mode** runs one prompt across several **lanes** at once — each lane its own model and sampling config — and streams the results side by side. It's the quick way to answer "how do these two models continue the same prompt?" or "what does `temperature` 0.2 vs 1.2 do here?". Open it from any Workbench chat: flip the **Single / Compare** toggle in the composer. (The old `…/workbench//compare` link still works — it just redirects you into the chat with Compare mode active.) ## 1. Set it up 1. Type your **prompt** once at the top — it's shared by every lane. 2. Each **lane** has its own **model** picker and full sampling config: max tokens, temperature, `top_p` / `top_k` / `min_p`, the presence / frequency / repetition penalties, `seed`, and `stop`. 3. **Duplicate** a lane to compare the same model at different settings, or add more lanes for more models (up to 6). ## 2. Run Click **Run all** to fire every lane, or the per-lane **Run** to launch just one. Lanes stream **independently and side by side** — a slow or cold lane doesn't hold up the others. **Stop all** cancels everything in flight. Running compare does **not** touch your single-pane chat history. ## 3. Saved runs Every **Run all** is saved automatically as a snapshot — no save button. A **compare-run history** list appears at the bottom of the pane, newest first — each row shows the timestamp, the shared prompt, and the lane count; expand a row to see the per-lane model chips. Click **Restore** to reload that run: the lanes, prompt, and every stored completion come straight back. The most recent 20 runs are kept per chat; older ones fall off. ## 4. Heatmap Toggle **Heatmap** on a lane (and choose how many alternatives to record) to colour each token by probability — **green = confident, red = surprising** — using the same colour ramp as the [Loom](/tutorial/workbench/loom). **Hover** a token to see the top alternatives the model considered. ## 5. Billing All lanes bill to one API key (the chat's key, or your default). Each lane is a **separate generation**, so a 4-lane run counts as 4 generations against your usage and budget. ## Tip: cold starts Always-on models respond immediately. Putting an **on-demand large model** (e.g. 405B) in a lane triggers a cold boot for *that* lane — it will sit "warming up" for a few minutes before tokens appear, while your other lanes stream normally. See [Cold-boot waiting](/tutorial/examples/cold-boot). # Examples Short, copy-pasteable end-to-end snippets for each feature. The curl examples assume `ACS_API_KEY` + `ACS_API_BASE` are exported (see [Quick start](/tutorial/overview#quick-start)); the Python examples use the same `openai` SDK client. Each page below shows one shared **example response** after the curl and Python snippets — both calls return the same JSON; the SDK just wraps it in typed objects. Always-null fields (`service_tier`, `system_fingerprint`, `kv_transfer_params`, etc.) are dropped from the shown responses for brevity. Your own output won't match token-for-token: the displayed requests don't pin a `seed`, so under the default `temperature=1.0` sampling varies. Add a `seed` to reproduce a specific run, or `temperature=0` for greedy decoding. ## Guided walkthrough - [**Hands-on Colab workshop**](https://colab.research.google.com/github/acsresearch/base-models-workshop/blob/main/workshop_student.ipynb) — the from-zero path through most of what's below, run end-to-end in your browser: completions → sampling → `logprobs` → reading activations → building a steering vector, with exercises and an instructor solutions notebook — both in [our tutorial repo](https://github.com/acsresearch/base-models-workshop). Start here if the per-feature snippets feel piecemeal. ## Token-level inspection - [`logprobs`](/tutorial/examples/logprobs) — top-*k* logprobs per generated token. - [`prompt_logprobs`](/tutorial/examples/prompt-logprobs) — top-*k* logprobs at each *prompt* position (gotchas around rank-vs-actual-token). - [`echo`](/tutorial/examples/echo) — include the prompt in the response. ## Streaming + concurrency - [`stream`](/tutorial/examples/stream) — SSE token-by-token. - [Batch rollouts](/tutorial/examples/batch-rollouts) — 16 concurrent per key, with the canonical `asyncio.gather` pattern. ## Recovery patterns - [Cold-boot waiting](/tutorial/examples/cold-boot) — keep one request open through a large-model startup. - [Budget-cap recovery](/tutorial/examples/budget-cap) — handle `429 budget_exceeded` (the non-retryable kind). ## Evaluation - [Evaluate with Inspect](/tutorial/examples/evaluate-with-inspect) — run base-model evals through Inspect's `openai-api-completions` provider (raw prompts, no chat template). # logprobs Ask for the top-*k* alternative tokens at each generated position (`k ≤ 100`). Useful for calibration, classification by next-token probabilities, or just inspecting what the model "almost said". (The exact cap is per-model — read `capabilities.max_logprobs` from `GET /v1/models` rather than hard-coding it.) ## curl ```bash curl -s "$ACS_API_BASE/completions" \ -H "Authorization: Bearer $ACS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"model": "llama-8b", "prompt": "The capital of France is", "max_tokens": 1, "logprobs": 5}' ``` ## Python ```python resp = client.completions.create( model="llama-8b", prompt="The capital of France is", max_tokens=1, logprobs=5, ) choice = resp.choices[0] print(choice.text, choice.logprobs.top_logprobs[0]) # {' a': -1.74, ' Paris': -1.99, ...} ``` ## Example response ```json { "id": "cmpl-91ea94c6ed0e5b75", "object": "text_completion", "model": "meta-llama/Llama-3.1-8B", "choices": [ { "index": 0, "text": " a", "logprobs": { "text_offset": [0], "tokens": [" a"], "token_logprobs": [-1.7437232732772827], "top_logprobs": [ { " a": -1.7437232732772827, " Paris": -1.9937232732772827, " one": -2.5562233924865723, " the": -2.5562233924865723, " also": -3.1187233924865723 } ] }, "finish_reason": "length" } ], "usage": {"prompt_tokens": 6, "completion_tokens": 1, "total_tokens": 7} } ``` Note that base `llama-8b` ranks `" a"` above `" Paris"` — it's a continuation predictor, not a Q&A assistant, so it continues the sentence rather than answering it. `" Paris"` is a close second (0.25 logprob behind), so an instruction-tuned model would likely put it first. This base model won't, even at `temperature=0` — greedy decoding just takes the top-ranked token, here `" a"`. ## Gotcha `logprobs` is an integer (top-*k*), not a boolean — passing `true` returns `400 invalid_request`. Every numeric param rejects booleans this way, so a typo like `max_tokens: true` fails loudly instead of silently becoming `1` (see [Strict validation](/tutorial/api#strict-validation)). Only `echo` and `stream` take booleans. # prompt_logprobs Return the model's top-*k* logprob predictions at each prompt position — useful for inspecting what the model expected at each step, and (with extra care, see the Gotcha) a building block for scoring how likely a given piece of text is under the model. ## curl ```bash curl -s "$ACS_API_BASE/completions" \ -H "Authorization: Bearer $ACS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"model": "llama-8b", "prompt": "The capital of France is", "max_tokens": 1, "prompt_logprobs": 5}' | jq '.choices[0].prompt_logprobs[:2]' ``` ## Python ```python resp = client.completions.create( model="llama-8b", prompt="The capital of France is", max_tokens=1, extra_body={"prompt_logprobs": 5}, ) # resp.choices[0].prompt_logprobs[i] is a {token_id: {logprob, rank, decoded_token}} # dict for prompt position i, or None for the very first position (no # conditioning context). Here we just print the top-5 at the first few positions. for i, choices in enumerate(resp.choices[0].prompt_logprobs[:3]): if choices is None: continue print(f"position {i}:") for token_id, info in choices.items(): print(f" rank={info['rank']:>2} logprob={info['logprob']:+.3f} {info['decoded_token']!r}") ``` ## Example response Just `choices[0].prompt_logprobs[:2]` (what the curl `jq` filter shows): ```json [ null, { "14924": {"logprob": -1.179518699645996, "rank": 1, "decoded_token": "Question"}, "755": {"logprob": -2.179518699645996, "rank": 2, "decoded_token": "def"}, "2": {"logprob": -2.742018699645996, "rank": 3, "decoded_token": "#"}, "791": {"logprob": -3.679518699645996, "rank": 4, "decoded_token": "The"}, "16309": {"logprob": -4.179518699645996, "rank": 5, "decoded_token": "Tags"} } ] ``` Position 0 is `null` (no conditioning context for the first token). At position 1 the model's top-1 prediction is `"Question"`, not the actual prompt token `"The"` — which here happens to show up at rank 4. If your prompt token isn't in the top-*k*, vLLM still appends it as an extra entry with `rank > k` so you can recover its logprob; see the Gotcha for how to use that to score a sequence. ## Full vocabulary (`prompt_logprobs=-1`) Pass `prompt_logprobs=-1` to get the model's **entire** next-token distribution at each prompt position, not just the top *k* — the whole `vocab_size` (~128k for Llama) of logprobs per position. This is the raw material for exact surprisal, KL between models, or full-distribution interpretability, with no truncation. A fixed top-*k* covers most uses — ranking candidates, classification by next-token probability, sequence scoring (see the Gotcha) — and works at any prompt length. Reach for `-1` when you need the tail of the distribution, and budget for the prompt-length cap below. ```bash curl -s "$ACS_API_BASE/completions" \ -H "Authorization: Bearer $ACS_API_KEY" \ -H "Content-Type: application/json" \ -H "Accept-Encoding: gzip" \ -d '{"model": "llama-8b", "prompt": "The capital of France is", "max_tokens": 1, "prompt_logprobs": -1}' ``` Two things to know: - **Prompt-only, capped prompt length.** The full distribution is enormous (~vocab_size values *per token*), so `-1` is accepted only up to `capabilities.full_vocab_max_prompt_tokens` (from `GET /v1/models`) — currently **1024 tokens** for every model, a few pages of text (this page's example prompt is 6 tokens). A longer prompt returns `400 full_vocab_prompt_too_long`. Use a fixed top-*k* (e.g. `prompt_logprobs=20`) for longer prompts. Completion `logprobs` can't be `-1` — full vocab is prompt-only. Budget for the download on long prompts: at ~128k vocab the body is roughly 4 MB per prompt token before compression, and the response streams — first bytes can take a while while the model server serializes. - **Send `Accept-Encoding: gzip`.** Most HTTP clients (the OpenAI SDK, `requests`, `httpx`, browsers, `curl --compressed`) do this by default and transparently decompress; the wrapper gzips the response, which shrinks these large logprobs bodies several-fold. ### Full distribution over generated text `-1` covers prompt positions only — vLLM rejects `logprobs=-1` on generated tokens, so those stay top-*k*. To get the full distribution at every step of a generation anyway, do it in two passes: generate first, then re-score prompt + completion as one prompt. ```python gen = client.completions.create( model="llama-8b", prompt=prompt, max_tokens=64, seed=7 ) full_text = prompt + gen.choices[0].text rescored = client.completions.create( model="llama-8b", prompt=full_text, max_tokens=1, extra_body={"prompt_logprobs": -1}, ) # rescored.choices[0].prompt_logprobs[i] is now the full ~128k-entry # distribution at position i — including every generated position. ``` It's the same forward pass over the same tokens, so the distributions match what the model saw while generating (up to float noise). Cost: one extra prefill. The combined text must fit the full-vocab prompt cap (`capabilities.full_vocab_max_prompt_tokens`). ## Gotcha **Scoring a sequence.** `rank=1` is the model's top prediction at each position, *not* necessarily your actual prompt token — and if your token wasn't in the top-*k*, vLLM still appends it as an extra entry with `rank > k`, so its logprob is recoverable. To get the true log-likelihood of your prompt: set `echo=true` to recover the prompt tokens, then at each position pick the entry whose `decoded_token` matches (use `prompt_logprobs=20` to keep the top-*k* generous, and fall back to the appended extra entry when the actual token ranked lower). `max_tokens` must be `≥ 1`, so set it to `1` and ignore the single generated token. # echo Prepends the prompt to the generated text in `choices[0].text`. Combined with `prompt_logprobs` and `max_tokens=1` (the wrapper's minimum), it turns a request into pure text-scoring — only one token is generated, which you ignore. ## curl ```bash curl -s "$ACS_API_BASE/completions" \ -H "Authorization: Bearer $ACS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"model": "llama-8b", "prompt": "Once upon a time", "max_tokens": 8, "echo": true}' ``` ## Python ```python resp = client.completions.create( model="llama-8b", prompt="Once upon a time", max_tokens=8, echo=True, ) print(resp.choices[0].text) # "Once upon a time there was a man who was very poor" ``` ## Example response ```json { "id": "cmpl-b986142b3fd7d215", "object": "text_completion", "model": "meta-llama/Llama-3.1-8B", "choices": [ { "index": 0, "text": "Once upon a time there was a man who was very poor", "logprobs": null, "finish_reason": "length" } ], "usage": {"prompt_tokens": 5, "completion_tokens": 8, "total_tokens": 13} } ``` Note that `text` contains both the prompt and the 8 generated tokens, concatenated with no separator; `usage.completion_tokens` counts only the generated portion. (`prompt_tokens=5`, not 4, because the tokenizer auto-prepends a `<|begin_of_text|>` BOS token to Llama prompts.) ## Gotcha `echo` only mirrors the prompt back; it doesn't add a separator, so split by length if you need just the continuation. # stream Server-sent events: each chunk is a partial completion, terminated by `data: [DONE]`. Set the request-level `timeout` high enough to span the full generation. ## curl ```bash curl -N -s "$ACS_API_BASE/completions" \ -H "Authorization: Bearer $ACS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"model": "llama-8b", "prompt": "Five reasons to learn Rust:\n1.", "max_tokens": 80, "stream": true, "stream_options": {"include_usage": true}}' ``` ## Python ```python stream = client.completions.create( model="llama-8b", prompt="Five reasons to learn Rust:\n1.", max_tokens=80, stream=True, stream_options={"include_usage": True}, ) for chunk in stream: if chunk.choices: # final usage chunk has choices=[] print(chunk.choices[0].text, end="", flush=True) elif chunk.usage: print(f"\n[tokens: {chunk.usage.total_tokens}]") ``` ## Example response First few SSE frames — the wire format; the Python SDK parses each `data:` payload into a chunk object. (Per-chunk `logprobs: null` and `stop_reason: null` keys are omitted here for brevity — they're present in the real wire output.) ```text data: {"id":"cmpl-ae7c4833c82a33d6","object":"text_completion","model":"meta-llama/Llama-3.1-8B","choices":[{"index":0,"text":" It","finish_reason":null}],"usage":null} data: {"id":"cmpl-ae7c4833c82a33d6","object":"text_completion","model":"meta-llama/Llama-3.1-8B","choices":[{"index":0,"text":"’s","finish_reason":null}],"usage":null} data: {"id":"cmpl-ae7c4833c82a33d6","object":"text_completion","model":"meta-llama/Llama-3.1-8B","choices":[{"index":0,"text":" fast","finish_reason":null}],"usage":null} data: {"id":"cmpl-ae7c4833c82a33d6","object":"text_completion","model":"meta-llama/Llama-3.1-8B","choices":[{"index":0,"text":".\n","finish_reason":null}],"usage":null} ... (one chunk per token until max_tokens or stop) ... data: {"id":"cmpl-ae7c4833c82a33d6","object":"text_completion","model":"meta-llama/Llama-3.1-8B","choices":[{"index":0,"text":" again","finish_reason":"length"}],"usage":null} data: {"id":"cmpl-ae7c4833c82a33d6","object":"text_completion","model":"meta-llama/Llama-3.1-8B","choices":[],"usage":{"prompt_tokens":9,"completion_tokens":80,"total_tokens":89}} data: [DONE] ``` The final content chunk carries `finish_reason` alongside its token (not on an empty text). Then — because the request set `stream_options.include_usage: true` — a terminal chunk arrives with `choices: []` and a populated `usage` object, where you get the token counts for a streamed completion. Then `[DONE]` closes the stream. Drop `stream_options` and you'll skip the usage chunk and get only the content frames (the OpenAI spec requires this opt-in for usage). ## Gotcha Use `curl -N` to disable buffering, otherwise you'll see the whole response arrive at once. Each SSE chunk's `finish_reason` is `null` until the last one. # Batch rollouts The per-key concurrency cap is 16; `asyncio.gather` over a single `AsyncOpenAI` client gives the maximum throughput without extra credentials. ## curl ```bash # Shell version — xargs -P 16 fans out 16 concurrent requests (the per-key cap): seq 1 32 | xargs -I{} -P 16 curl -s "$ACS_API_BASE/completions" \ -H "Authorization: Bearer $ACS_API_KEY" -H "Content-Type: application/json" \ -d '{"model": "llama-8b", "prompt": "Sample {}: once upon a time", "max_tokens": 32}' ``` ## Python ```python import asyncio import os from openai import AsyncOpenAI client = AsyncOpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"]) prompts = [f"Sample {i}: once upon a time" for i in range(32)] async def one(p): r = await client.completions.create(model="llama-8b", prompt=p, max_tokens=32) return r.choices[0].text async def _main(): # asyncio.gather() must be called from inside a running event loop, so # wrap it in a coroutine and hand THAT to asyncio.run(...). return await asyncio.gather(*(one(p) for p in prompts)) results = asyncio.run(_main()) ``` ## Gotcha Requests beyond the 8 active slots queue server-side, up to 64 waiters. A larger burst gets `429 queue_full`; sustained traffic above 600 request starts/minute gets `429 rate_limited`. Both include `Retry-After`, but the best fix is client-side backpressure: cap your own fan-out with `Semaphore(8)` so work does not pile up at the gateway. A single request is also bounded: `prompt_count × n × max_tokens` must be at most 32,000, where `prompt_count` is the number of prompts in your list (a single string — or one pre-tokenized `list[int]` — counts as 1). If you send a batch of prompts or `n>1`, set `max_tokens` explicitly. ## Tag your workload **Please set an `X-Acs-Workload` header on your requests** — `batch` for bulk / offline / eval jobs (like these rollouts), `interactive` for live, latency-sensitive calls. It's a usage signal we record to understand traffic and plan capacity / keep-warm policy; it does *not* change how your request is scheduled or prioritised, and an unrecognised value is simply ignored. Setting it accurately is a small thing that genuinely helps us tune the service for everyone. The in-browser Workbench is tagged `interactive` automatically. # Evaluate with Inspect [Inspect](https://inspect.aisi.org.uk/) (UK AI Safety Institute) is a widely-used framework for LLM evals. Its **`openai-api-completions`** provider talks to a raw OpenAI-compatible `/v1/completions` endpoint — raw prompts, **no chat template** — which is exactly what this API is, so you can run base-model evals against any model we host. Install Inspect (≥ 0.3.245) plus `openai`, which the provider needs: ```bash pip install "inspect-ai>=0.3.245" openai ``` ## Point Inspect at this API Inspect's `openai-api` family reads credentials from `_API_KEY` / `_BASE_URL`, where `` is a name you pick. Using `acs`: ```bash export ACS_API_KEY="sk-acs-..." # your key from the dashboard export ACS_BASE_URL="https://infra.acsresearch.org/v1" inspect eval your_task.py --model openai-api-completions/acs/llama-8b ``` The id after the last `/` is a **short model id from [`GET /v1/models`](/tutorial/models)** (e.g. `llama-8b`, `trinity-truebase`, `llama-405b`) — not the HF repo name. ## A minimal task ```python # capitals.py from inspect_ai import Task, task from inspect_ai.dataset import Sample from inspect_ai.scorer import includes from inspect_ai.solver import generate @task def capitals(): return Task( dataset=[ Sample( input="The capital of Spain is Madrid.\n" "The capital of Germany is Berlin.\n" "The capital of France is", target="Paris", ) ], solver=generate(), scorer=includes(), ) ``` ```bash inspect eval capitals.py --model openai-api-completions/acs/llama-8b --temperature 0 ``` Because these are **base models**, write prompts as raw text the model *continues*, not chat turns — few-shot prefixes and logprob scoring work well; instruction-style prompts often won't. The two example lines above are load-bearing: from the bare prompt `The capital of France is`, llama-8b's greedy continuation is ` a city of many faces. It is` and the eval scores 0. With the prefix its first sampled token is ` Paris` and the eval scores 1. ## Settings that matter here - **Concurrency** — keep `--max-connections` small (≈ 8). Each key allows **8 active + 64 queued** requests and **600 starts/min**; a large eval fan-out otherwise hits `429 queue_full` / `rate_limited`. (Same limits as [Batch rollouts](/tutorial/examples/batch-rollouts).) - **Pick an always-on model** (`llama-8b`, `trinity-truebase`) for big runs, or expect the first calls to wait through a **cold boot** on on-demand models like `llama-405b`. See [Cold-boot waiting](/tutorial/examples/cold-boot). - **Sampling + logprobs** — Inspect's `GenerateConfig` maps onto our params: `temperature`, `top_p`, `top_k`, `max_tokens`, `seed`, `stop`, `logprobs` / `top_logprobs`, and `num_choices` → `n` (≤ 16). Logprob-based scorers ride on [`logprobs`](/tutorial/examples/logprobs). - **Pre-tokenized prompts** — the provider can send token ids via `prompt_token_ids` message metadata; we forward them to the model **verbatim** (no re-tokenization). See [`prompt`](/tutorial/api#prompt) in the API reference. To recover the ids the server uses for a text prompt, send it once with `prompt_logprobs=0`: each prompt position after the first then carries exactly its own token id. The first position has no logprobs entry (nothing precedes it) — on the Llama models it is the BOS token, so prepend `128000` yourself. vLLM's `return_token_ids` request field is rejected as an unknown param. ## Get help Eval-specific question, or a provider bug? Email [infra@acsresearch.org](mailto:infra@acsresearch.org) or use the in-app **Feedback** button. If a specific call failed, include its `X-Request-Id`. # Cold-boot waiting Large on-demand models sleep when idle. The first request after scale-to-zero takes a few minutes — usually under 10, occasionally longer for the largest models — while Modal allocates GPUs and loads the model. Keep that **one request** open: the wrapper sends harmless whitespace bytes for non-streaming JSON (or SSE comments for `stream=true`) so Railway and your client do not treat the connection as idle. Set a client timeout of at least 15 minutes. Do not add a retry loop just to handle startup — the original request should eventually return the completion. ## curl ```bash curl --fail-with-body --no-buffer --max-time 900 \ "$ACS_API_BASE/completions" \ -H "Authorization: Bearer $ACS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"model": "llama-405b", "prompt": "Hello", "max_tokens": 4}' ``` ## Python ```python from openai import OpenAI client = OpenAI( api_key=ACS_API_KEY, base_url=ACS_API_BASE, timeout=900, max_retries=0, # the first request itself absorbs the cold boot ) resp = client.completions.create( model="llama-405b", prompt="Hello", max_tokens=4, ) print(resp.choices[0].text) ``` ## Wire-format note For `stream=false`, you may see whitespace arrive before the JSON object if you inspect raw bytes. Leading whitespace is valid JSON and normal clients buffer it before parsing. For `stream=true`, the keepalives are SSE comment frames and are ignored by OpenAI SDKs and EventSource clients. If startup exceeds the wrapper's 14-minute safety budget, the already-open response ends with a JSON/SSE error payload. That is an allocation failure, not a signal to blindly retry forever. ## Interrupted request (rare): the connection drops mid-cold-boot There is one case where retrying *is* the right move. The keepalive bytes commit an HTTP **200** status as soon as they start flowing — before the model has produced anything — and the body is sent as a chunked stream. If the wrapper process is restarted while still holding your request open (for example, a deploy lands during your cold boot), the connection closes **before the final chunk**, so you get a truncated response with no completion. How that surfaces depends on your client: - **Python / OpenAI SDK (`stream=false`)**: a **connection error**, not a readable response — `openai.APIConnectionError` (an incomplete chunked read underneath, `httpx.RemoteProtocolError`). Catch it and retry. - **`curl` (`stream=false`)**: partial bytes (often just whitespace) and a **non-zero exit** from the truncated transfer. Retry. - **`stream=true` (SDK or EventSource)**: the SSE stream closes **without** a terminal completion or `error` frame. Treat "stream ended, no terminal frame" as interrupted and retry. This is uncommon — it needs a server restart to coincide with an in-flight cold boot — and it is **safe to retry**. **Do not confuse this with a real error response.** A genuine failure (allocation timeout, budget exceeded, upstream error) arrives as a complete, parseable OpenAI-shaped JSON `{"error": {...}}` body (with a `200` for late cold-boot failures, since the status was already committed — see above). That is a real outcome with a code to act on, **not** a retry signal. The retry guard is "the transfer was *interrupted*" (transport error / truncated body / no terminal SSE frame) — **not** merely "the JSON has no `choices`," which an `{"error": ...}` body also lacks. # Budget-cap recovery When a token budget runs out, requests return `429 budget_exceeded`. There is no automatic refill mid-period — monthly caps reset on the 1st (UTC), the daily cap at midnight (UTC). The `message` always names which budget was hit (see [Account → Budgets](/tutorial/account)). ## curl ```bash curl -i "$ACS_API_BASE/completions" \ -H "Authorization: Bearer $ACS_API_KEY" -H "Content-Type: application/json" \ -d '{"model": "llama-8b", "prompt": "Hi", "max_tokens": 8}' # HTTP/1.1 429 Too Many Requests # The message is dimension-specific — one of: # "Monthly token budget exhausted for this API key." # "Monthly token budget exhausted for this user (aggregate across keys)." # "Daily token budget exhausted for this API key." # "Monthly input-token budget exhausted for this API key." (or output-token) # {"error": {"code": "budget_exceeded", "message": "Daily token budget exhausted for this API key.", ...}} ``` ## Python ```python from openai import APIStatusError try: resp = client.completions.create(model="llama-8b", prompt="Hi", max_tokens=8) except APIStatusError as e: # The wrapper nests the error code under body["error"]["code"], not at # the top level — so e.code (which the openai SDK reads from # body["code"]) is None here. Dig into e.body instead. err = (e.body or {}).get("error", {}) if isinstance(e.body, dict) else {} if e.status_code == 429 and err.get("code") == "budget_exceeded": # Don't retry — monthly caps are sticky until the 1st (UTC), the daily # cap until tomorrow. err["message"] names which dimension was hit. # Email infra@acsresearch.org to raise the budget. raise SystemExit(f"Budget exhausted ({err.get('message')}) — emailing the team for a raise.") raise ``` ## Gotcha `429 budget_exceeded` is *not* a transient rate-limit — exponential backoff just burns time. Inspect `error.code` and short-circuit non-retryable cases (concurrency limits, by contrast, simply queue).