Build your own local model proxy ​

Pragma ships a built-in assistant: a chat panel in the web app that acts on your workspace through the same tools AI agents use — always with your confirmation. Out of the box it runs on Pragma's platform model, and you can swap in your own instead: a hosted provider (Anthropic, OpenAI, Google Gemini, …) with your own key, or — the subject of this page — an endpoint that only your machine can reach.

That local endpoint does not have to be a model server. It can be a small proxy you write, translating Pragma's requests onto anything that can answer them:

  • a local model server you already run,
  • a CLI coding agent you already pay for (Claude Code, Codex),
  • an internal gateway your company operates.

In this mode your browser calls the endpoint directly — requests to your model never come from Pragma's servers, so the endpoint can stay on localhost.

This page is the complete specification for writing such a proxy. It is self-contained: everything Pragma sends and expects is described here, so you never need Pragma's source code. It is written to be handed to an AI coding agent — see the prompt block at the end. The only prerequisites on the Pragma side are an account and a model credential (described next).

What you're building ​

┌─────────────┐   HTTP (localhost)   ┌────────────┐          ┌──────────────┐
│ Pragma      │ ───────────────────▶ │ your proxy │ ───────▶ │ any backend  │
│ (browser)   │ ◀─────────────────── │            │ ◀─────── │ that answers │
└─────────────┘                      └────────────┘          └──────────────┘

The mental model that makes everything else fall into place:

  • Pragma's browser drives the conversation loop. Your proxy only answers one model turn at a time. It never calls Pragma.
  • Tools execute inside Pragma, never on your machine. When the model wants a tool, your proxy returns the tool call to the browser; Pragma runs it (with its own user-confirmation flow) and sends the result back in the next request.
  • Every request is stateless. Pragma re-sends the full conversation history each time. Your proxy may exploit that statefully (see session continuity) but must never require it.

Pragma-side setup ​

In the Pragma web app, open Settings → AI model (your personal model, or the organization-wide page if you administer one) and set up a model credential:

  • Service: choose OpenAI-compatible endpoint or Anthropic-compatible endpoint — this selects which wire dialect the browser speaks (both are specified below).
  • Address: your proxy's URL. For the OpenAI dialect enter a base ending in /v1 (Pragma appends /chat/completions); for the Anthropic dialect enter the bare origin (Pragma appends /v1/messages).
  • Where the service can be reached: Only on your own network — this is what makes your browser call the proxy directly.
  • Key: leave empty (your proxy is on localhost). If you set one, it arrives as a header (see below) — verify it if you care.
  • Model: optional free text, passed through as the model field.

Use the Test setup button after building: it sends exactly the request the assistant sends, so a passing test means the assistant works.

The wire contract ​

Common behavior (both dialects) ​

  • stream is always false. Answer with one complete JSON body. Never send server-sent events.
  • Message text is always plain strings — Pragma never sends content-part arrays, images, or files.
  • Pragma sends an output-token budget (default 2000). You may ignore it (recommended when your backend manages its own budget); honouring it strictly will clip long answers.
  • Tool-call ids you mint must be unique per call — never reuse the tool name; two calls to one tool must stay distinguishable.
  • Re-sent history is value-equal, not byte-equal: Pragma stores the assistant turn it parsed and re-serialises it. Consecutive text blocks come back joined into one string (no separator), and tool-call argument JSON comes back re-stringified (JSON.parse → JSON.stringify: same values, possibly different bytes). Irrelevant for a stateless proxy; critical if you fingerprint transcripts (Tier 2).
  • Every tool round is answered completely: the follow-up request carries exactly one result per call you returned, never a subset. A call the user rejected or that Pragma gated still gets a result (its content says so) — no call is ever left unanswered, so a parked handler always gets released.
  • You may split one model turn into successive single-call tool rounds (some backends reveal tool calls one at a time). Pragma's loop handles it — but each round counts against its per-turn tool-round budget (default 5), so batch calls when your backend surfaces them together (a short debounce after the first call works well).
  • CORS: the browser dials you directly, so preflights must pass. Answer OPTIONS with 204 and set on every response: access-control-allow-origin: <reflect the Origin header> (or *), access-control-allow-methods: GET, POST, OPTIONS, and access-control-allow-headers: content-type, authorization, api-key, anthropic-version, anthropic-beta, x-api-key, anthropic-dangerous-direct-browser-access.
  • Errors: answer non-2xx with {"error":{"message":"…"}} (a top-level {"message":"…"} also works). That sentence is shown to the user verbatim — write it for a human ("the backend is not running", not "ECONNREFUSED").
  • Bind to 127.0.0.1. Nothing outside your machine should reach the proxy.

What Pragma never sends: stream: true, image or file parts, tool definitions that are not JSON-Schema objects, or a system prompt that changes mid-conversation.

Dialect A — OpenAI-compatible (Chat Completions) ​

POST {base}/chat/completions with header Content-Type: application/json (plus Authorization: Bearer <key> or api-key: <key> only when a key is configured).

Request body — the complete set of fields Pragma sends:

json
{
  "model": "optional-model-name",
  "messages": [
    { "role": "system", "content": "…" },
    { "role": "user", "content": "…" },
    { "role": "assistant", "content": "…or null",
      "tool_calls": [ { "id": "call_abc", "type": "function",
        "function": { "name": "create_item", "arguments": "{\"title\":\"X\"}" } } ] },
    { "role": "tool", "tool_call_id": "call_abc", "content": "…result JSON or text…" }
  ],
  "max_tokens": 2000,
  "stream": false,
  "tools": [
    { "type": "function", "function": {
        "name": "create_item", "description": "…",
        "parameters": { "type": "object", "properties": { }, "required": [] } } }
  ]
}

Notes: model is omitted when the credential has none. The budget field is max_tokens or max_completion_tokens depending on provider configuration — accept both. tools is omitted when there are none. Assistant content is null when the turn was nothing but tool calls.

Response — the only fields Pragma reads:

json
{
  "model": "what-actually-answered",
  "choices": [ {
    "finish_reason": "stop | tool_calls | length",
    "message": {
      "role": "assistant",
      "content": "…or null",
      "tool_calls": [ { "id": "call_abc", "function": {
          "name": "create_item", "arguments": "{\"title\":\"X\"}" } } ]
    } } ],
  "usage": { "prompt_tokens": 123, "completion_tokens": 45 }
}

choices must be an array — its absence is how Pragma detects that the address is not a chat-completions endpoint, and it tells the user so. Always include "role": "assistant" in the message: the verification script below (like any client that replays your response message verbatim into its next request) depends on it. finish_reason, usage, and model are optional but reported to the user when present (in usage, report the provider's own numbers; total context including cached tokens is fine). An empty content with no tool_calls is treated as a suspicious answer — when your backend genuinely produces nothing, substitute a fixed sentence (e.g. "I have no further output for this turn.") rather than an empty string.

Dialect B — Anthropic-compatible (Messages) ​

POST {base}/v1/messages with headers content-type: application/json, anthropic-version: 2023-06-01 (plus x-api-key only when a key is configured).

Request body:

json
{
  "model": "optional-model-name",
  "max_tokens": 2000,
  "stream": false,
  "system": "…optional system text…",
  "messages": [
    { "role": "user", "content": "plain string OR array of blocks" },
    { "role": "assistant", "content": [
        { "type": "text", "text": "…" },
        { "type": "tool_use", "id": "toolu_abc", "name": "create_item",
          "input": { "title": "X" } } ] },
    { "role": "user", "content": [
        { "type": "tool_result", "tool_use_id": "toolu_abc",
          "content": "…result…" } ] }
  ],
  "tools": [ { "name": "create_item", "description": "…",
               "input_schema": { "type": "object", "properties": {} } } ]
}

Response — Pragma reads the content block array (text and tool_use blocks), stop_reason, model, and usage.input_tokens / usage.output_tokens:

json
{
  "id": "msg_x", "type": "message", "role": "assistant", "model": "…",
  "content": [
    { "type": "text", "text": "…" },
    { "type": "tool_use", "id": "toolu_abc", "name": "create_item",
      "input": { "title": "X" } } ],
  "stop_reason": "end_turn | tool_use",
  "stop_sequence": null,
  "usage": { "input_tokens": 123, "output_tokens": 45 }
}

Conversation lifecycle — fixtures ​

The four request shapes your proxy will see, in order (OpenAI dialect; the Anthropic dialect is the same flow with the block shapes above). S is the system message, sent on every request.

1. Fresh turn — [S, user] → answer text, finish_reason: "stop".

2. Follow-up — [S, user, assistant, user] → the whole history again plus one new user message.

3. Tool round — [S, …history…, user] where the model should act → answer with tool_calls and finish_reason: "tool_calls":

json
{ "content": "One line of intent, or null",
  "tool_calls": [ { "id": "call_1a2b", "type": "function", "function": {
      "name": "create_item", "arguments": "{\"title\":\"Verify proxy\"}" } } ] }

4. Tool result continuation — the next request appends your assistant turn and the results, ending in one role:"tool" message per call:

[S, …history…, assistant(tool_calls), tool(call_1a2b)]

Answer it like any turn: more tool_calls, or final text. Several trailing tool messages arrive when the model made several calls in one turn.

Tier 1 — stateless replay (build this first) ​

Each request, run your backend once over the full transcript and translate the answer back. Always correct; the only cost is tokens (each turn re-reads the whole history). ~100 lines. Recipe:

  1. Parse the request; reject non-conforming bodies with a 400 and a human sentence.

  2. Encode the transcript into whatever your backend takes. For a prompt-driven backend, a plain-text replay works:

    You are resuming an existing conversation. Transcript so far:
=== TRANSCRIPT ===
USER: …
ASSISTANT: …
ASSISTANT called tool create_item({"title":"X"}) (call id call_1a2b)
TOOL RESULT (call id call_1a2b): {"id":"item_1","status":"created"}
=== END TRANSCRIPT ===
Continue the conversation: reply to the last USER entry.

    Put the system text wherever your backend expects instructions (system prompt option, an AGENTS.md in its working directory, …), not in the transcript.

  3. Declare Pragma's tools to the backend (most agent backends: as an in-process or local MCP server passing the JSON Schemas through unchanged). When the backend calls one, do not execute anything: capture the name and arguments, mint a unique call id, and answer the HTTP request with tool_calls. Then either abandon that backend session (Tier 1) or park the call (Tier 2).

  4. Translate the final answer into the response shape above.

Tier 2 — session continuity (cache riding) ​

Replay is O(N²): turn n re-processes n−1 turns of history, none of it cached from your provider's perspective if each replay is a fresh session. Keeping the backend session alive across HTTP requests changes the economics completely: only the delta (one user message or one tool result) is new input, and the entire history rides the provider's prompt cache. In practice a continued turn costs a few input tokens where a replay costs tens of thousands — this is the difference that makes a subscription-backed proxy pleasant instead of wasteful.

The design that makes it work:

Session registry. Keep live backend sessions in memory, indexed two ways:

  1. By pending tool-call id. A request whose trailing messages are tool results belongs to the session that emitted exactly those call ids. Consume the ids on match so a retried request cannot double-route.
  2. By transcript fingerprint. A request ending in plain user text belongs to the idle session whose absorbed transcript equals everything before that last message. Hash a normalized transcript: per message keep (role, text, [(callId, toolName, arguments)], [(callId, resultText)]), with consecutive text blocks joined and argument JSON parsed and re-serialised with sorted keys — this is what survives the browser's round trip (see "value-equal, not byte-equal" above).

Anything that matches neither — proxy restart, evicted session, edited history, changed system prompt/tool set/model — falls back to Tier 1 replay in a fresh session. That fresh session is a full member of the registry, not a throwaway: keep it alive, and once its turn completes index its absorbed transcript (and register its secret bridge path, if your backend dials back over HTTP) like any other session — the next request then continues it live instead of replaying again. The fallback is why Tier 2 is safe: it is never wrong, only cheaper when it hits. Log which path each request took; the log is how you verify the cache is actually riding.

Config key. Hash (model, system text, tool definitions) into every index. Two conversations with different configurations must never share a session — the session was built with the other one's instructions. Since the config key already covers the system text, exclude system messages from the transcript fingerprint (including them also works; just be consistent on both sides of the hash).

Parked tool bridge. When the backend calls a tool, the tool handler blocks on a promise and the proxy answers the browser with tool_calls. The browser's next request (carrying the results) routes back via index 1 and resolves the promise with the result text — the backend continues in the same session, so the tool round costs only the result tokens. Give the handler a long timeout (browser round trips include a human confirming).

Housekeeping (a session is a live process or context — cap it):

  • Maximum live sessions (e.g. 6); evict the least-recently-used idle one.
  • Idle timeout (e.g. 30 min).
  • A watchdog per turn (e.g. 10 min of silence → abort, return an error).
  • On eviction/death, release any parked tool handler with an error message.
  • On shutdown, explicitly close any MCP transports / server-sent-event streams the backend still holds open — they otherwise keep the process alive indefinitely.

Backend notes — hard-won pitfalls ​

Generic, whatever the backend:

  • Use the backend's sanctioned headless surface with the login it already has. Do not extract OAuth tokens or impersonate the vendor's own client against their API — that breaks vendors' terms.
  • Disable everything local: the backend must not read files, run commands, or search the web on behalf of a Pragma conversation. Empty temp working directory, read-only or disabled built-in tools, no user config loaded.
  • Pass Pragma's tool schemas through unchanged (raw JSON Schema). Do not re-model them.
  • The credential's free-text model field passes through to your backend. An opinionated backend (a vendor CLI) fails on a foreign model name — either validate/map the field, or document that it must be left empty.
  • A subscription-backed CLI is not anonymous: it may inject account context (the account holder's name, for one) into the conversation, and there may be no option to suppress it. Say so to your users.
  • Run on a modern Node (20+) and guard for it at startup. Both vendor SDKs claim to support Node 18 and both break under it in practice (the Claude Agent SDK aborts every turn instantly under Node 18.16; the MCP SDK expects a global crypto that early Node 18 lacks) — and an old nvm default produces a proxy that starts fine and fails every request.
  • Control the child environment. Launched from inside another agent harness, the backend child inherits that harness's variables (CODEX_*, task context, …). Strip what the backend does not need, but keep PATH, HOME, and temp/locale variables so the existing login still resolves.

Claude Code (@anthropic-ai/claude-agent-sdk, needs a logged-in claude CLI):

  • query({ prompt, options }); for a live multi-turn session pass an async iterable of user messages as prompt and keep pushing into it. User messages are { type: 'user', message: { role: 'user', content }, parent_tool_use_id: null }.
  • Options for a clean bridge: tools: [] (disables all built-ins), settingSources: [] (loads no CLAUDE.md), persistSession: false, cwd set to an empty temp dir, systemPrompt replaces the default. envreplaces the subprocess environment — spread process.env in.
  • Serve Pragma's tools as an in-process MCP server: mcpServers: { pragma: { type: 'sdk', name: 'pragma', instance } }. The high-level McpServer helpers are Zod-only; for raw JSON Schema passthrough call instance.server.registerCapabilities({ tools: {} }) then setRequestHandler(ListToolsRequestSchema | CallToolRequestSchema, …). Without registerCapabilities, every tools/list is refused.
  • Correlate calls in canUseTool(toolName, input, { toolUseID }) — record the id, return { behavior: 'allow', updatedInput: input }. Do not put the tools in allowedTools: an allowlisted tool is pre-approved by rule and skips canUseTool, which silently breaks the correlation.
  • Assistant messages arrive complete (when not streaming partials); read text and tool_use blocks off message.content. A result message ends the turn and carries usage.

Codex (@openai/codex-sdk, bundles the codex binary; uses ~/.codex/auth.json — a ChatGPT-app login works):

  • new Codex({ config }) → startThread({ workingDirectory, skipGitRepoCheck: true, sandboxMode: 'read-only', approvalPolicy: 'never', webSearchMode: 'disabled' }) → thread.runStreamed(input) and consume item.completed / turn.completed / turn.failed events.
  • A "live session" is the retained thread, not a long-running local process: every runStreamed call spawns a fresh short-lived codex exec that resumes the persisted thread (it only outlives the call while parked on a bridged tool call). Each run also re-initializes its MCP connection from scratch — never carry MCP transport state across runs.
  • Codex has no in-process MCP: mount a streamable HTTP MCP server on your proxy under a secret per-session path and point the thread at it via config: { mcp_servers: { pragma: { url, tool_timeout_sec: 3600, default_tools_approval_mode: 'approve' } } }.
    • default_tools_approval_mode: 'approve' is mandatory: the default (auto) treats un-annotated tools as writes needing approval, and the never approval policy then refuses the call.
    • With the @modelcontextprotocol/sdk server transport, build a fresh transport + server pair per HTTP request (its stateless pattern); a reused transport 500s the client's initialized notification (or answers "Server already initialized").
    • Hand every HTTP method on the bridge path to that transport — the MCP client speaks more than POST JSON-RPC (initialize → notifications/initialized → a GET server-sent-events stream → tools/list → tools/call); parsing the GET yourself as JSON silently breaks tool discovery.
    • Register the bridge path for every session that can receive a backend connection — including short-lived replay-seeded ones — or valid tools/list calls 404 before the model can act.
  • Your ~/.codex/config.toml is shared with the desktop app — disable its MCP servers for proxy threads (mcp_servers.<name>.enabled = false).
  • System text: write it into an AGENTS.md in the thread's working directory — and repeat the critical role/tool instruction in the seed prompt itself: codex exec does not reliably pick up an AGENTS.md from a non-repository temp directory in every version. If the model claims the bridge tool is unavailable, the usual causes are that missing instruction, MCP state carried across runs, or the desktop app's own MCP servers leaking in.

Verification ​

Self-verify with this script before touching Pragma (OpenAI dialect; adapt the shapes for the Anthropic dialect). It exercises every routing path a Tier 2 proxy has; a Tier 1 proxy passes it too, just slower.

js
// smokeTest.mjs — run with the proxy up. Spends a few real model turns.
const BASE = process.env.PROXY_URL ?? 'http://127.0.0.1:8490';
const TOOLS = [{ type: 'function', function: {
  name: 'create_manageable',
  description: 'Create a work item in the Pragma workspace.',
  parameters: { type: 'object', properties: {
    title: { type: 'string' }, kind: { type: 'string', enum: ['task', 'requirement'] } },
    required: ['title', 'kind'] } } }];
const SYSTEM = { role: 'system',
  content: 'You are the Pragma assistant. Use tools to act on the workspace. Be brief.' };

async function turn(label, messages) {
  const res = await fetch(`${BASE}/v1/chat/completions`, {
    method: 'POST', headers: { 'content-type': 'application/json' },
    body: JSON.stringify({ max_tokens: 2000, stream: false, tools: TOOLS,
      messages: [SYSTEM, ...messages] }) });
  const body = await res.json();
  if (!res.ok) throw new Error(`${label}: HTTP ${res.status} ${JSON.stringify(body)}`);
  const message = body.choices[0].message;
  console.log(`${label}: finish=${body.choices[0].finish_reason}`,
    JSON.stringify(message).slice(0, 160));
  return message;
}

const h = [];
h.push({ role: 'user', content: 'Say hello in exactly three words.' });
h.push(await turn('1 fresh', h));
h.push({ role: 'user', content: 'Now say goodbye in exactly two words.' });
h.push(await turn('2 follow-up', h));
h.push({ role: 'user', content: 'Create a task titled: Verify stateful proxy' });
const third = await turn('3 tool round', h);
if (!third.tool_calls?.length) throw new Error('expected tool_calls');
h.push(third);
h.push({ role: 'tool', tool_call_id: third.tool_calls[0].id,
  content: '{"id":"manageable_task__xyz789","status":"created"}' });
h.push(await turn('4 tool result', h));
h.push({ role: 'user', content: 'What is the id of the task you just created?' });
const fifth = await turn('5 post-tool', h);
if (!JSON.stringify(fifth).includes('xyz789')) throw new Error('lost tool context');
const foreign = [
  { role: 'user', content: 'Remember the code word: pamplemousse.' },
  { role: 'assistant', content: 'Understood, I will remember it.' },
  { role: 'user', content: 'What is the code word?' }];
const sixth = await turn('6 replay fallback', foreign);
if (!JSON.stringify(sixth).toLowerCase().includes('pamplemousse'))
  throw new Error('replay fallback lost history');
console.log('All six flows passed.');

Acceptance criteria:

  1. All six flows pass.
  2. For Tier 2: the proxy log shows flows 2–5 taking the live session paths (continuation / tool-result routing) and only flow 6 replaying; and the fresh (uncached) input tokens collapse on continued turns — log your backend's cache-hit/fresh split per turn, because the total reported prompt_tokens barely discriminates a live session from a replay.
  3. Then, in Pragma: Test setup passes, and an assistant conversation with at least one confirmed tool action completes.

Security and terms ​

  • Localhost only, secret unguessable paths for anything the backend dials back (like an HTTP MCP bridge), no credentials stored in the proxy. That secret path is a bearer credential: keep it out of request logs and debug endpoints.
  • If your backend is a subscription-backed coding agent, that subscription's terms govern this use. Running a proxy for your own personal setup is a different matter from distributing one as part of a product — vendors restrict third parties routing product traffic through consumer plans (Anthropic prohibits it explicitly; OpenAI steers products to API keys). Read the vendor's current terms yourself; they change without notice.

Hand this page to an agent ​

Build a local proxy per the specification in this document. Target dialect: OpenAI-compatible (or Anthropic-compatible — pick one). Backend: <your backend, e.g. Claude Code via @anthropic-ai/claude-agent-sdk>, using the login already present on this machine. Start with Tier 1 (stateless replay), verify with the smoke-test script, then implement Tier 2 (session continuity) and verify the live-session routing paths in the proxy log. Bind to 127.0.0.1. Do not execute any tool locally — bridge every tool call back over the wire as specified.