The first time I wired an LLM API into a production feature, I treated it like any other third-party API — send a request, get a response, move on. That assumption cost me a very confusing week of debugging. LLM APIs are not REST endpoints that happen to be smart. They're probabilistic systems wearing an HTTP interface, and every part of my integration approach had to change once that sunk in.
Mistake #1: Trusting the Output Shape
My first integration asked a model to return JSON and parsed the response directly with JSON.parse. It worked in testing. It broke in production the moment a user's input nudged the model into wrapping its answer in a markdown code fence, or adding a friendly sentence before the JSON. The fix wasn't a smarter prompt — it was defensive parsing: strip code fences, extract the first valid JSON substring, and validate the result against a schema before trusting it.
// Never trust raw model output — validate before you use it
import { z } from 'zod';
const ResponseSchema = z.object({
intent: z.enum(['refund', 'support', 'sales']),
confidence: z.number().min(0).max(1),
});
function parseModelOutput(raw: string) {
const jsonMatch = raw.match(/\{[\s\S]*\}/);
if (!jsonMatch) throw new Error('No JSON found in model output');
const parsed = JSON.parse(jsonMatch[0]);
return ResponseSchema.parse(parsed); // throws if shape is wrong
}Mistake #2: No Timeout, No Fallback
LLM calls are slow and occasionally hang. My early integrations had no timeout configured, which meant a single stuck request could tie up a serverless function until the platform's own hard limit killed it. Now every model call gets an explicit timeout well below the infrastructure ceiling, plus a fallback path — a cached response, a simpler rule-based answer, or a graceful 'try again' message. The feature should degrade, not disappear.
Streaming Changes Everything About UX
Once I switched from waiting for a full completion to streaming tokens as they arrived, user-perceived latency dropped dramatically even though the total generation time didn't change. People don't mind waiting for text as long as something is visibly happening. Implementing this properly meant rethinking the frontend as an incremental renderer rather than a request/response consumer — appending tokens to state, handling partial JSON gracefully, and designing loading states around 'first token' latency rather than 'last token' latency.
// Streaming a completion and updating UI incrementally
const stream = await client.messages.stream({
model: 'claude-sonnet-5',
max_tokens: 1024,
messages: [{ role: 'user', content: userInput }],
});
for await (const event of stream) {
if (event.type === 'content_block_delta') {
appendToUI(event.delta.text);
}
}Cost Is a Design Constraint, Not an Afterthought
The first version of a summarization feature I built sent the full document on every request, even for follow-up questions about the same document. The bill made that obvious within a week. Prompt caching, trimming irrelevant context, and choosing a smaller model for simple sub-tasks (and reserving the frontier model for the parts that actually need it) turned an expensive feature into a sustainable one. Treat token budget like you'd treat a database query budget — profile it, don't guess.
Retries Need to Be Idempotent-Aware
Rate limits and transient errors are normal, not exceptional, when calling LLM APIs at any real volume. Exponential backoff with jitter is table stakes. What tripped me up was retrying a call that had side effects — like a function-calling flow that had already written to a database before the response failed to parse. The lesson: separate the model call from the side effect, and only commit the side effect after the full response has been validated.
An LLM API isn't a black box you call once and trust — it's a collaborator you validate, timeout, cache, and gracefully disagree with.
Looking back, every hard lesson traces to the same root cause: treating a non-deterministic system like a deterministic one. Once I started designing for variability — in output shape, latency, and cost — the integrations became boring in the best possible way. Boring is what you want in production.