LLMs Will Not Return Clean JSON (And What to Do About It)

Sure! Here's the rewritten resume tailored for the Senior Backend Engineer role:

```json
{
  "companies": [
    {
      "company": "Acme Corp",
      "bullets": ["Built distributed systems...", "Reduced latency by 40%..."]
    }
  ]
}

There's a valid JSON object in there. It's wrapped in markdown code fences. There's a sentence before it and a helpful sign-off after it. `JSON.parse()` throws on all of it.

This is the structured output problem: <TypographyMark>you can't reliably prompt your way to consistent JSON</TypographyMark>. You need a layer between the LLM and your application code that handles the mess.

## Why Prompting Alone Doesn't Fix It

The instinct is to add more instruction to your system prompt. "Return ONLY valid JSON. No markdown. No explanations. Just the JSON object." I tried this. It helps. It does not solve it.

The issue is that LLMs are trained to be helpful and conversational. That training pulls against the instruction to be silent and machine-readable. The model has competing objectives, and sometimes the conversational one wins — especially on longer outputs, complex schemas, or when the model is uncertain about part of the response and wants to annotate it.

OpenAI's structured output and Anthropic's tool-use features help significantly, but they're not universally available across providers, and they add latency and cost. If you're routing through OpenRouter to swap models, or using smaller/faster models, you don't always have access to constrained decoding.

The practical answer is: assume the output is dirty, and write code that cleans it.

## The Three-Layer Approach

Before showing the extraction code, here's the architecture:

<RoughDiagram
  direction="LR"
  nodes={[
    { id: "a", label: "LLM Response",      color: "blue" },
    { id: "b", label: "Layer 1: Prompt", color: "yellow" },
    { id: "c", label: "Layer 2: Extraction", color: "green" },
    { id: "d", label: "Layer 3: Zod", shape: "diamond", color: "red" },
    { id: "e", label: "Return Parsed Data", color: "purple" },
    { id: "f", label: "Retry or Fallback", color: "orange" },
  ]}
  edges={[
    { from: "a", to: "b" },
    { from: "b", to: "c" },
    { from: "c", to: "d" },
    { from: "d", to: "e", label: "valid" },
    { from: "d", to: "f", label: "invalid" },
  ]}
/>

## The Extraction Layer

Here's the function I built for ColdCraft's `llm.ts`:

```typescript
function extractJSON(text: string): string {
  // Strip markdown code fences
  let cleaned = text
    .replace(/```json\s*/gi, "")
    .replace(/```\s*/g, "")
    .trim();

  // Find the first JSON structure (object or array)
  const objStart = cleaned.indexOf("{");
  const arrStart = cleaned.indexOf("[");

  if (objStart === -1 && arrStart === -1) return cleaned;

  const start =
    objStart === -1
      ? arrStart
      : arrStart === -1
        ? objStart
        : Math.min(objStart, arrStart);

  const isArray = cleaned[start] === "[";
  const openChar = isArray ? "[" : "{";
  const closeChar = isArray ? "]" : "}";

  // Walk character by character to find the matching close bracket
  let depth = 0;
  let inString = false;
  let escaped = false;

  for (let i = start; i < cleaned.length; i++) {
    const char = cleaned[i];

    if (escaped) { escaped = false; continue; }
    if (char === "\\") { escaped = true; continue; }
    if (char === '"') { inString = !inString; continue; }
    if (inString) continue;

    if (char === openChar) depth++;
    else if (char === closeChar) {
      depth--;
      if (depth === 0) return cleaned.slice(start, i + 1);
    }
  }

  return cleaned.slice(start);
}

const RewrittenExperienceSchema = z.object({
  companies: z.array(
    z.object({
      company: z.string(),
      bullets: z.array(z.string()),
    })
  ),
  reasoning: z.string().optional(),
});

export async function createStructuredOutput<T extends z.ZodType>(
  prompt: string,
  schema: T,
  options?: { model?: string; systemPrompt?: string }
): Promise<z.infer<T>> {
  const chat = getLLMClient(options?.model);

  const messages = [
    new SystemMessage(
      `You are a helpful assistant that ONLY responds with valid JSON.
Never include markdown code blocks, explanations, or any text outside the JSON.
${options?.systemPrompt || ""}`
    ),
    new HumanMessage(prompt + "\n\nRespond with ONLY valid JSON, no other text."),
  ];

  const response = await chat.invoke(messages);
  const content = response.content as string;

  const jsonStr = extractJSON(content);
  const parsed = JSON.parse(jsonStr);

  return schema.parse(parsed);
}

try {
  const rewritten = await createStructuredOutput(prompt, RewrittenExperienceSchema);
  for (const company of rewritten.companies) {
    result.set(company.company, company.bullets);
  }
} catch (error) {
  console.error("Failed to rewrite bullets:", error);
  // Fall back to original — resume stays usable
  for (const exp of experience) {
    result.set(exp.company, exp.highlights || []);
  }
}

return new ChatOpenAI({
  model: modelName,
  apiKey: process.env.OPENROUTER_API_KEY,
  configuration: {
    baseURL: "https://openrouter.ai/api/v1",
  },
});