How to Validate LLM-Generated Code Before Users See It, Even for Domain-Specific Languages

To make LLM-generated code reliable in production, do not trust the first answer. Run every generated snippet against the real runtime, or a sandboxed copy of it, before the user sees it. If it fails, send the exact error back to the model and let it try again a limited number of times. Wrap that loop in structured output so you can extract the code cleanly, detect up front whether the user wants code at all, and measure first-try and second-try success on a fixed test set.

This matters most for domain-specific languages (DSLs), query languages and in-house scripting languages, where models have seen little training data and confidently invent syntax. The rest of this article walks through the architecture step by step, using the assistant OSKI built inside Lokad’s platform as a worked example.

Why LLM-generated code fails in production

Generated code fails for a short list of reasons, and each one points to a different control:

Mainstream languages hide some of these problems because models have seen millions of examples. A DSL exposes all of them. In practice, “the model is good at code” says little about how it will do in your language. You find out by executing its output.

Step 1: Execute generated code against the real runtime

The single most effective control is simple to state: code that has not run does not reach the user.

There are three levels of checking, and you want the highest one your runtime allows:

1. Parse or compile. Cheap and fast. Catches syntax errors and many unknown identifiers. For a DSL, expose the language’s own parser as a service if you can.
2. Execute in a sandbox. Run the script against a safe dataset or a read-only copy of the user’s environment. This catches runtime errors, type mismatches and missing fields the parser cannot see.
3. Check the result. Compare output shape with what the request implies: expected columns, row counts in a sensible range, no empty tables where data should exist. For known tasks, use real assertions.

Use the actual runtime, not a model-based “review” of the code. Asking a second LLM whether the code looks right is useful for style and intent, but it is not a substitute for a compiler.

Design the sandbox before you need it

The execution environment is part of your security boundary. A practical checklist:

• Run with a dedicated service identity that has read-only access, scoped to the user’s own data.
• Set hard limits on CPU time, memory, output size and wall-clock time.
• Block network egress unless the language needs it.
• Never pass model output to eval or a shell inside your application process.
• Log every execution with the prompt, the code, the result and the error, so failures can be replayed.

OWASP lists this class of risk as LLM05:2025 Improper Output Handling and recommends treating model output like input from any untrusted user, validating it before it reaches backend functions and encoding it for the context where it is displayed. Generated code is the extreme case of that rule.

Worked example: Lokad’s Envision assistant

Lokad builds supply chain optimization software, and its clients write forecasting and inventory logic in Envision, a domain-specific language Lokad created. New users need time and help to write their first scripts. In 2025 OSKI built an assistant directly into the platform.

The assistant connects securely to the Envision runtime and runs generated code there. Scripts that fail go back for another pass, so users receive code that has already executed. The results reported in the Lokad AI assistant case study: 82% of generated scripts run on the first try, 94% by the second, and users reach a first working Envision script about 70% faster.

The 12-point gap between first and second try is the value of the retry loop in that system. Without execution, those scripts would have reached users broken.

Step 2: Retry with error feedback, and cap it

When execution fails, the most useful thing you can give the model is the precise error: message, line number, the failing token and, if available, a short hint from the language reference about the construct involved. Research on “self-debugging” found that models fix more of their own mistakes when given execution feedback than when asked to review code blind, which matches what teams see in practice.

A minimal loop looks like this. It is illustrative C#, not a drop-in library:

// Illustrative: a capped generate-execute-repair loop
public async Task<AssistantResult> GenerateVerifiedAsync(
    Conversation convo, string userRequest, CancellationToken ct)
{
    const int maxAttempts = 3; // first try + 2 repairs
    var attempts = new List<Attempt>();

    for (var i = 1; i <= maxAttempts; i++)
    {
        var prompt = PromptBuilder.ForCode(convo, userRequest, attempts);
        var draft = await _llm.GenerateStructuredAsync<CodeDraft>(prompt, ct);

        var run = await _sandbox.ExecuteAsync(draft.Code, convo.UserScope, ct);
        attempts.Add(new Attempt(i, draft, run));
        _metrics.RecordAttempt(i, run.Succeeded, run.ErrorCategory);

        if (run.Succeeded)
            return AssistantResult.Verified(draft, run, attemptNumber: i);
    }

    // Do not show unverified code as if it works
    return AssistantResult.Failed(attempts.Last());
}

Rules that keep the loop healthy:

• Cap attempts. Two or three is enough for most systems. Each extra attempt adds latency and cost while fixing fewer errors.
• Send only the last attempt and its error, plus a short summary of earlier failures. Pasting every previous draft wastes context and can anchor the model on the same mistake.
• Classify errors. Syntax, unknown symbol, type, timeout and permission errors need different hints. A timeout usually needs a narrower query, not a syntax fix.
• Fail honestly. If all attempts fail, tell the user, show the error and offer the closest draft clearly labeled as unverified, or route them to a person.
• Stream progress. “Checking the script…” is better than a silent ten-second wait.

Step 3: Use structured output to separate code from prose

Free-text responses that mix explanation and fenced code blocks are fragile to parse. Structured output solves the extraction problem. With OpenAI’s Structured Outputs (text.format with type: "json_schema" and strict: true) and Claude’s structured outputs (output_config.format with a JSON schema, now generally available), the response is constrained to your schema.

A schema for a code assistant can stay small:

{
  "type": "object",
  "properties": {
    "mode": { "type": "string", "enum": ["explain", "generate"] },
    "explanation": { "type": "string" },
    "code": { "type": "string" },
    "assumptions": { "type": "array", "items": { "type": "string" } }
  },
  "required": ["mode", "explanation", "code", "assumptions"],
  "additionalProperties": false
}

Two caveats from the providers’ own docs are worth building around. A safety refusal or a response cut off at the token limit may not match the schema, so check the stop reason or refusal field before parsing. And schema support covers a subset of JSON Schema, so keep schemas flat.

Remember what structured output does not do: it guarantees the JSON is well formed, not that the code inside it runs. It is a parsing tool, not a validation tool.

Step 4: Detect the mode before generating anything

Most assistants that write code also answer questions. Users ask “what does this function do?”, “who should I talk to about pricing?” and “write me a script that ranks suppliers by lead time” in the same chat window. If every message goes through the code path, explanations get padded with unnecessary scripts. If none do, users have to ask twice.

A cheap first step classifies intent:

• Explain: concepts, documentation, “how does X work”, troubleshooting advice.
• Generate: a concrete task with inputs and an expected output.
• Route: the question needs a human, such as commercial, account or project-specific topics.
• Clarify: the request is code-shaped but missing something essential.

Use a small, fast model or a structured-output call for the classifier, and log its decisions so you can review misroutes. Lokad’s assistant uses a multi-step check to decide whether a message calls for an explanation or for code; in Envision mode it writes the script and formats it with syntax highlighting in the chat. When a question needs a person, it suggests the right Lokad contact, which it does in about 78% of cases. Together with the explain mode, that cut the time users spend searching documentation or looking for the right person by about 60%.

Step 5: Ground the model in your language

For a DSL, the model needs reference material it did not see during training. In order of impact:

1. Real usage patterns. Before writing prompts on the Lokad project, the team studied how people used the platform and Envision: the questions they asked most and where they got stuck. Those patterns became the assistant’s core behaviors. This is cheaper and more useful than writing a generic system prompt.
2. Verified examples. A library of short scripts that are known to run, each tagged by task. Retrieve the closest few per request.
3. Language reference excerpts. Function signatures, types and common errors, retrieved by relevance rather than pasted in full.
4. Schema of the user’s data. Table and column names the script is allowed to reference. Many “hallucinated field” errors disappear when the model sees the real schema.

Every example in the library must itself pass the execution check. A wrong example teaches the model the wrong pattern in every response that retrieves it.

Step 6: Keep conversation memory outside the model

Code requests are iterative: “now group it by warehouse”, “exclude discontinued items”. The assistant needs the previous script, the previous result and the user’s corrections. LLM calls are stateless, so memory is your job.

Practical choices:

• Store each turn as an event: user message, mode, generated code, execution result, attempt count.
• Rebuild context from that log for each request, keeping the latest working script in full and summarizing older turns.
• Scope memory per user and per workspace, and apply the same retention rules as other user data.

On the Lokad assistant, conversation memory sits on Lokad’s own Azure event store, so context survives across messages in a real-time, event-driven system. 99% of conversations keep their context.

Step 7: Measure first-try and second-try success

If you only track “did the user get working code”, you cannot tell whether a prompt change improved the model or just made the retry loop work harder. Measure attempts separately.

This is the same idea as the pass@k metric used in code-generation research, applied to your own traffic: pass@1 is your first-try rate, and a capped retry loop with execution feedback is how you move closer to pass@k in production.

Build a fixed evaluation set

Production metrics drift with user mix. Keep a frozen set of real requests, anonymized, with known-good solutions or result assertions. Run it before every prompt change, model upgrade or example-library update. Report first-try and by-second-try success side by side, and do not ship a change that improves one while quietly hurting the other.

For reference, the Lokad assistant reached 82% first try and 94% by second try. Your numbers will depend on the language, the task mix and how strict your result checks are, so treat them as a benchmark for what execution-backed generation can achieve, not a target to copy.

Step 8: Safety and governance checklist

Before launch, confirm each item:

• Generated code only runs in an isolated sandbox with read-only, user-scoped credentials.
• Execution has time, memory and output limits.
• Code is rendered as text with proper encoding; nothing generated is injected into the page as HTML.
• Prompt injection in user data or retrieved documents cannot change the assistant’s tools or permissions.
• Every request, attempt and error is logged with a correlation ID.
• Users can see whether code was verified, and on what data.
• Final failures route to documentation or a named team.
• Data sent to the model provider is covered by your contract and privacy notice.

Common mistakes

• Showing the first draft while validation runs. Users copy it before the check finishes.
• Unlimited retries. Costs spike on hard requests, and latency becomes unacceptable.
• Letting the model grade itself. Without execution, the model will often call broken code correct.
• One giant system prompt. Retrieval of relevant examples and reference sections works better than pasting the whole manual.
• No baseline. Without a fixed evaluation set, every prompt tweak is a guess.

Where to start

If you already have an LLM feature that writes code, add execution and attempt-level metrics first. You will learn your real first-try rate within days, and that number tells you whether to invest in examples, error feedback or a different model. If you are planning a new assistant for a DSL or an internal platform, our AI development team can help design the sandbox, retry loop and evaluation set, and our .NET developers build the runtime integration when your platform runs on .NET, as Lokad’s does.