Integrating the OpenAI API into a Logistics Platform: Architecture, Costs, and Lessons 

Most "we added AI" blog posts stop at the chat bubble. This one is about the boring stuff that decides whether an OpenAI integration actually helps operators or just becomes a demo you have to keep resetting.

I built and shipped OpenAI features inside a US logistics platform — dispatcher tooling, operator Q&A, structured extraction — and the lessons below are what I would tell someone about to do the same.

1. Decide what the LLM is actually for

There are three useful shapes for LLMs in an operational product:

1. Conversation over your data — operator asks a question, LLM answers using your database.
2. Structured extraction — take messy input (PDF, email, chat) and return typed JSON.
3. Assistive drafting — draft an update, a message, a summary. Human edits and sends.

Anything outside these three is usually a science project. Pick one and ship it well.

2. Ground responses in your data — always

The single most damaging mistake is letting the model answer from its training data when the user assumed it was answering from their data. Solve it with retrieval:

• Turn the operator question into a small SQL query or a vector search over your domain (loads, drivers, invoices).
• Pass the retrieved rows into the prompt as authoritative context.
• Instruct the model to answer only from that context, and say "I do not know" otherwise.

The rule I use in the system prompt:

Answer only from the CONTEXT block below. If the answer is not there, say so and offer to run a different search.

This one instruction removed 80% of the "the AI made something up" reports we got.

3. Use structured outputs for anything downstream

If the response feeds another system — a database insert, a UI card, a state machine — do not parse free text. Use structured outputs:

const completion = await openai.chat.completions.create({
  model: "gpt-4.1-mini",
  messages,
  response_format: {
    type: "json_schema",
    json_schema: {
      name: "load_summary",
      strict: true,
      schema: {
        type: "object",
        properties: {
          origin: { type: "string" },
          destination: { type: "string" },
          weight_lbs: { type: "number" },
          notes: { type: "string" },
        },
        required: ["origin", "destination"],
        additionalProperties: false,
      },
    },
  },
});

Strict JSON schema plus a validator on your side means the output is either usable or clearly broken — no half-parsed strings quietly writing bad data to your database.

4. Cost control is a product decision

Every model call has a cost and a latency. Budget both up front:

• Cache aggressively. If two operators ask the same question about the same load in the same minute, one LLM call is enough.
• Cascade models. Try a small/cheap model first; escalate to a bigger one only if the small one is unsure.
• Trim context. More context ≠ better answers. Retrieve the top 3–5 rows, not the top 50.

For internal tooling, a per-user rate limit (e.g. 60 requests/hour) is not user-hostile — it is how you keep the invoice predictable.

5. Streaming for UX, batching for jobs

• Chat/Q&A — use SSE streaming. First token in ~800 ms feels instant even if the full response takes 6 seconds.
• Batch extraction over 10,000 rows — do not stream. Run it as a background job with concurrency limits and retries (see the background workers guide).

6. Logging and evaluation

Store every prompt/response pair (with a retention policy your legal team is happy with). Then:

• Sample 50 pairs a week and label them: correct, wrong, unhelpful.
• Track that as a metric next to latency and cost.
• Build a small offline evaluation set for the top 20 questions and re-run it whenever you change the prompt or the model.

Without this loop, "the AI got worse" becomes an unfalsifiable feeling and you cannot fix it.

7. Safety and boundaries in a B2B context

A dispatcher tool does not need "creative freedom." Constrain hard:

• No arbitrary code execution unless you literally need it.
• No PII leaving your infrastructure. If your data is regulated, self-host the retrieval and only send anonymized context to the model.
• A hard allowlist of intents. If the user asks something outside the allowlist, respond with a clear "I cannot help with that here" and route them to the human channel.

8. Where AI actually moved the needle

For the logistics platform, three surfaces were worth it:

• Operator Q&A over live data — cut lookup time for "where is this load" from 90 seconds to 8 seconds.
• Automated summary of email threads attached to a load — pinned to the top of the record for context.
• Structured extraction from carrier-submitted PDFs — replaced a manual transcription step.

Everything else we tried (autonomous decisions, "AI-powered pricing") was either a demo or a lawsuit waiting to happen.

Closing thought

LLMs are not a feature; they are a component. Treat them like any other backend dependency with cost, latency, failure modes and quality metrics. The teams that ship real AI value are the ones doing the boring parts well.

If your product has a backend and you want to ship a well-scoped AI feature — get in touch. I have done the boring parts.