What you stop paying for
The most obvious benefit of running a local model is the elimination of per-token API costs. LLM providers charge per token — input tokens for the prompt and output tokens for the completion. A typical feedback analysis prompt (the feedback text plus task instructions) might be a few hundred input tokens, with a completion of a hundred or two hundred tokens.
At published rates for capable frontier models, this is a small amount per item — fractions of a cent. But across thousands of items per month, and if you are re-analyzing feedback as categories or prompts evolve, the per-token cost accumulates. Moving to a local model replaces a recurring API bill with the fixed cost of whatever hardware runs the model.
What you start paying for instead
Local inference is not free — it is a different cost structure. The main categories are hardware (purchase or rental), electricity, and maintenance overhead.
If you are running a model on existing hardware — a developer workstation with a capable GPU, or a server that already runs other workloads — the incremental cost is primarily electricity. A GPU running inference at moderate utilization uses meaningful wattage, but spread across a high volume of analysis jobs the per-item electricity cost is negligible.
If you need to purchase or rent dedicated hardware for the model, that upfront or monthly cost needs to be weighed against the API bill you are avoiding. The break-even calculation depends on your feedback volume, the model you would use via API, and the hardware cost in your context.
- Existing hardware with idle GPU — primarily electricity cost, often near-zero per analysis item.
- Dedicated GPU server (rented) — fixed monthly cost; compare against expected API token spend.
- CPU-only inference — runs on commodity hardware but is significantly slower; works for low-volume use cases.
- Cloud VM with GPU — shifts from a per-token model to a per-hour model; may or may not be cheaper depending on volume.
Quality trade-offs at different model sizes
Not all local models are equal. The practical consideration for feedback analysis is that quality degrades as you move to smaller models, and smaller models are typically what run on commodity hardware without a powerful GPU.
For feedback analysis specifically — sentiment classification, pain point categorisation, feature request tagging — the task is structured and relatively constrained compared to open-ended generation. A capable 7B or 13B model with a well-written prompt performs reliably on clear, unambiguous feedback. It will make more errors on sarcastic feedback, domain-specific jargon, or feedback in languages it was not trained heavily on.
The honest position: a 70B model running locally matches or approaches frontier hosted model quality on these tasks. A 7B model is meaningfully weaker but often good enough for the majority of your feedback volume. The VADER fallback is weaker still but handles high-volume sentiment tracking well. Which tier you need depends on how much your downstream decisions rely on the classifications being correct.
| Setup | Hardware required | API cost | Analysis quality |
|---|---|---|---|
| VADER (built-in fallback) | Any CPU | None | Sentiment only; fast and reliable |
| Local 7B model (Ollama) | Moderate GPU or fast CPU | None | Good for clear feedback; weaker on edge cases |
| Local 70B model (Ollama) | High-end GPU (24GB+ VRAM) | None | Near-frontier on structured analysis tasks |
| Hosted API (OpenAI, Anthropic) | None (your server) | Per-token | Frontier quality; provider retention terms apply |
Latency: the hidden cost
Local models are often slower than hosted APIs, depending on your hardware. A frontier hosted API returns a response in one to three seconds for a typical feedback prompt. A local 7B model on a CPU might take ten to thirty seconds. A local 70B model on a good GPU might take three to eight seconds.
For Rereflect this is less critical than it would be for a real-time user-facing app — analysis runs in background Celery jobs, so a user uploading a CSV of feedback is not waiting for each item to process synchronously. The feedback is queued, processed, and the dashboard updates as results come in. Slower inference means the dashboard reflects results later, not that the user experiences a hang.
If you are processing very high volumes (tens of thousands of items) on a slow local model, the queue will take longer to drain. Right-size your model to your volume and hardware, or use the hosted API path for bursts.
Making the decision
The clearest case for a local model is when privacy requirements rule out sending feedback to a cloud API, and you have hardware that can run a capable model without a significant new cost. In that case, you get privacy and zero API cost at the price of some quality degradation relative to frontier models.
The clearest case for a hosted API is when you want the highest analysis quality with no hardware management, and your data handling terms permit sending feedback to the provider. You pay per token but avoid the operational overhead of running a model.
The VADER fallback is the right starting point for any team: it costs nothing, requires no configuration beyond the base stack, and gives you working sentiment data immediately. Graduate to a local or hosted model when you need the deeper categorisation that VADER does not provide.