Why categorization matters more than sentiment
Knowing that 40% of last month's feedback was negative is useful context. Knowing that 40% was negative, and that the dominant theme in that negative feedback was "CSV export failures in the billing module," is actionable. Categorization is what turns a sentiment trend into a product decision.
The categorization problem is harder than the sentiment problem. Sentiment is a single dimension — positive to negative — and a lexicon-based rule system can approximate it reasonably well for short text. Categorization requires understanding what a piece of text is about, matching that meaning to one or more predefined categories, and doing so consistently across thousands of items that are written in different styles, with different levels of detail, by different people.
Rereflect approaches this with a two-layer pipeline: a lightweight keyword pass for speed and a LLM pass for depth.
Layer 1: keyword-based classification
The first layer uses TF-IDF-weighted keyword matching to make fast preliminary categorization decisions. Each category in your taxonomy is associated with a set of terms, and incoming feedback is scanned for those signals. Matches above a confidence threshold generate a preliminary label.
Keyword matching is not sophisticated — it cannot understand context, handle synonyms gracefully, or catch the same concept expressed in five different ways. But it is fast, it is deterministic, and it is surprisingly effective for the subset of feedback that mentions the thing it is about directly. A ticket that says "the export button crashes" does not require a language model to recognize that it is about the export feature.
For items where keyword signals are ambiguous or absent — which is a meaningful fraction of real-world feedback — the keyword pass flags them for the LLM layer rather than forcing a low-confidence label.
Layer 2: LLM-based classification
The second layer sends feedback items (and their preliminary keyword signals) to the configured language model for deeper classification. The prompt includes your custom category taxonomy and descriptions, the raw feedback text, and any preliminary keyword signals from the first pass.
The LLM can handle things keyword matching cannot: paraphrasing, implied meaning, multi-topic feedback, and context-dependent categorization. A piece of feedback that says "I keep having to redo things after navigating away" is about state persistence or navigation, not about any single keyword — but an LLM with the right category descriptions can recognize that.
The tradeoff is cost and latency. Every LLM call consumes tokens and takes time. Rereflect uses the keyword layer to avoid sending items that can be classified confidently without a model, reserving LLM calls for the harder cases.
- Pain point classification — maps complaints and friction signals to your defined pain-point categories.
- Feature request classification — identifies requests for new capabilities and maps them to your feature category taxonomy.
- Urgency scoring — reasons about signals like churn risk, SLA mentions, angry tone, and escalation language to assign an urgency flag.
If no LLM is configured, Rereflect falls back to keyword-only classification and VADER sentiment. You get a coarser signal, but the pipeline still runs completely offline.
What drives accuracy
Categorization accuracy is not a fixed property of the system — it varies based on several factors you have some control over:
- Category description quality — vague category names produce vague classifications. A category described as "slowness, timeouts, and loading delays in the core editor" gives the model much stronger signal than one called "performance."
- Category distinctiveness — overlapping categories create ambiguity. If "billing" and "pricing" both exist and are not clearly distinguished, items that mention cost will land inconsistently.
- Feedback text length — very short items (one sentence or fewer) have less signal for the model to work with. Longer, more detailed feedback generally classifies more accurately.
- Model capability — a stronger model classifies more accurately, particularly on ambiguous or multi-topic items. Local smaller models trade some accuracy for privacy and cost.
- Language — English feedback generally classifies more accurately than other languages, depending on the model you have configured.
Checking and improving your results
No categorization system is perfectly accurate on the first run, and the right response to imperfect results is not to distrust categorization entirely — it is to use the errors to improve the taxonomy.
When you notice that a category is consistently attracting the wrong items, the usual cause is a description that is too broad or that overlaps with another category. Tightening the description or splitting the category tends to fix it. When items that should match a category keep missing it, the category probably needs broader or more varied description language.
The pipeline is a tool, not an oracle. Treat the first round of categorization as a draft that you refine by looking at what it gets wrong. Most teams find that a few rounds of taxonomy refinement produce results they trust enough to act on without reviewing every item.