Building practical AI features

Mapping a post to one of 30 fixed categories is classification, not generation. A hosted LLM would handle nuance well, but it costs money per post, adds latency and a network dependency, and sends user text to a third party. For Whistler I trained a fastText model and run it on the device in plain JavaScript: offline, private, and cheap enough to run on every post.

Generative tasks — drafting, summarising, open-ended answers — are where LLM APIs earn their cost. At SeamHealth that work is LLM evaluation and RLHF: prompt design, reviewing responses against rubrics, and human feedback that labels what failed.

Anything with a precise rule — validation, permissions, arithmetic — should be code, not a model. Models belong where the rules are fuzzy, and their output should feed deterministic code that decides what to do with it.

The Whistler classifier returns “uncategorised” when the input has fewer than two tokens, when no words are recognised, or when the top probability is below 0.35. A missing category is a minor inconvenience; a confidently wrong one puts a post in front of the wrong community.

With LLMs, the equivalent is detecting hallucinations and inconsistencies before they reach users — the LLM evaluation and RLHF work I did at SeamHealth.

The classifier originally reported 99% precision on held-out data. When I re-ran the pipeline for this portfolio, I found why. Each training sentence had been expanded into three near-copies — original, lowercased, punctuation stripped — before the random 85/15 split. After normalisation, 2,199 of the 2,238 validation lines also appeared in training. The model was being tested on sentences it had already seen.

Splitting by source sentence first, and then evaluating, gives 60–64% precision@1 across three random splits. Chance is 3.3%, so the model has learned something real — but 99% was the wrong claim. The fix is procedural: split before augmenting, group near-duplicates, and evaluate on real user text before quoting a number.