Watching the Model Miss
I was feeding my LLM clean, accurate audio data. The responses were still wrong — not factually, but interpretively. Here's the system I built to fix that.
The AI had the right data. It knew the chord progression, the rhythmic density, the timing. It called the harmony "wandering," even though for this kind of song (a waltzy 6/8 meter) that wasn't really the most prominent feature. Here's the walkthrough of the system I built to fix that.
That's not a model failure. It's not a prompt failure. It's what happens when you hand structured, dense, time-series data to an LLM without a frame — it attempts to narrate what it sees, confidently, and misses what any musician would immediately hear.
I stopped tweaking the prompt and started reshaping what the model received.
The two-layer harness
The fix came down to two layers that run before the model sees anything.
Layer 1 — A deterministic heuristics toolkit. When a user selects a section in StemSense, the system doesn't dump raw audio features into the prompt. It looks at what's selected. Drums? Analyze onset density, rhythmic stability, groove. Vocals? Pitch contour, phrasing. Multiple stems? Cross-stem analysis — beat alignment, register separation, density ratios. The key was also to have these deterministic tool output qualitative descriptions, not just number sequences.
Layer 2 — A framing and routing policy. The tool outputs are useful, but the model still had no frame of reference — no sense of what to focus on. The second layer looks at the measurements and asks: what's actually prominent here? It strips out everything that isn't the story of this section and builds focused context. For the waltz, instead of every harmonic feature weighted equally, the model receives: rhythmic stability is the headline, these chord movements are stylistically intentional.
Same model. Same prompts. Completely different output quality.
Why this matters beyond music
Audio data is time-series data. If you're feeding any kind of structured, sequential data to an LLM — events, logs, sensor output — the same problem exists. The model will narrate what it sees. It won't know what matters unless you tell it.
The solution isn't a better prompt. It's deciding, upstream, what the model should pay attention to — and encoding your domain expertise into that decision. Strict contracts over loose ones. Layers of abstraction over walls of data.
I experimented with the traces, looked at what variables actually moved output quality, and let that drive the design. Domain knowledge as a musician did a lot of the heavy lifting. But the pattern is transferable to any domain where raw data looks meaningful but requires expertise to interpret.
If you're building AI tools where domain expertise shapes the architecture, I'd like to hear what you're working on — get in touch.