TrackPulse: Trading a GPU Pipeline for One Multimodal API Call

The problem: screening audio in bulk shouldn't need a GPU

TrackPulse answers a narrow but genuinely annoying question: given a batch of audio tracks, which ones have vocals, what language are they in, and do any of them contain explicit content? That's a real gate in music distribution and licensing workflows — someone has to sort a folder of WAV files before they can be catalogued, licensed, or published, and doing it manually by ear doesn't scale past a handful of tracks.

The first two versions of this project solved it the "correct" way: a local ML pipeline running silero-vad for voice activity detection and faster-whisper for transcription, then a separate profanity pass on the transcript. It worked, but it meant shipping a Python inference stack, managing model weights, and ideally a GPU to keep latency sane. That's a lot of infrastructure for what is, at its core, a batch classification job.

Version 3 throws all of that out and replaces it with one Gemini 2.5 Flash call per track. No VAD model, no transcription model, no GPU — the audio bytes go in as inlineData, a JSON classification comes back. That's the whole pipeline.

Architecture: unzip in memory, stream results as they land

The upload contract is simple: users drop a ZIP of WAV files onto the page, and the API extracts and classifies each one. The interesting part is how results get back to the browser.

// src/app/api/analyze/route.ts
export const maxDuration = 300;

export async function POST(request: Request) {
  const buffer = Buffer.from(await file.arrayBuffer());
  const zip = await JSZip.loadAsync(buffer);

  const wavEntries = Object.entries(zip.files).filter(
    ([name, entry]) =>
      name.toLowerCase().endsWith(".wav") &&
      !name.startsWith("__MACOSX") &&
      !entry.dir,
  );

  const stream = new ReadableStream({
    async start(controller) {
      controller.enqueue(jsonLine({ type: "start", total: wavEntries.length, filenames }));

      for (let i = 0; i < wavEntries.length; i++) {
        controller.enqueue(jsonLine({ type: "progress", index: i, filename, status: "processing" }));
        const track = await analyzeTrack(wavBuffer, filename);
        controller.enqueue(jsonLine({ type: "result", index: i, track }));
      }

      controller.enqueue(jsonLine({ type: "done" }));
      controller.close();
    },
  });

  return new Response(stream, { headers: { "Content-Type": "application/x-ndjson" } });
}

A batch of 20 tracks against a synchronous JSON endpoint means the user stares at a spinner for however long the slowest possible response takes — could be minutes. Instead, the route holds a single connection open and emits newline-delimited JSON events (start → progress/result per track → done) as each classification finishes. The client reads the response body as a stream, splits on newlines, and updates one TrackCard at a time:

// src/hooks/useAnalysis.ts
const reader = response.body!.getReader();
let buffer = "";

while (true) {
  const { done, value } = await reader.read();
  if (done) break;

  buffer += decoder.decode(value, { stream: true });
  const lines = buffer.split("\n");
  buffer = lines.pop()!; // keep the trailing partial line for the next chunk

  for (const line of lines) {
    const event: StreamEvent = JSON.parse(line);
    // dispatch on event.type: start / progress / result / error / done
  }
}

The lines.pop() line matters more than it looks — NDJSON chunks don't arrive newline-aligned, so the last element of every split is a partial line that has to be carried forward instead of parsed. Get that wrong and you get sporadic JSON.parse failures on large batches that are nearly impossible to reproduce locally.

This is also why maxDuration = 300 is set explicitly on the route: without it, a serverless platform's default timeout would kill a large batch mid-stream long before the last track finishes.

Trusting an LLM's JSON, but not that much

Gemini is asked for a strict schema — vocal, confidence, language, explicit, words_found — with responseMimeType: "application/json" to constrain the output format. But "constrained" isn't "guaranteed," so every field gets validated and coerced before it's trusted:

return {
  filename,
  status: "done",
  vocal: typeof raw.vocal === "boolean" ? raw.vocal : false,
  confidence: typeof raw.confidence === "number"
    ? Math.max(0, Math.min(1, raw.confidence))
    : 0,
  language: raw.vocal && typeof raw.language === "string" ? raw.language : null,
  explicit: typeof raw.explicit === "boolean" ? raw.explicit : false,
  words_found: Array.isArray(raw.words_found) ? raw.words_found : [],
  error: null,
};

Confidence gets clamped to [0, 1] even though the prompt already asks for that range — models drift, and a UI rendering 147% confidence is a worse failure mode than silently clamping. language is only trusted if vocal is also true, so an instrumental track can't end up mislabeled with a stray language guess.

Transient failures (429, 500, 503) get one retry with a fixed backoff before the track is marked error and the batch moves on — one bad track shouldn't take down the other nineteen:

const isRetryable =
  lastError.message.includes("429") ||
  lastError.message.includes("500") ||
  lastError.message.includes("503");

if (attempt === 0 && isRetryable) {
  await new Promise((r) => setTimeout(r, 2000));
  continue;
}

What v3 gave up, and what it gained

The honest trade-off: v3 is now dependent on an external API's uptime and pricing instead of infrastructure under direct control, and per-track cost scales with usage instead of being amortized over owned hardware. For a tool that processes batches on demand rather than continuously, our team judged that trade worth it — the operational simplicity of "no model management, no GPU, deploy anywhere Next.js runs" outweighs the recurring API cost for this use case. It's the kind of call that depends entirely on request volume, and it's worth revisiting if that volume changes.

The rest of the stack

Everything above the API route is intentionally boring: Next.js 16 with React 19, Tailwind CSS 4, and shadcn/ui components (Card, Badge, Progress) built on Radix primitives. Per-track and total elapsed-time counters come from a small useElapsed hook rather than a timer library, and next-themes handles dark/light mode with the preference persisted across sessions. None of that needed to be more sophisticated than it is — the interesting engineering in this project is entirely in the streaming contract and the LLM-output validation, not the UI chrome.

Takeaway

TrackPulse is a case study in recognizing when a bespoke ML pipeline is solving a problem that a well-prompted multimodal model already solves — and in not trusting that model's output at face value once you make the switch. The streaming architecture (NDJSON over a single held-open response, reconstructed carefully on the client) is what makes a multi-minute batch job feel responsive instead of like a hung request, and it's a pattern our team applies well beyond audio: any workload where you're fanning out to a slow external API benefits from streaming partial results back to the user as they complete, rather than making them wait for the whole batch.