TAPE: What I Learned Turning Fight Broadcasts into Numbers a Coach Can Trust

(ML, Computer Vision) A system that turns ordinary MMA broadcast footage into verified, coach-ready fight analytics. Here's the product, the stack, and the lessons that cost me real money and sleep.

TAPE watches MMA fights the way an analyst would, from a single ordinary broadcast video, and turns them into numbers a coach can trust: what a fighter throws, what he goes to after the jab, how his output changes round by round. Every number links back to timestamps on the tape. When the data can't support an answer, the site says so instead of guessing.

This post is the honest version of what it took to build: the computer vision pipeline, the training and evaluation harness, the labeling tool, and the product site.

How it works

Under the hood it's a five-stage pipeline:

video → identity → pose → strike detection → classification → analytics

Identity figures out who is who. Pose extracts where every body part is. The detector flags anything that looks like a strike, deliberately over-firing (around 90 to 98 percent recall) so it misses almost nothing. The classifier cleans up that stream, and analytics computes tendencies with confidence intervals and sample sizes on everything. Every stage is allowed to say "I don't know" instead of guessing.

You can watch the pipeline see. Every clip in the product has an "Overlay AI Intelligence" toggle that draws the pose skeletons and tracker boxes over the broadcast frame, one color per fighter.

A human sits in the middle on purpose. My review tool plays each flagged moment in a slow-motion loop and takes a verdict in a single keystroke, about 7 or 8 per minute. The published stats come only from what a human confirmed, and every one links back to its moments on tape.

The technical lessons

Measure the input before you build on it. Every time I finally measured something I had assumed, it was worse than assumed. The funniest version: I spent a while consuming what I believed were foot keypoints for kick detection. When a render finally forced me to check the index map, they were face points. Nothing gets consumed without a ten-frame overlay render now.

The simple thing kept beating the fancy thing. I ran a bake-off that entered five state-of-the-art models against my measurement harnesses. The winners were a 200-line classical tracking gate, a column swap, and a segmentation mask. CoTracker3, a point tracker I was excited about, failed on 87 percent of easy controls. The one expensive model that earned a job, SAM 3, works like a fire department: the cheap gate self-diagnoses when a fight is too messy for it, and only those fights go to the cloud.

Don't trust a confidence score that never drops. HMR2, the 3D mesh model, reports confidence 1.0 on every frame forever. Under occlusion it invents a plausible limb and swears by it, and its body template gives every fighter identical 0.25 meter forearms, including the longest-reach fighter in UFC history. A model that can't abstain will eventually lie to you at full confidence, and everything downstream inherits the lie.

Every automatic fixer I built made things worse. Identity is the foundation of the pipeline, since one mistake poisons every layer above it, and my worst fight had 105 identity swaps per minute. But every automated swap-fixer I built or borrowed (a geometry gate, an embedding corrector, even the foundation tracker outside its certified windows) flipped fighters that my own eyes confirmed were already correct. The fix that survived is deliberately boring: only re-slot identities on frames a certified escalation actually covered, and hard-fail the pipeline if anything else tries. That took the worst fight from 105 swaps per minute to zero.

Decide the pass bar before the experiment runs. Every experiment states its acceptance bar up front, and the verdict comes from the numbers, not from how I feel about them. This killed my favorite headline claim. I wanted to show that the system sees Topuria's famous third-round surge: ground truth says his output ramps 2.14x, but the machine measured 1.12x, four runs in a row. The reason turned out to be physics. At 30fps, strikes in a finishing flurry merge into one velocity peak, so no candidate ever forms. That's a capture limit of broadcast footage, not a tuning problem, so the product discloses it instead of claiming it. Shipping the refusal was the right call, and it stung.

Small data will happily fool you. The classifier corpus is a few thousand human verdicts, small enough that leakage and threshold games can hand you a fake champion. So: nested leave-one-fight-out cross-validation, paired bootstrap on the deltas, and one pre-registered bar. CatBoost earned its spot by winning six out of six folds with a confidence interval that excludes zero.

LLMs make good witnesses and bad judges. As a standalone strike judge, Gemini ran about 27 percent false positives, useless as a referee. But fed into the classifier as one feature among many, its verdicts passed every adoption bar and got selected first in every fold. My favorite micro-result: giving the model a slow-motion "replay" clip made it strike-happy and cost 14 points of accuracy.

Cloud GPUs bill you for every lesson. SAM 3 leaked past 90GB of RAM and froze my Mac Studio, so heavy models run on Modal now. A missing skip-if-exists guard silently relaunched a batch and burned money. A 90-minute function cap killed a pose run at 93 percent complete, and the job restarted from scratch. Each mistake is now a rule in the repo: dry-run by default, hard budget caps, per-fight paths, resumable checkpoints.

The product lessons

Most of the site says "insufficient data," on purpose. The pack behind the site has 724 stat cells, and 447 of them currently say "insufficient data" with the reason and what would unlock them. A coach who catches you bluffing once will never trust the sheet again, so refusals are designed as real states: a stat that needs more data shows a progress meter toward its pre-registered gate, and a thing the broadcast physically cannot support says so in a visually distinct way.

The claim leads, the proof sits one click down. A coach wants the read: what does this guy do, what does it open. A skeptic wants sample sizes, confidence intervals, and provenance. So the page leads with the claim, keeps the evidence line (CI and n) always visible, and stages the full audit trail underneath, from timestamps down to the registered upload they index. Every stat opens a picture-in-picture viewer where its receipts play as a timestamped list of moments on the tape.

No number reaches the screen without provenance. The build fails if any stat-rendering component contains a digit character in its source, so every number on screen must arrive as a string from the verified data layer. A well-meaning UI edit can never invent, round, or "fix" a statistic. The AI analyst lives under the same rule: it never computes, it routes plain-English questions to a closed registry of verified queries and explains the results. If the data can't answer, it says why.

Per-round precision would be dishonest. Per-round counts sit on a Poisson floor of roughly plus or minus 20 percent, so the product speaks in trends and conditionals instead. Rates get honest denominators too: kick rates only count minutes where legs are actually visible, because about 18 percent of broadcast frames crop them out. And the site discloses that ground truth is one annotator's judgment, which misses an estimated 11 to 15 percent of strikes, so every count is a floor.

Where it stands

Twenty-two fighters on file, about 1,250 hand-logged strikes and a few thousand adjudicated detector candidates, a live site with per-fighter reports and a printable scout sheet, and a contributor labeling flow that quietly measures reviewer accuracy against known answers before verdicts count. It has been a solo build so far, which is exactly why that contributor flow exists. The single-camera constraint turned out to be the point: anyone can buy a multi-camera lab feed, but a system that gets trustworthy numbers out of the footage that actually exists for every fighter is worth building.

Tech stack

Python, PyTorch, YOLO26x-pose, Meta Sapiens on Modal H100s, SAM 3 for escalations, CatBoost, Gemini as a feature extractor, and a Next.js + React site with server-rendered SVG charts, Supabase auth and storage, Claude API for the analyst, deployed on Vercel.

See it live

TAPE is live in private preview. The fighter directory and per-fighter reports are open to explore, so you can see how a broadcast becomes a verified tendency profile; the AI analyst and the contributor tools stay invite-only. Have a look.