TAPE: Computer vision analysis for combat sports

(ML, Computer Vision) A system that turns ordinary MMA broadcast footage into verified, coach-ready fight analytics.

TAPE watches MMA fights the way an analyst would, from a single ordinary broadcast video, and turns them into deep insights: what does a fighter throw after a certain punch? how long is the window to counter him after a jab?

This post is a brief write-up of what it took to build the CV pipeline, the training and evaluation harness, the labeling tool, and the product site.

How it works

Under the hood it's a multi-step pipeline:

video → identity → pose → strike detection → classification → analytics

To go from a single YouTube video link to analysis takes multiple layers and different types of ML models, each serving their purpose. The first layer is identity.

Identity figures out who is who, distinguishing fighter A from fighter B. Once we're confident in our naming assignments of each bounding box, we can extract the pose from each fighter, giving us a "stick-figure" representation.

The next layer is the detector, which flags anything that looks like a strike, deliberately over-firing (around 90 to 98 percent recall) so it misses almost nothing. Simply put, it surfaces anything that remotely could be a strike, giving us a lot to clean up.

The classifier layer is the one that cleans up that stream, and analytics computes tendencies with confidence intervals and sample sizes on everything. Every stage is allowed to say "I don't know" instead of guessing.

The cool thing is that with every detected strike, the website links it back to the exact timestamp of the original video with an "Overlay AI Intelligence" toggle that draws the pose skeletons and tracker boxes over the broadcast frame, one color per fighter. I'd imagine that this by itself is useful to coaches (like film study).

As for running this pipeline, a human sits in the middle on purpose. I've built an internal labeling tool that plays each flagged moment in slow-motion so that I (or anyone else) can label what it is (strike vs not strike or what kind of strike it is).

The technical lessons

• Measured, not assumed: In the beginning, I accidentally spent time consuming what I thought were foot keypoints for kick detection. They were face points. I only caught it when I checked the skeletal pose render overlaid on the source video, forcing me to check the index map.
• Simple beat fancy, repeatedly: It's tempting to go "model shopping". I put five state-of-the-art models against each other to see how they performed in certain tasks, especially the identity layer. The models lost a bake-off to a deterministic gate, a column swap, and a segmentation mask. The one model that earned its keep, SAM 3.1, only runs when a cheap gate flags a fight as too messy for anything simpler.
• Confidence scores lie if you let them: HMR2 reports 1.0 confidence on every frame because it makes 3D meshes regardless of whether the joint anchor confidence is high or low. It also gives every fighter identical 0.25-meter forearms, including to Jon Jones, one of the longest reach in UFC history. A model that can't say "I don't know" will eventually be wrong with high conviction.
• The pass bar gets set before the run: I found how important it is to set a passing bar before the test. Don't try to make the test results fit what you want.
• Small data needs bigger checks: the classifier corpus is a few thousand human verdicts, small enough to convince you to adopt certain approaches. Everything runs through nested leave-one-fight-out cross-validation with the bar set in advance.
• Cloud GPUs bill you for every mistake: If you're going to run paid inference on the cloud, make sure to have recoverability, visibility, and non-overwriting checks in your process. One time, I ran a batch overnight, and a 1.5 hour limit that I arbitrarily set per run killed a job at 93 percent complete. Dry-run defaults and resumable checkpoints are the rules in the repo now.

The product lessons

• Most stats say "insufficient data" on purpose: 447 of the site's 724 stat cells currently do, each with the reason and what would unlock it. I would rather have data that doesn't lie than show fighters/coaches their analysis that isn't true. Fights have real consequences.
• The proof: confidence intervals, sample sizes, and the full audit trail of each clip/timestamp stay visible at all times. For footage that's available publicly, the website opens a picture-in-picture viewer of its receipts.

• No number reaches the screen without provenance: the build fails if any stat-rendering component contains a digit character in its source, so every number must arrive as a string from the verified data layer. A well-meaning UI edit can never invent or round a statistic. The AI analyst follows the same rule: it never computes, it only routes plain-English questions to a closed registry of verified queries.

• Per-round precision would be dishonest: counts sit on a Poisson floor of roughly plus or minus 20 percent, so the product speaks in trends, not point estimates. Kick rates only count minutes where legs are actually visible (about 18 percent of frames crop them out), and ground truth is disclosed as one annotator's judgment, missing an estimated 11 to 15 percent of strikes. Every count is a floor.

Where it stands

This project showed me the unglamorous, yet necessary side of ML projects. Although it depends on what kind of data you need, for this project, I had to tediously hand label to get things started. Before I had any gradient boosted tree, or my custom trained identity models, or 3D-CNN strike detection models, I had to label things myself to get it off the ground.

Thinking long term, the "ideal" result could be a model that ingests pure RGB video and is able to discern everything from scratch - where the fighters are, where the cage is, what's the camera angle, what strike is this, did a strike land, etc. I think this because pose-based models are lossy. Although pose-based approaches are great to get started with, you lose some critical information by distilling a fighter down to a stick figure.

Ideally, a model that takes pure RGB can see that a strike landed because of the splatter of blood upon impact, which is information that isn't included in pose-based approaches. But these types of projects tend to have flywheel effects. Start with a lightweight approach, use a small custom model to help you make faster decisions, which help to make better decisions in the future with bigger and better trained models.

Tech stack

Python, PyTorch, YOLO26x-pose, Meta Sapiens on Modal H100s, SAM 3 for escalations, CatBoost, Gemini as a feature extractor, and a Next.js + React site with server-rendered SVG charts, Supabase auth and storage, Claude API for the analyst, deployed on Vercel.

See it live

TAPE is live in private preview. The fighter directory and per-fighter reports are open to explore, so you can see how a broadcast becomes a verified tendency profile; the AI analyst and the contributor tools stay invite-only. Have a look.