How do you test AI video search? What the IPVM ranking leaves open

2026-11-09 · Karl Wang

IPVM's 2026 ranking of generative AI models for video surveillance analytics compares 12 foundation models on two tasks: video search and event description. The full methodology is paywalled, so I can't quote its scores — but the two tasks themselves are worth unpacking, because most vendor demos blur them together.

Video search is temporal grounding, not image search

Given a clip and a question like "a person in a red jacket approaching the exit after 10 pm", the model must return a time range, not a frame. The standard way to score that is:

IoU (Intersection over Union): how much the predicted range overlaps the true range. IoU ≥ 0.5 usually counts as a hit.
Recall@1 / Recall@5 / Recall@10: whether the right moment is in the model's top 1, 5 or 10 answers.
False positives on negative queries: questions about events that are not in the video. This is where models hallucinate most.

That last point matters more than the first two. A model that always finds something looks great on positive queries and terrible in production. Every test set I build includes negative queries — for example, "someone climbing over the fence" in a clip where nobody does.

Event description needs a fact checklist, not BLEU

For event description, n-gram scores like BLEU or ROUGE reward fluent text, not correct text. The honest way to score it is a human fact checklist: who, where, what, in what order — and a hallucination rate for details that were never visible.

The test setup I use

Staged clips: 30-second MP4s, one scenario each, with ground truth time ranges marked by hand first.
Tiered queries: easy (one obvious event), hard (sequence — "opens the door, then enters"), and negative (event absent).
One platform, all models: the same clip and the same question go to each model, and results are stored in the same format so they can be compared side by side.
Token and latency logging: a model that is twice as accurate at ten times the cost is a product decision, not just a benchmark win.

That is what this site is: the test platform, opened up. Upload a clip, ask a question, and see the timestamped answer — including the cases where the honest answer is "not found".