traffic-video-analysis
A Python library that turns drone footage of a traffic jam into measured stop events — stabilization, tracking, road-frame kinematics, overlays.
The problem: the jam wave is visible, but nobody measured it
EP 03 is about phantom jams — traffic waves with no accident, no bottleneck, no
cause except drivers reacting to drivers. The hero clip is 17.04 seconds of aerial
4K drone footage (3840×2160, 25 fps, 426 frames) of a dense jam, shot while the drone
pans and rotates about 70°. The wave is right there on screen. The problem is that
"right there on screen" is not a number: to say how fast the wave moves you need every
car's position in road coordinates, over time, while the camera itself won't hold still.
That is what tva exists to produce. It was built for this clip first and
is designed to grow across episodes.
How it works: world plane first
The pipeline's core assumption is that a (locally) planar road seen from the air
means camera motion is exactly a per-frame homography. So: stabilize by estimating
per-frame homographies into a reference plane from tracked feature points, with RANSAC
rejecting the moving vehicles (in a jam, most cars are static anchors anyway — about
1,300 inliers per frame on the hero clip). Detect and track with YOLO + ByteTrack on
the native-resolution frames. Then push every detection through its frame's homography
into the reference plane, where camera pan cancels and a stopped car becomes a fixed
world point. Speeds come from a constant-acceleration Kalman filter with an RTS
smoother per track — velocity is a filter state, never a finite difference — with
measurement noise scaled by the local homography Jacobian and re-weighted from
residuals. A road centerline derived from the trajectories themselves gives each car
a 1-D arc-length coordinate s(t), and moving/stopped hysteresis turns
that into stop-onset events with sub-frame timing. Every stage writes plain JSON and
PNG artifacts to a work directory, so each step is inspectable and cacheable.
What's real today
Version 0.1.0, and it has exactly one production run on the books — which the repo documents honestly. On the EP-03 hero clip, VisDrone-trained YOLOv8x weights found 1,541 tracks and 94,486 observations (roughly 170–300 vehicles per frame), resolving to 726 world tracks and 154 stop-onset events. A COCO-trained model (yolo11m) found 4 detections on a frame where the VisDrone weights find ~170 — the README is blunt that this is a domain gap, not a scale problem, and that tiling does not rescue it. The full run took about 3 hours from raw clip to finished focus-lane overlay; the repo's retro doc records where the time went and sets a 1.5-hour target for the next clip of the same shape. The roadmap lists what is not done yet: wave-front fitting through the stop events to get an actual wave speed, long-range homography refinement against drift, and a browser annotator for verifying stop events. Hand annotation exists but is explicitly a refinement layer on top of measured geometry, not the source of it.
Run it yourself
Python 3.10+, with numpy, OpenCV, scipy, Pillow, and ultralytics. The pipeline is one CLI subcommand per stage, each reading and writing the same work directory:
python -m tva init <video> --work <dir> python -m tva stabilize --work <dir> python -m tva detect --work <dir> --model weights/visdrone-yolov8x.pt --imgsz 1920 python -m tva world --work <dir> python -m tva spacetime --work <dir> # space-time diagram: shockwaves as diagonals python -m tva render --work <dir> # speed-colored overlay video
The VisDrone weights are not committed; the README links the HuggingFace download.
If you only read one file, make it examples/hero-wave.md — the full
EP-03 run, commands and numbers included.
The full README, verbatim
traffic-video-analysis (tva)
Reusable library for analyzing traffic footage: camera stabilization, vehicle detection/tracking, road-frame kinematics, and overlay/diagram rendering. Built for Math vs Vibes traffic videos (first user: EP-03 phantom-jam hero clip), designed to grow across episodes.
Core idea: world plane first
Aerial/elevated traffic footage of a (locally) planar road means camera motion is exactly a per-frame homography. So the pipeline is:
- Stabilize — estimate per-frame homographies to a reference plane (frame 0's ground plane) from tracked feature points; RANSAC rejects moving vehicles, and in a jam most cars are static anchors anyway.
- Detect + track — YOLO + ByteTrack on the native-resolution frames → per-frame boxes with persistent ids.
- World kinematics — map every detection through its frame's homography into the reference plane. Camera pan/rotation cancels: a stopped car is a fixed world point. Speeds come from a constant-acceleration Kalman filter + RTS smoother per track (velocity is a state, never a finite difference), with per-observation measurement noise scaled by the local homography Jacobian (projective amplification) and re-weighted from each segment's residuals (motion blur). Moving/stopped hysteresis → stop-onset events with sub-frame timing (the jam wave front, measured).
- Road model — centerline in the reference plane (derived from vehicle
trajectories, refinable by hand annotation) gives a station coordinate
s= arc length along the road. Car state becomes 1-D:s(t). - Render — QA sheets, per-car overlay markers (speed-colored), space–time diagrams where shockwaves appear as diagonal boundaries.
Manual annotation is a refinement layer on top of the measured geometry, not the source of it.
Artifacts
Each analysis lives in a work directory of plain JSON + PNG artifacts so every stage is inspectable and cacheable:
| file | producer | contents |
|---|---|---|
meta.json |
init |
source path, width/height, fps, frame count |
homographies.json |
stabilize |
per-frame 3×3 H (frame px → reference plane), inlier stats |
tracks.json |
detect |
per track: class, per-frame box center/size/conf |
world_tracks.json |
world |
per track: reference-plane positions, smoothed speed, state runs, stop events |
centerline.json |
world |
road spine polyline in the reference plane |
roads.json |
roads |
per-road centerline, lane offsets, measured boundaries |
statics.json |
hand | known-fixed zones (parking lots, depots) where registration drift reads as fake speed; render hides tracks inside |
focus_lane.json |
hand + edit |
hand-picked loud lane: road + offset band, include/exclude click deltas, optional no_backfill ids |
manual_tracks.json |
edit |
hand-keyframed cars (image px); lifted to world at render time, linked forward onto their future real track |
qa/ |
all | visual checks per stage |
Usage
python -m tva init <video> --work <dir>
python -m tva stabilize --work <dir> # cv2 only
python -m tva detect --work <dir> --model weights/visdrone-yolov8x.pt --imgsz 1920
python -m tva world --work <dir>
python -m tva spacetime --work <dir> # space-time diagram PNG
python -m tva speedqa --work <dir> # raw vs smoothed speed profiles
python -m tva render --work <dir> # speed-colored overlay video
python -m tva render --work <dir> --roads --highlight 33,49 # optional layers
python -m tva edit --work <dir> --port 8123 # focus-lane editor (browser)
Focus lane (hero-clip storytelling)
Seed focus_lane.json with a road index + rough offset band, then curate
in tva edit: click cars loud/quiet, shift-click keyframes for missed
cars. The render gives loud cars neon styling inside a spotlight corridor
fitted from their paths, and recovers what detection missed early —
backwards fragment stitching, held-position backfill for already-stopped
cars, manual cars linked forward onto the real track they become. The
editor runs the same recovery pass, so what you see is what renders.
Process notes + speedup backlog: docs/retro-ep03-hero.md.
render draws cars only by default (--roads adds the inferred road/lane
overlay; --highlight enlarges + outlines specific track ids for
storytelling). anchors re-registers each frame against provably-static
vehicles near the analysis centerline — same-plane landmarks, because an
elevated carriageway parallax-shifts against the ground plane whenever the
camera translates.
Model choice matters. COCO-trained YOLO (yolo11m etc.) is near-blind to
nadir/top-down aerial vehicles (4 detections on a frame where VisDrone
weights find ~170; tiling does not help — it's a domain gap, not a scale
problem). For drone footage use VisDrone-trained weights:
weights/visdrone-yolov8x.pt, from
https://huggingface.co/mshamrai/yolov8x-visdrone (not committed; re-download
if missing). detect picks vehicle classes by name so COCO and VisDrone
models both work unmodified.
See examples/hero-wave.md for the EP-03 phantom-jam run.
Roadmap
- Browser annotator (port from ep-03 hero-wave) reading/writing these schemas: verify/nudge stop events, refine centerline, mark lanes.
- Wave-front fitting through stop-onset events → wave speed (km/h via lane-width or car-length pixel scale).
- Anti-drift: long-range homography refinement against the reference frame (current v1 chains adjacent frames; QA = stopped-car world jitter).
- Detection masks feeding back into stabilization feature selection.
- Overlay renderers that compose into episode ffmpeg pipelines.
Repo post · repo created 2026-07-31, posted 2026-08-06. Wave-front fitting is still on the roadmap, so the jam's official speed remains "vibes." The 154 measured stops disagree.