Automatic tracking on a PTZ camera
Automatically detecting, tracking and following vessels using an existing electro-optical system.
Thales LAS · Apprenticeship
- Role
- Apprentice engineer, design and build of the pipeline
- Setting
- Proof of concept in an R&D environment
- Sensor
- PTZ camera, electro-optical system (EOS)
- Environment
- Linux machine with no internet access
This project is carried out in an industrial setting. The description deliberately stays at the level of method and publicly available technologies: no data, no figures and no architecture details are published.
Key points
- Full closed-loop pipeline: capture, detection, tracking, camera control
- A DETR-family detector trained on an image set from the site itself
- Multi-target tracking with ByteTrack and a Kalman filter
- Operating envelope established by measurement, limits included
How it works
- 1
Capture
The camera streams over RTSP. The stream is decoded continuously, then prepared for inference. A single decoder feeds both the display and the analysis: with two, the operator would see a slightly different frame from the one the system processes, which matters when designating a target by clicking on it.
- 2
Detection
An RF-DETR detector locates the vessels, from cargo ship to jet ski. This family emits a fixed number of predictions outright, removing the duplicate-suppression step that anchor-based detectors depend on. The model is trained on imagery from the site, annotated in a locally hosted CVAT, and inference is then compiled with TensorRT to hold real time.
- 3
Tracking
ByteTrack predicts where each target should be using a Kalman filter, then matches new detections to existing tracks. The filter renders a second service: the detection box jitters between frames even on a stationary target, and smoothing gives the control loop a steady set-point rather than a vibrating one.
- 4
Camera control
The offset between target and frame centre becomes a rotation speed, sent to the camera over ONVIF in continuous mode. Two difficulties compound the initial proportional controller: gain depends on zoom, a setting valid at wide angle making the system unstable at full telephoto; and the loop carries a delay, since the controller always acts on information that is already out of date.
Context
Coastal surveillance means spotting the vessels moving through an area and keeping an eye on them. On equipped sites this watch relies on long-range motorised cameras, which an operator steers by hand from a supervision desk. That manual approach, however, ties up one operator per camera, depends on sustained attention that fatigue erodes, and leaves no usable record of what was observed. My assignment is to determine whether this watch can be handed to an automatic processing chain, on the hardware already installed, without degrading the quality of observation the operator expects.
Why the sea is a hard case
Detection on a maritime scene is nothing like detection on roads or indoors. The background is never the same twice, and several difficulties compound.
- Swell creates patterns that move constantly without ever forming an object
- The same class of vessel takes a few pixels far off and much of the frame close in
- Lighting varies widely: backlight, haze, rain, dusk
- Zoom changes how motion is perceived, which rules out a single control setting
- Video definition stays limited given the distances observed
Technical choices
Three criteria guided the choice of every building block, set before any comparison: see tiny targets, hold real time on the available hardware, and be integrable into a product. The third proved the most discriminating.
- The DETR family rather than anchor-based detectors: it avoids duplicate filtering, where too strict a setting merges an aligned buoy and vessel, and too permissive a one lets redundancies through
- RF-DETR builds on a backbone pre-trained without annotations, which means it needs far fewer examples to adapt to an image set assembled in a few weeks
- ByteTrack keeps low-score detections for a second association attempt: when a small vessel moves away and its score drops, its track survives instead of being cut
- Licensing weighed as much as performance. Some blocks that are stronger on paper would force publishing the source of any product embedding them, which is disqualifying for a proof of concept meant to become a product. Every block retained carries a permissive licence
- ONVIF for camera control, because it is a standard: the system will work with cameras other than the one installed
What the system does, and where it stops
Field trials show the system can do multi-target detection,follows a normal-sized vessel for several minutes, in real time, without degrading the image shown to the operator. It does, however, lose the smallest and fastest targets. The operating envelope is therefore known, which is worth more than a system whose point of failure is a mystery.
Finding the weak link
The system loses small, fast targets, but which of the three links is responsible: detection, tracking or control? The answer decides where the next effort goes. To settle it, I compared recordings of successful pursuits with failed ones on two measures: the number of frames where the target goes undetected, and the average offset between target and frame centre. The correlation shows up on the first, not the second. In other words, when the target is detected the camera aims true; it is the moments when it is not that break up the pursuit. Detection availability is therefore what limits the system, and the effort moved from tuning the controller to improving the detector.
- Session recording makes it possible to replay the same scene with two different settings. Since sea and light are never the same twice, a live comparison would prove nothing
- A tuning bench built into the application changes a parameter without restarting.
- Without those two tools the finding would have stayed an impression
What I took away
- Measure before deciding: without a reproducible test protocol you optimise what you can measure rather than what actually limits you
- The full vision pipeline, from building the image set to deploying an optimised inference engine
- Control-theory basics learned in the field, designing a loop that must stay stable despite its own delay
- A criterion I had never considered: licensing, which can make a high-performing library unusable
- Packaging an application for an isolated machine, where nothing can be downloaded at install time