Selective inference instead of every-frame analysis
Codec metadata, scene changes, scene-cut frames, periodic sampling, and expected event timing determine when each model should run.
Production AI and Private Infrastructure
ScheduleSense - Multimodal Broadcast Intelligence and Dynamic EPG
Multimodal broadcast intelligence combining scene-aware selective inference, computer vision, speech, OCR, compact-LLM reasoning, and dynamic EPG reconciliation.
Broadcast monitoringComputer visionSpeech AIMedia indexingEPG systems
The system
ScheduleSense extends an existing broadcast product so it can compare what is actually playing with the planned electronic program guide. The system correlates video codec information, scene transitions, sampled frames, speech, on-screen text, detected logos and advertisements, and the timing of current and upcoming EPG events.
The architecture avoids treating a live channel as an undifferentiated sequence of frames. Codec metadata and scene-change information identify moments where deeper visual analysis is most useful, while periodic samples maintain context between transitions. This reduces unnecessary decoding and inference without discarding the timing evidence needed for broadcast operations.
A Llama 3.1 1B decision layer receives structured signals rather than raw television video. It evaluates OCR, logo and advertisement detections, Parakeet speech output, current-program context, and next-event timing to determine whether observed content indicates that the planned schedule should be reconciled.
Engineering relationship
Over a two-year program, six OPTIME engineers added major AI and content-intelligence functionality to the customer’s existing broadcast product. The team integrated real-time media processing, selective GPU inference, multiple model families, schedule state, metadata generation, and notification of the customer’s surrounding infrastructure.
The work preserved deterministic media and timing responsibilities while introducing probabilistic signals in a bounded decision workflow. The feature set was completed and delivered to the customer; no claim is made about downstream rollout beyond that delivery.
Engineering constraints
- Analyze continuous broadcast streams without applying every AI model to every decoded frame.
- Correlate probabilistic speech, OCR, logo, advertisement, and scene signals with deterministic EPG timing.
- Recognize schedule divergence caused by overruns, shifted programs, and breaking-news events.
- Update program timing and metadata without exposing the customer’s proprietary integration protocol or schemas.
- Operate within GPU video-decoder limits while retaining inference headroom for multimodal analysis.
- Preserve traceable media events and operator-facing context around automated schedule recommendations.
What OPTIME engineered
- A C++ media pipeline built with GStreamer and NVIDIA DeepStream for continuous broadcast processing.
- Codec-metadata and scene-cut analysis used to select high-value frames for deeper inference.
- YOLO26 execution around scene transitions for logos, advertisements, and other relevant visual signals.
- PaddleOCR processing for on-screen text and Parakeet-TDT speech recognition for spoken context.
- Structured multimodal signal fusion aligned to current and next EPG events.
- A Llama 3.1 1B reasoning layer that evaluates structured signals rather than independently watching raw video.
- Dynamic EPG behavior for extending a current event, shifting following programs, or inserting a Breaking News event.
- Metadata describing detected advertisements and content transitions plus notification of the customer’s existing infrastructure.
Architecture
Live broadcast stream
The existing product supplies continuous channel media and associated schedule context.
GStreamer and DeepStream
C++ pipelines perform media handling, hardware decoding, preprocessing, and inference integration.
Codec metadata and scene-cut analysis
Stream metadata and scene transitions identify when deeper analysis is likely to provide useful evidence.
Selected frames and periodic samples
Scene-cut frames and periodic samples preserve context without applying all models to every frame.
YOLO26, PaddleOCR, and Parakeet-TDT
Computer vision, text extraction, and speech recognition produce time-aligned observations.
Structured multimodal signals
Detections, recognized text and speech, timing, and program context become a bounded evidence set.
Llama 3.1 1B decision layer
The compact language model evaluates structured evidence against current and upcoming EPG events.
EPG reconciliation
The system can extend or shift scheduled events or insert a Breaking News event when content diverges.
Updates and notifications
EPG changes, advertisement/content metadata, and notifications reach the customer’s existing infrastructure.
Key engineering decisions
Structured signals before language-model reasoning
Llama 3.1 1B receives OCR, detection, speech, program, and schedule evidence. It does not independently interpret an unbounded raw video stream.
EPG state as part of the inference context
Current and next scheduled events provide temporal expectations that help distinguish an ordinary transition from an overrun or breaking-news interruption.
Decode capacity governs practical density
On NVIDIA L40S, selective inference reduces AI work enough that hardware video-decoding capacity becomes the practical stream-density limit before inference fully saturates the GPU.
Platform evolution
The existing broadcast product evolved from media monitoring into a content-aware system that can reconcile observed programming with schedule state and produce operational metadata.
The customer’s own integration protocols and metadata schemas remain outside the public description.
Verified capability
The completed system combines selective scene-aware analysis, YOLO26, PaddleOCR, Parakeet-TDT, Llama 3.1 1B, and EPG timing to detect meaningful differences between planned and observed broadcast content.
It can extend a current event, shift following events, insert a Breaking News event, update the EPG, generate content-transition metadata, and notify existing customer infrastructure. The implementation was completed and delivered to the customer.
Verified result
The multimodal content-intelligence and dynamic EPG functionality was completed and delivered to the customer.
In the L40S configuration, hardware video decoding becomes the practical stream-density limit before AI inference fully saturates the GPU.
Technology & Engineering Role
- C++
- Performance-sensitive broadcast media, timing, and inference orchestration.
- GStreamer
- Live media pipeline construction, decoding, sampling, and model integration.
- NVIDIA DeepStream
- GPU media analytics and inference pipeline integration.
- CUDA
- NVIDIA GPU execution across the selective multimodal processing path.
- NVIDIA L40S
- Current hardware target for video decoding and multimodal inference.
- Parakeet-TDT
- Speech recognition and spoken-content signals.
- YOLO26
- Logo, advertisement, and relevant visual-signal detection around scene changes.
- PaddleOCR
- Extraction of on-screen text from selected frames.
- Llama 3.1 1B
- Reasoning over structured multimodal evidence and EPG context.
- Video codec metadata
- Scene and stream signals used to avoid unnecessary full-frame inference.
- EPG data
- Planned event timing and program context for schedule reconciliation.