questionable TP alignment of TD annotation

[owencking]

Bug Description

I am worried about whether the TP to which the TD is aligned is the same TP that was the source of the image processed by the VLM.

I ran the captioner on input MMIF from SWT. It produced this in its output view:

        {
          "@type": "http://mmif.clams.ai/vocabulary/TextDocument/v1",
          "properties": {
            "document": "d1",
            "origin": "v_1:tf_18",
            "provenance": "derived",
            "mime": "application/json",
            "text": {
              "@value": "AL GALLETTA BLUEBERRY GROWER",
              "@language": "en"
            },
            "id": "v_2:td_6"
          }
        },
        {
          "@type": "http://mmif.clams.ai/vocabulary/Alignment/v1",
          "properties": {
            "source": "v_0:tp_919",
            "target": "v_2:td_6",
            "id": "v_2:al_6"
          }
        },

Here are the TF and TP annotations that is referenced:

        {
          "@type": "http://mmif.clams.ai/vocabulary/TimeFrame/v6",
          "properties": {
            "label": "chyron and person",
            "classification": {
              "chyron and person": 0.9387489855289459
            },
            "targets": [
              "v_0:tp_919",
              "v_0:tp_920",
              "v_0:tp_921",
              "v_0:tp_922",
              "v_0:tp_923",
              "v_0:tp_924",
              "v_0:tp_925",
              "v_0:tp_926"
            ],
            "representatives": [
              "v_0:tp_919"
            ],
            "timeUnit": "milliseconds",
            "id": "v_1:tf_18"
          }
        },

        {
          "@type": "http://mmif.clams.ai/vocabulary/TimePoint/v5",
          "properties": {
            "timePoint": 459026,
            "label": "IN",
            "classification": {
              "GLOTW": 3.5924065741710365e-05,
              "CR": 3.7618651731463615e-06,
              "IN": 0.947616696357727,
              "KU": 0.001030643587000668,
              "B": 2.926498436818542e-25,
              "S": 1.0204522123136162e-11,
              "M": 5.168930283794282e-10,
              "Y": 1.1250751413172111e-05,
              "F": 1.7080129310897973e-08,
              "E": 0.0006756898364983499,
              "P": 0.050621531903743744,
              "-": 4.5278543439053465e-06
            },
            "id": "v_0:tp_919"
          }
        },

However, this is the frame from the video sought for 459026 (and found at 00459025).

Note that there is no text in that image, not even faintly. However, here are some nearby frames

(from 00459892) (from 00461661)

Question: Is it possible that the app is pulling an image from later than the time point it is seeking?

Here is the cataloging aid where we discovered this.

Reproduction steps

Full MMIF file: cpb-aacip-259-9c6s1g2d_NJN_News_pre-1984_2.mmif.json

Media file: cpb-aacip-259-9c6s1g2d.mp4

Expected behavior

The VLM should perform captioning on the exact frame referenced by the TP annotation.

Log output

Screenshots

No response

Additional context

No response

[keighrim]

wait... If the TP alignment was wrong and VLM saw that non-text image, how did it get the OCR test out of it...?

[keighrim]

Isn't this line pulling out the image too soon?
https://github.com/WGBH-MLA/visaid_builder/blob/36437a0d8753bf52eb2e2a7bd41b9a1924137fb3/visaid_builder/create_visaid.py#L168

[keighrim]

import os

from mmif import Mmif, DocumentTypes
from mmif.utils import video_document_helper as vdh

pwd = os.path.dirname(os.path.abspath(__file__))
m = Mmif( open(f'{pwd}/cpb-aacip-259-9c6s1g2d.mmif').read(), validate=False)
vdoc = m.get_documents_by_type(DocumentTypes.VideoDocument)[0]
vdoc.location = f'file://{pwd}/cpb-aacip-259-9c6s1g2d.mp4'
tf = m['v_1:tf_18']
fn = vdh.get_representative_framenum(m, tf)
print(f'frame {fn}')
vdh.extract_frames_as_images(
    vdoc, [fn], as_PIL=True
)[0].save(f'rep_tf18_frame{fn}.jpg')

[keighrim]

closing as non-issue (see clamsproject/clams-vocabulary#15 (comment) possible root cause)

[owencking]

It appears there is a real issue here, but probably not an issue with this app. @keighrim could you please take another look? If the issue is real, maybe it needs to be transferred to another repo?

From @keighrim's experiment above, it appears the MMIF utils functions (the time unit converter, followed by the frame extracter) are pulling the frame 13757.

Here is some code to see the msec time of that frame:

import av

video_path = "./cpb-aacip-259-9c6s1g2d.mp4"
container = av.open(video_path)
video_stream = next((s for s in container.streams if s.type == 'video'), None)

fnum = 1
for frame in container.decode(video_stream):
    ftime = int(frame.time * 1000)
    print(fnum, ftime)
    fnum += 1

Here is the relevant range frame numbers and msec times:

13756 458991
13757 459025
13758 459058
13759 459091

Here are two candidate frames given that we were seeking the frame at 459026ms:

frame number 13757 at 459025ms (1ms before time sought)

frame number 13758 at 459058ms (32ms after time sought)

Notice that the latter is the image output in the message above. However, it is not at the 13757th frame in the video.

What is the discrepancy here? Maybe this is due to a difference between av and cv2?

Again, this is probably not an issue with the SmolVLM2 captioner. Maybe it relates to video_document_helper.py in the MMIF utils?

[keighrim]

Hi Owen, thanks for digging deeper into this. Looking back, I realize my earlier comments pointed at the cause but didn't really spell it out. Let me try to be more explicit this time.

I don't think this is a cv2-vs-PyAV difference; I believe it's an off-by-one in the counting script. That is, "my" frame 13757 is not "your" frame 13757.

In your PyAV script, you start counting at 1:

fnum = 1
for frame in container.decode(video_stream):
    ftime = int(frame.time * 1000)
    print(fnum, ftime)
    fnum += 1

The MMIF video_document_helper, on the other hand, counts from 0:

mmif-python/mmif/utils/video_document_helper.py

Line 136 in 0dc017f

cur_f = 0

So every frame number you reported is shifted by 1 relative to what vdh uses. Your output:

13756 458991
13757 459025    <-- vdh's "frame 13756" (0-indexed)
13758 459058    <-- vdh's "frame 13757" (0-indexed)
13759 459091

What you call "frame 13758" (with the chyron) is exactly what vdh calls frame 13757: the same frame my code snippet extracted. There is no cv2-vs-PyAV discrepancy, both libraries decode the same frames via the same FFmpeg backend (I believe). The difference is purely in your numbering convention.

That being said, we need to understand that frame numbers are not absolute anchor points. More broadly, to my understanding, "frame number" is not even a thing in MPEG-4 (or most video containers). As far as I know, frames are identified by presentation timestamps (PTS), not by sequential indices (e.g. FFmpeg's av_seek_frame operates on timestamps, not indices). A "frame number" is always an abstraction imposed by whatever software is doing the decoding, and different tools can choose different starting points (0 vs 1) or even different counting semantics (decode order vs presentation order, B-frame handling, etc., as observed in clamsproject/clams-vocabulary#15 (comment)). I'm no MPEG expert, but this is my understanding of why comparing frame numbers across tools without first aligning on convention will always be fragile (I learned this from working on clamsproject/clams-vocabulary#15).

To be clear, the original confusion you reported is real and worth investigating. When I closed this issue, I was trying to say that the problem isn't in the captioner or in video_document_helper, but in the inconsistency between how the visaid and the CLAMS apps select frames for a given timepoint. I've opened a proper issue for that: WGBH-MLA/visaid_builder#4

[keighrim]

Also, I noticed that

$ ffmpeg -i cpb-aacip-259-9c6s1g2d.mp4 2>&1
ffmpeg version n8.0.1 Copyright (c) 2000-2025 the FFmpeg developers
  built with gcc 15.2.1 (GCC) 20260209
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-amf --enable-avisynth --enable-cuda-llvm --enable-lto --enable-fontconfig --enable-frei0r --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libdav1d --enable-libdrm --enable-libdvdnav --enable-libdvdread --enable-libfreetype --enable-libfribidi --enable-libglslang --enable-libgsm --enable-libharfbuzz --enable-libiec61883 --enable-libjack --enable-libjxl --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libplacebo --enable-libpulse --enable-librav1e --enable-librsvg --enable-librubberband --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpl --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-nvdec --enable-nvenc --enable-opencl --enable-opengl --enable-shared --enable-vapoursynth --enable-version3 --enable-vulkan
  libavutil      60.  8.100 / 60.  8.100
  libavcodec     62. 11.100 / 62. 11.100
  libavformat    62.  3.100 / 62.  3.100
  libavdevice    62.  1.100 / 62.  1.100
  libavfilter    11.  4.100 / 11.  4.100
  libswscale      9.  1.100 /  9.  1.100
  libswresample   6.  1.100 /  6.  1.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'cpb-aacip-259-9c6s1g2d.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: isom
    creation_time   : 2014-12-21T11:06:11.000000Z
  Duration: 00:28:32.64, start: 0.000000, bitrate: 843 kb/s
  Stream #0:0[0x1](eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, progressive), 480x360 [SAR 1:1 DAR 4:3], 714 kb/s, 29.97 fps, 29.97 tbr, 30k tbn, start 0.033367 (default)
    Metadata:
      creation_time   : 2014-12-21T11:06:11.000000Z
      handler_name    : MP4 Video Media Handler
      vendor_id       : [0][0][0][0]
      encoder         : AVC Coding
  Stream #0:1[0x2](eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 125 kb/s (default)
    Metadata:
      creation_time   : 2014-12-21T11:06:11.000000Z
      handler_name    : MP4 Sound Media Handler
      vendor_id       : [0][0][0][0]

while other videos, for example hichy-comp set,

$ for m in hi-chy-comps/*.mp4 ; do ffmpeg -i $m 2>&1 | grep -E "(start|tbn)" ; done
  Duration: 01:02:58.78, start: 0.000000, bitrate: 787 kb/s
  Stream #0:0[0x1](und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(tv, smpte170m/smpte170m/bt709, progressive), 480x360, 651 kb/s, SAR 45:44 DAR 15:11, 29.97 fps, 29.97 tbr, 30k tbn (default)
  Duration: 00:53:53.36, start: 0.000000, bitrate: 834 kb/s
  Stream #0:0[0x1](eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 480x360, 700 kb/s, SAR 100:99 DAR 400:297, 29.97 fps, 29.97 tbr, 11988 tbn (default)
  Duration: 00:31:57.85, start: 0.000000, bitrate: 748 kb/s
  Stream #0:0[0x1](und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(tv, smpte170m/smpte170m/bt709, progressive), 480x360, 612 kb/s, SAR 45:44 DAR 15:11, 29.97 fps, 29.97 tbr, 30k tbn (default)
  Duration: 00:29:16.51, start: 0.000000, bitrate: 1193 kb/s
  Stream #0:0[0x1](und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(progressive), 480x360, 992 kb/s, SAR 1:1 DAR 4:3, 29.97 fps, 29.97 tbr, 30k tbn (default)
  Duration: 01:03:06.54, start: 0.000000, bitrate: 755 kb/s
  Stream #0:0[0x1](und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(progressive), 480x360, 618 kb/s, SAR 1:1 DAR 4:3, 29.97 fps, 29.97 tbr, 30k tbn (default)
  Duration: 00:59:22.25, start: 0.000000, bitrate: 1514 kb/s
  Stream #0:0[0x1](und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(progressive), 480x360, 1313 kb/s, SAR 1:1 DAR 4:3, 29.97 fps, 29.97 tbr, 30k tbn (default)
  Duration: 00:32:30.18, start: 0.000000, bitrate: 631 kb/s
  Stream #0:0[0x1](und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(progressive), 480x360, 494 kb/s, SAR 45:44 DAR 15:11, 29.97 fps, 29.97 tbr, 30k tbn (default)
  Duration: 00:59:05.00, start: 0.000000, bitrate: 840 kb/s
  Stream #0:0[0x1](eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 480x360, 711 kb/s, 29.97 fps, 29.97 tbr, 30k tbn (default)
  Duration: 00:59:12.09, start: 0.000000, bitrate: 840 kb/s
  Stream #0:0[0x1](eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 480x360, 710 kb/s, 29.97 fps, 29.97 tbr, 30k tbn (default)
  Duration: 00:27:42.04, start: 0.000000, bitrate: 839 kb/s
  Stream #0:0[0x1](eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 480x360, 710 kb/s, 29.97 fps, 29.97 tbr, 30k tbn (default)
  Duration: 00:29:20.53, start: 0.000000, bitrate: 842 kb/s
  Stream #0:0[0x1](eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 480x360, 712 kb/s, 29.97 fps, 29.97 tbr, 30k tbn (default)
  Duration: 00:59:22.26, start: 0.000000, bitrate: 841 kb/s
  Stream #0:0[0x1](eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 480x360, 711 kb/s, 29.97 fps, 29.97 tbr, 30k tbn (default)
  Duration: 00:58:45.61, start: 0.000000, bitrate: 839 kb/s
  Stream #0:0[0x1](eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 480x360, 710 kb/s, 29.97 fps, 29.97 tbr, 30k tbn (default)
  Duration: 00:28:20.03, start: 0.000000, bitrate: 840 kb/s
  Stream #0:0[0x1](eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 480x360, 710 kb/s, 29.97 fps, 29.97 tbr, 30k tbn (default)
  Duration: 00:30:04.99, start: 0.000000, bitrate: 843 kb/s
  Stream #0:0[0x1](eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 480x360, 713 kb/s, 29.97 fps, 29.97 tbr, 30k tbn (default)
...

So this particular video is literally missing "the first" frame, only audio between 0μs - 33367μs.

[keighrim]

I spent way too much of this weekend in a rabbit hole of MPEG specs, OpenCV documentation and bug reports, video encoding history, NTSC timing, etc. And it turns out the problem runs deeper than I originally thought.

vdh has a real bug

In my last comment I said the discrepancy was "purely in your numbering convention" (the off-by-one). That was only half the story. After digging into the container metadata:

Stream #0:0: Video: h264, 29.97 fps, 30k tbn, start 0.033367

That start 0.033367 means the first frame's PTS (timestamp recorded in the video file at the encoding time) is 1001 ticks (at timescale 30000), not 0. This is 1001/30000 = 0.033367 seconds -- exactly one frame duration.

The way vdh is implemented based on cv2, it reads frames sequentially (cur_f=0, 1, 2, ...), where cv2's 0-indexed frame N corresponds to the "physical" frame with PTS = (N+1) x 1001 ticks. But vdh's frame/fps conversion computes time assuming frame 0 = time 0, which is not the case for this video.

	value	source
vdh's math	13757 / 29.97 --> 459026ms	round(round(f/fps, 3) * 1000)
actual PTS of extracted frame	13758 x 1001 / 30000	459058.6ms
PTS of the previous frame	13757 x 1001 / 30000	459025.233...ms

The MMIF says 459026ms. The frame vdh actually extracted and classified is at PTS 459058.6ms. Any tool that seeks by closest timestamp (as visaid does) will likely land on 459025.233...ms instead. (Note: frame.time in Owen's code snippet showed integer millisecond values, which themselves still suffer from rounding errors in this conversion.)

This can't be fixed within cv2

The obvious fix is time = frame / fps + start, accounting for the PTS start offset. But cv2 does not expose the container's PTS start offset directly. One could read CAP_PROP_POS_MSEC after decoding a frame, but that property has documented reliability problems (opencv#12112, opencv#15749, opencv#27819). The root issue is that cv2 hides the container's timestamp model behind a frame-number abstraction that doesn't map cleanly to how MPEG-4 containers actually work.

Transferring to mmif-python

This is a bug in video_document_helper, not in this app. Transferring to the mmif-python repo to track the fix at the SDK level. The fix will involve replacing cv2 with direct FFmpeg access (PyAV, ffmpeg subprocess, or another binding -- still exploring options) so that vdh can read actual PTS from decoded frames instead of computing timestamps from frame numbers.

In addition to this bug fix, I'm proposing to eliminate the timeUnit property from the MMIF vocabulary once and for all -- see clamsproject/clams-vocabulary#15 for more evidence of the evilness of frame numbers as timestamps.