Report motion vectors

[fhunleth]

A side effect of encoding video is calculating motion vectors in hardware. It would be nice to get the motion vectors in Elixir to support things like simple motion detection and optical flow calculations. It wouldn't be as good as what can be done with OpenCV, but it's computationally cheap and would provide for some fun experiments.

[mmmries]

Hey @fhunleth I've been playing around with picam recently and I'm interested in helping out with this effort. I've never worked with hardware encoding before, but I can learn new stuff 😄 . Do you have any suggestions for where I should start?

Maybe a library that implements this for some other use-case? Or the name of the c libraries for interacting with the encoding that happens in hardware?

[mmmries]

Maybe this is a good place to start? https://picamera.readthedocs.io/en/release-1.13/recipes2.html?highlight=vectors#recording-motion-vector-data

[fhunleth]

Hi @mmmries!!! Sorry for the late response.

I think that you found a good place to start. I vaguely remember finding information about this by reading the MMAL header files. Unfortunately, I forgot which one and would need to do it again. I suspect that you might have found a better location anyway.

I hope that this doesn't come across as discouragement, but I think the days of picam are limited due to the Raspberry Pi switching to libcamera. Implementing motion vector support here in picam will definitely be the easier route for now, of course. However, as soon as there's a viable libcamera-based Elixir library, I'll want to enable it for the Pi 4 on Nerves (picam doesn't work on the Pi 4) and it would let me switch the RPi 3 and RPi Zero 2W to 64-bit mode and use the OTP 25 ARM JIT there.

[mmmries]

That's super helpful context @fhunleth. Maybe I'll take a look at libcamera to see if building a wrapper is something I can contribute to. I have a couple of Zero 2W boards and I'm working on a DIY home security setup that records clips whenever motion is detected. I've been trying a few things in python from the picamera library I linked above and right now I'm just learning a lot about what the cameras are capable of and getting familiar with what is possible, what happens in the GPU, etc.