Investigate display of FMV data on map

[annehaley]

@aashish24 will provide example data and requirements

[BryonLewis]

I will be implementing this on DIVERS-H

The basic process is that an FMV file has some embedded KLV data inside of it. When importing kwiver is used to extract that klv data from the FMV. The KLV data seems to be positional information about frames and in the RGD app it extracts out the following:
ground_frames: multiple polygons for each frame indicating the viewing area of the video
ground_union: the total area that is displayed for the FMV video
flight_path: a listing of points for the duration of the flight
frame_numbers: a pickle file that has the frames inside of it (most likely converted to json at some point)

Some generic notes/process for implementation:

• Backend
  • Models:
    • Copy the base models for FMV and FMVMeta .
      • NOTE: The models have custom mixins for inheritance, which will need to be modified
    • Models will have to be pointed at DIVERS-H concepts of files
    • I think I'll need a new maplayer type FMVTrackedLayer. I'm trying to think if I could fit this is a special VectorLayer with attached fmv video but I think the customization on the UI side means that it needs to be a new layer type.
  • Endpoints
    • Need an endpoint to retrieve the FMV URL, the geojson polygons for the frames, the union polygon and the flight path points. I think they can be returned in raw JSON for this test version. In the future the path points and union polygon I may want to do using vector Tiles.
  • Tasks
    • The system seems to utilize Kwiver and most logic is contained inside of [etl.py)[https://github.com/ResonantGeoData/ResonantGeoData/blob/main/django-rgd-fmv/rgd_fmv/tasks/etl.py] where the main function is read_fmv_file
      • Items in here may have to be modified given some local calls to rgd.utility and the rgd_fmv models (I.E. any changes to the model Mixins may have to be accounted for here)
    • Update the ingestion script to allow for ingestion of FMV data and checking for KLV data instead of it
• Front-End
  • Background
    • The processing script for uploads should also include the ability to process these FMV files and reject if they don't contain KLV video data
    • Concept of a new Layer Type FMVTrackedLayer that will call a REST endpoint to get the FMV url, and the geojson data for frame points as well as union and flight path points.
    • Support for uploading FMV with KLV data (KLV data verification probably needs to be done during the processing). This will create new FMVTrackedLayers
  • UI
    • You begin being able to turn on/off the layer using the normal interface for all layers
    • This layer underneath it has options for toggling on/off the flight path and the union geometry. The union geometry should be a slightly shaded polygon to indicate it is a path that we are flying over.
    • There are also playback controls for playing/pausing and advancing frames. There should probably be keyboard shortcuts for this as well (spacebar + left/right arrows)
    • There should also be a seek bar for seeking through the video
    • The legends should also have some mini playback controls for playing/pausing and seeking
    • Somewhere on the screen we may want an indication of the current time and the length of the video

[BryonLewis]

Update: RGD was relying on a worker group of dependencies including kwiver which seems to no longer be supported (last pypi release was 2020). There is a kitware/kwiver docker image so I can probably use that as part of my build process for the worker container and copy over the binaries or use it as my final base.

Update 2: The kwiver/kwiver docker container formats seems to be quite different from what was being used in RGD. It was using direct output and regexing values. That seems to crazy for me so kriver has a json output but it is obtuse. It's easier to output as CSV and then convert to JSON for it to be understandable.

[BryonLewis]

PR Created in DIVERS-H here: OpenGeoscience/divers-h#36

• Kwiver Information
  • kwiver has a command called dump-klv that takes Key-Label-Values from a video file like an mpg and extracts geospatial information from it.
  • KLV seems to be limited to certain video file containers like mpeg. MP4 doesn't seem to hold this data
  • The dump-klv command will return to the console a list of the frames and the key value pairs by default.
  • There are options to export to a csv or to a JSON file.
  • The JSON file output is a dense array of values and positons for values that make it hard to read.
  • Decided to export to a CSV and then convert to a JSON for processing the data
• Kwiver Deployment
  • Kwiver has a repo here: https://gitlab.kitware.com/kwiver/kwiver
  • Older versions (2020) had a binary package built for python 3.8.X but hasn't been released since 2020.
  • The current build has a docker image where it is built and can run kitware/kwiver
    • docker run -v $PWD:/data -it kitware/kwiver:latest dump-klv /data/video.mpg -l /data/output.csv -e csv
    • Above will map the current directory to a docker container and run the dump-klv argument with the export argument of a -e to create a CSV and place it in the output.csv file
  • Kwiver Docker Build
    • https://gitlab.kitware.com/kwiver/kwiver/-/blob/main/docker/Dockerfile?ref_type=heads
    • This docker build is based on a custom ubuntu 20.04 version and it goes through and installs some requirements.txt using python 3.8.X. These dependencies are used to perform dynamic linking of the kwiver executable when it is built using cmake.
    • It seems as though you need a python 3.8 and the installation dependencies to run kwiver because the executable isn't built statically (it's relying on library files in linux)
  • DIVERS-H Kwiver Process
    • I found it easiest to just base my containers off of the kitware/kwiver image and install python 3.10 for django separately and modify the entrypoints in the docker-compose so I reference either
    • DockerFile
    • Docker-Compose
      • Note the override of the entrypoint to launch django/celery using python3.10
• Ingestion Process
  • The ingestion process works the same as other DIVERS-H where I can point to a json file with references to urls to download and process the file.
  • The file is tagged as 'fmv' and that allows it to go through a new task that runs dump-klv to create a CSV, then creates a JSON file and finally creates a geojson file
  • There is a small process in the function that if the KLV doesn't find corners for the projection of the video it will calculate them the center point and the size of the video
  • The GeoJSON file has three groups of data
    • Flight Path - This a list of points that the drone or camera is at and has the frameId for this list. I also included on these points all of the data exported from the KLV for the video
    • Ground Frame - geojson polygon representing a frame location as a bounding box
    • Ground Union - Taking all of the frames over time and creating a union geometry to see the full extents of the video.
  • After creation of a geoJSON data it creates an FMVLayer and for all the vector features it creates an FMVVectorFeature
  • Also transcodes the video into a web-safe version for displaying
• Models
  • FMVLayer
    • Represents the layer containing FMV data.
    • Main Elements are the fmv_video (S3FileField) and the geojson_file (S3FileField)
    • There are also some other fields regarding properties of the fmv_video: frame count, duration, width, height for the video
  • FMVVectorFeature
    • VectorFeature for FMV files.
    • The geojson file created during creation is converted into an FMVVectorFeature
• Endpoints
  • There are standard endpoints for all map layers that were updated to support the new fmv layer type. This also includes a new tile server endpoint for FMVVectorFeatures
  • The main endpoint `GET /fmv-layer/${id} returns back:
    • The FMV presigned URL for the transcoded version
    • The FMV information like frame count, width and height
    • A dictionary mapping of available FrameIds to the matching ground_frame in the geojson file. This goes through the DB of FMV Vector Features and looks for these and converts them into geoJSON
• Client Notes:
  • Much like an ImageSource layer in maplibre there is a VideoSource layer where you provide a URL and coordinates of where to map the video.
  • Additionally, I created a list of other vector layers for the flight_path, ground_frames and the ground_union.
  • An event listener is hooked up to the VideoSource and every time a frame is changed it will update the current position of the bounding box for the frame and move it around
  • NOTE There are some limitations with the current VideoSource implementation that prevent the frame from updating unless the video is playing. So, seeking doesn't update what is displayed on the map.

[BryonLewis]

I have created a PR in DIVERS-H that supports this functionality at: OpenGeoscience/divers-h#36
The PR has more information about the functionality and the code changes that made it.
Closing issue.