cvtracer/README at master · patchmemory/cvtracer · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
###############################################################################
#                                                                             #
#  This is the cv-tracer software package, designed for tracking and          #
#  analyzing dynamics of fish swimming in groups. It can be downloaded at     #
#                                                                             #
#    https://github.com/patchmemory/cvtracer                                  #
#                                                                             #
#  Software designed and developed by Adam Patch and Yaouen Fily at Florida   #
#  Atlantic University. Began with github.com/vivekhsridhar/tracktor.         #
#                                                                             #
###############################################################################

  In the cv-tracer package, one will find the modules TrAQ, Analysis, and GUI
  These three modules provide necessary tools for different stages of tracking
  birds-eye video of fish swimming in a tank, reviewing the results alongside
  video to check for errors, and analyzing the results across trials.

  Scripts for executing all tasks for tracking can be found in

    scripts/track

  subdirectory of each base module (e.g. TrAQ/scripts, Analysis/scripts).

  This document provides a basic overview of how to use this software and is
  broken into three sections:

    1. Tracing and processing tracks
    2. Analyzing results across trials
    3. Reviewing kinematics alongside video

  This software is designed in python with free libraries so that it may be
  straightforwardly modified to incorporate more advanced methods of tracking
  and analysis.


  #############################
  Tracing and processing tracks
  #############################

    One can completely track and process a video trial, by executing the
    following commands and providing relevant input parameters.

      >  python prepare.py [input_video] -opts
      >  python trace.py [trial] -opts
      >  python process.py [trial] -opts

    These each perform the following tasks:

      prepare.py: Loads video for the first time and takes down information
                  about the run like the raw video file, the number of
                  individuals, the type of fish, the total length of the run,
                  and any other relevant input for characterizing the trial.
                  The script then generates a Trial object to store all of this
                  information.

                  Once the trial object has been created, the script will
                  prompt the user to utilize a GUI to locate the tank edges and
                  store the result as a Tank object inside the trial object.

                  A Group object (which contains Individual objects) within the
                  Trial object will also be generated to store and analyze data
                  at the time of tracing and processing.

                  Finally, the script will generate a directory with a concise
                  name reflecting that of the original video file, moving that
                  file to this new directory as raw.mp4. The Trial object will
                  also be stored in this directory with the name trial.pik

        trace.py: Loads Trial object and utilizes Tracer and VideoCV objects to
                  track coordinates of fish throughout that trial.

                  At each frame, Tracer will gather and sort the findings of
                  VideoCV in order to best trace the location of fish
                  throughout the run, ignoring fish misidentified outside of
                  the Tank, as identified in prepare.py. The resulting
                  coordinates are stored as Individuals in the Trial's Group.

                  Trial object, now containing Tank, Group, and all relevant
                  trial-specific information, is again saved in the
                  trial-specific location, as found at the beginning of
                  tracing.


      process.py: After the initial tracking has occurred, it is useful to now
                  better calculate kinematic quantities like velocity and
                  acceleration in both translational and rotational coordinates
                  (latter w.r.t. director angle).

                  Using these quantities, it is then useful to provide some
                  initial statistical analysis without any cuts, for the sake
                  of quickly summarizing the behavior of fish in each run.

                  With further development, this stage would also be the place
                  to fix errors in the initial tracks which may have issues
                  with occlusions and jumping points. At this point in time, it
                  only really makes sense to perhaps identify frames and points
                  of interest for the sake of troubleshooting in the future.
                  For our first paper, these issues are handled by simple
                  ad-hoc cuts which will be made to obviously erraneous data
                  during the next stage, Analysis, so that all Trials will be
                  treated uniformly.


  ###############################
  Analyzing results across trials
  ###############################

    Once the tracking of a set of trials is complete, one can run the following
    command and relevant input parameters in order to analyze data given a set
    of cuts.

      >  python analyze.py [trial_list] -opts

    While the analysis itself may be modified, this will perform a few basic
    tasks regardless of the specific analysis.

     analysis.py: First, all Trial objects will be loaded from their save
                  locations and stored in an Archive object. This object has
                  the ability to call functions on each trial (e.g. make a
                  consistent set of cuts) and to combine statistical results
                  across trials, storing results according to analysis-specific
                  cuts applied.

                  Once results have been computed and collected, they can be
                  combined and stored in the Archive object's result
                  dictionary. This allows easy access via the Archive object
                  later.

                  Utilizing the Analysis.Plot module, one can efficiently plot
                  distributions and results for each trial and across trials.

                  Trial-specific results will be stored in the Trial's home
                  directory while the results across trials will be stored in
                  results/.


  ####################################
  Reviewing kinematics alongside video
  ####################################

    While one can, in principle, follow all of the above steps, it is always
    useful to verify kinematic quantities being calculated correctly via some
    kind of visual sanity check. The easiest way to do this is to utilize the
    cvDataViewer, found in the GUI module. One can execute this GUI by simply

      >  python GUI/cvDataViewer.py [trial_directory] -v cv.mp4

    This will open a window containing a video of the tracked fish as well as a
    side panel displaying time-dependent results of three quantities: distance
    to the wall, speed, and angular speed.

    This tool facilitates checks on tracking throughout a run by allowing users
    to see directly if quantities are being properly calculated across frames
    of the video.

    A couple examples of how one might use this information...

      One may find that the tracer is having trouble locating fish at every
      frame, resulting in jumps from track-to-track. By viewing this in the
      cvDataViewer, one may step back to the tracking stage and try using a
      different set of contour-tracing parameters to better identify fish and
      then go back to verify after a second round of tracking has completed.

      Another may find that while the tracing has been done optimally, there
      are still undesirable errors where fish appear to be jumping across the
      tank or turning in an unphysical way. Despite the shortcomings of the
      tracking software, this provides a basis for eliminating erroneous frames
      by making cuts at the stage of Analysis, in order to consistently treat
      systematic errors across runs.


  ############
  Dependencies
  ############

    The following python libraries are required for full functionality of
    cv-tracer:

      - argparse
      - copy
      - matplotlib
      - numpy
      - opencv (cv2, which version?)
      - pickle
      - PyQt5
      - pyqtgraph
      - scikit-learn
      - scipy