custom dataset

[ajtao]

Hello,

I have 2 questions:

1. In your instructions, you state: "You first have to run the complete tracking pipeline (without tracking, with a pre-trained CAMELTrack or with a SORT-based tracker, like oc-sort), on train, validation (and testing) sets for the dataset you want to train, and save the "Tracker States".

• Can you make it clear, does this mean running with or without tracking? I'm not sure what you meant by "without tracking"?
• When i ran the suggested command, such as "uv run tracklab -cn cameltrack dataset=dancetrack dataset.eval_set=train", this results in the tracking being run and then evaluation follows. But if tracking is disabled, then how could evaluation succeed? The code throws an error about "track_id" missing, which is because i turned tracking off :)
• Can you also say a bit more about what this process is actually doing? I assume that this helps to create pose and reid information that is needed for CAMELTrack to be trained with? And is all that information saved within the tracker state?
• Why does tracking have to be run if the ground truth tracks are already available?

2. I have my own dataset where i already have detections/pose/reid features calculated. Do i still need to regenerate pose/reid using the CAMELTrack pipeline or can i somehow build the tracker state myself?

Thanks

[yashb042]

This indeed was confusing. Let me put down what I understood and answer a few questions -

There are 2 pipelines, one called tracking(Pipeline1) and other being training (Pipeline2). The tracking pipeline uses mostly the existing models (like reid, Yolo, etc.) + Training.
This link - https://github.com/TrackingLaboratory/CAMELTrack?tab=readme-ov-file#-tracking, is for final output. That means both pipeline 1 and 2 are being used to generate the trackings (meaning the final output, not the first pipeline. I think this is where it gets confusing).

Can you make it clear, does this mean running with or without tracking? I'm not sure what you meant by "without tracking"?

Should be answered.

When i ran the suggested command, such as "uv run tracklab -cn cameltrack dataset=dancetrack dataset.eval_set=train", this results in the tracking being run and then evaluation follows. But if tracking is disabled, then how could evaluation succeed? The code throws an error about "track_id" missing, which is because i turned tracking off :)

What do you mean when you say "tracking is disabled?"

Think of tracking (pipeline1) output as only used for model training and nothing else. If you have your own dataset, the training process might be leveraging tracking output multiple times that's why it is saved separately as pklz.

P.S. - I am just a user like you so the answers might not convince you at all. 😀

[VlSomers]

@yashb042 thanks for your answer. @ajtao did you figure this out? Let me know if I can help further. Basically, you need to generate TrackerState in pklz format before training CAMEL. That TrackerState should contain all information relevant for training CAMEL (i.e. bbox, keypoints, reid embeddings, ...). You could use ground truth bbox/keypoints for training, but we found out using real keypoints and bbox leads to better performance (because the training is then closer to real inference on noisy data). We however don't need to generate a track_id using a tracking module (e.g. ByteTrack), because (for training CAMEL) we assign each detection a track_id by matching it with the closest ground-truth. Running without tracking means that the pipeline is run without a tracking module, so not track_id is generated in the final TrackerState (you could do it, but not necessary as this track_id will not be used later on for training CAMEL). Finally, you could generate your own TrackerState manually with your ground truth, but (i) you would be missing reid features, (ii) you would probably need a lot of data augmentation to add noise to these ground truth, to make it look more like a real inference data.

[ajtao]

Hi @VlSomers, thanks for the reply. Thanks @yashb042 as well.
I fully understand that we want to use predicted pose/bboxes for training cameltrack. It's just that the README didn't explictly explain that the "Tracker States" contained that information. That part makes sense now.

I just want to confirm that the config file i'm using to prepare the tracker states for training is correct.
Does the following look correct? I've basically turned off tracking part of pipeline. Is that it?

I saved this as "camel_b4train.yaml" and i'm running "tracklab -cn cameltrack_b4train dataset=dancetrack dataset.eval_set=val" for the val split.

Does all this look correct?

# The defaults' list contains the files that will be used
# to create the final config file. This item *must* be
# the first element in the file.
# 'tracklab --help' shows the available modules/datasets/visualizations/engines/...
defaults:
  - dataset: example_video
  - eval: trackeval
  - engine: offline
  # - visualization: keypoints
  - visualization: none
  #- modules/bbox_detector: yolo_ultralytics
  - modules/pose_estimator: yolo_ultralytics-pose
  - modules/reid: kpr
  # - modules/track: camel
  - _self_

# Pipeline definition:
# - Defines the order in which the modules will be called
# - If loading a tracking state from disk, remove modules whose outputs are already provided in the loaded state
# Use 'pipeline: []' for an empty pipeline
pipeline:
  #- bbox_detector
  - pose_estimator
  - reid
  # - track

num_cores: 4  # Number of CPU cores to use for data loading and processing

# Logging
experiment_name: "CAMELTrack"
print_config: false
use_wandb: false

# Paths
data_dir: "${project_dir}/data"
model_dir: "${project_dir}/pretrained_models"

# State files loading/saving
state:
  load_file: null
  save_file: "${project_dir}/states/${experiment_name}-${dataset.eval_set}.pklz"
  load_from_public_dets: false  # Use public detections in the pipeline
  load_from_groundtruth: false  # Use ground truth detections in the pipeline

# Flag parameters
test_tracking: true
eval_tracking: true
use_rich: true  # Use colorful terminal output

# Configure Hydra to use the current working directory and set up the output directory according to experiment_name
project_dir: "${hydra:runtime.cwd}"
hydra:
  job:
    chdir: True
  run:
    dir: "outputs/${experiment_name}/${now:%Y-%m-%d}/${now:%H-%M-%S}"
  sweep:
    dir: "multirun_outputs/${experiment_name}/${now:%Y-%m-%d}/${now:%H-%M-%S}"

[ajtao]

@VlSomers FWIW, i'm using my own custom yolox detector checkpoint + RTMPose with your framework. I've got it plugged in and working for inference with cameltrack, but i just want to re-train cameltrack now.

[VlSomers]

Seems alright to me at first glance! Can you confirm your "pose_estimator" outputs both bbox and keypoints? I suggest you keep the 'track' module in the pipeline enabled (even if the track_id are not used afterwards at training), and you use a tracker that relies on ReID embeddings (eg bpbreid_strongsort) to make things easier to validate afterwards. After you tracker_state is generated, then do two things to validate it:

1. run tracklab again with an empty pipeline, and "load_file: path/to/your/trackerstate" config, and the visualization enabled. Then make sure the video is ok: bbox and keypoints are good, and tracking seems to work. If tracking with bpbreid_strongsort is working, then it means you reid embeddings are relevant. Even if we don't care about track_id, visualizing the tracking is a nice way to validate the reid embeddings are good and there was no issue generating them.
2. write a script to load the tracker state pklz, then use a panda dataframe visualizer to see what is the content inside the various dataframes, to make sure you understand what they contain, and that all relevant information is there. You should see some content like the screenshot of this user on this github issue.

[ajtao]

Thanks @VlSomers .

I believe that my pose estimator is simply adding pose values into the detection dict that comes from the bbox detection, so i think Yes.

class RTMPose(ImageLevelModule):
    input_columns = []
    output_columns = ["keypoints_xyc", "keypoints_conf"]

    def __init__(self, device, model, **kwargs):
        super().__init__(batch_size=1)
        self.device = device
        self.model = instantiate(model, device=self.device, backend='onnxruntime')

    def preprocess(self, image, detections: pd.DataFrame, metadata: pd.Series) -> Any:
        return image

    def process(self, sample: Any, detections: pd.DataFrame, metadatas: pd.DataFrame):
        bboxes = detections.bbox.ltrb().values
        keypoints, scores = self.model(sample[0].numpy(), bboxes)
        detections["keypoints_xyc"] = list(np.concatenate([keypoints, scores[..., np.newaxis]], axis=-1))
        detections["keypoints_conf"] = list(np.mean(scores, axis=1))
        return detections

Thanks very much for those pointers on ways to validate things, will definitely try them.

Will cameltrack/tracklab automatically take advantage of multiple GPUs for building tracker state?

[VlSomers]

Ok, so if you use a two stage pose estimator that inputs some bbox, you should either enable the bbox detector in your pipeline, or load bbox from a tracker_state. In you pipeline I see that you don't load a tracker state, and you don't enable the bbox_detector. I also see that you enable a "yolo_ultralytics-pose" pose estimator, but your code snippet shows a RTMPose. There is something strange in your config

[ajtao]

Yes, i'm on the same page now.

The config where i'm using my own detector and top-down pose estimation is different from above and indeed i have enabled bbox_detector in the pipeline. I'm calling my own yolox-trt module for detection. And i can visualize the outputs with pose and they look good for a single video so far.

Thanks very much for all your help, i should be good now.

[ajtao]

@VlSomers I've followed your suggestions. I generated a dancetrack-val.pklz file by running the default pipeline as-is to generate tracker state, but subtituting in just the tracker as bpbreid_strong_sort. This appeared to run successfully as far as i can tell. I can enable visualization from this state-generation run and i do get a video as expected.

When i try to now visualize this val tracker state pklz file, i'm getting an error in the load() function of tracker_state.py where it expects pandas columns of the following: ['matched_with', 'bbox_conf', 'image_id', 'bbox_ltwh', 'age', 'keypoints_xyc', 'time_since_update', 'costs', 'state', 'visibility_scores', 'keypoints_conf', 'hits', 'track_id', 'embeddings', 'category_id', 'video_id']

... but the pklz video file only has ['video_id', 'visibility_scores', 'bbox_ltwh', 'image_id', 'bbox_conf', 'keypoints_conf', 'keypoints_xyc', 'category_id', 'embeddings'].

This is rather confusing to me.
The expected fields appear to come from a file called summary.json, which is contained in the pklz file, but i don't know why these fields are misaligned with what my run produced.

Is it possible i just need to set the tracker to cameltrack to get the state to generate correctly?

[ajtao]

Uhg, i seem to have fixed this by setting state.save_file: none, and now i can visualize the pklz tracker state file i generated.

I hope you can explain why having a save file defines the fields that should be read in and why this caused a problem.
I really don't understand what is going on.
This is really hard to get right.

[VlSomers]

@ajtao so you ran the pipeline a first time (with bpbreid_strong_sort etc) to generate the pklz with state.save_file set to a value, and then you ran it a second time to load that pklz, right? On the second run, if you still have state.save_file set to a value, then it fails, but if you set it to none, then it succeed. Does that correctly describes you problem? If yes, then indeed this is not the expected behavior, I'm not sure what is happening. I will have a more detailed look once you confirm

[ajtao]

Yes that correctly describes the problem!