Out of memory in train_vqa.py

[satyenrajpal]

train_vqa.py is running into out of memory issues. I managed to get some runs through but now it consistently goes out of memory. Even removing the multiprocessing does not solve that. Any help?
I have enough memory to run train_nav.py and train_eqa.py so it's not a problem of resources. I'm using AWS p3.2xlarge.

While I greatly appreciate you releasing the code, it would be great if you could add a bit of documentation in the code.

[satyenrajpal]

I ran train_vqa.py without any multi processing and just this for loop to check if the optimization is losing memory or just the dataloader. Turns out that the problem is just in the dataloader, specifically this line. Virtual memory consumption starts at 88.1 GB and with 10 threads, it shoots upto 99GB in 10 epochs. Any idea how to solve this?

I did uncomment clear_memory() in _load_envs(). That helped to relieve some of the memory but not entirely.

[abhshkdz]

Have you tried running it with the cache flag off?

With caching on, rendered images from the first epoch are cached so that subsequent epochs are very fast, but this is memory-intensive.

There are obviously more efficient ways of setting this up, but I don't yet have code set up for it, sorry.

Alternatively, here's a pretrained VQA model: https://www.dropbox.com/s/jd15af00r7m8neh/vqa_11_18_2018_va0.6154.pt. This gets a top-1 accuracy of 61.54% on val, and 58.46% on test (with GT navigation).

[satyenrajpal]

Thanks for the pre-trained model! Yes, we ran it both with and without the to_cache flag. It still runs into memory issues for some reason. We'll run it with the pre-trained model and then get back to you.

[abhshkdz]

Got it, I'll look into memory leaks.