Clarify eval/train parallel runs in vqa, nav, and eqa main

[thomason-jesse]

From what I can tell, in each of train_vqa, train_nav, and train_eqa, a model with shared parameters is fed to a thread running eval() and at least one thread running train() (more if specified via command-line arguments).

However, the train() method runs substantially slower, so for a fixed number of epochs (say, the default of 1000), on my machine eval() reaches the epoch cap when train() is only on around 160. After that, the train() thread(s) keep running but there's no eval() thread to checkpoint them! So I gain nothing by letting them continue to run.

For the models presented in the paper, what was your training paradigm to address this? Do you use the last checkpoint (of highest accuracy) that epoch() makes regardless of how far along the train() threads are?

[ZhuFengdaaa]

I meet the same problem too. I'd prefer synchronise train&eval process rather than paralleled one.

[ZhuFengdaaa]

@thomason-jesse I do evaluating on validation and saving models just after every training epoch finished, then delete the process targeting eval, which works well. But I am encountering another issue. The result is somehow odd. You can refer my issue.

[thomason-jesse]

I've made a similar modification, using inter-processes communication to run an eval epoch only after the corresponding training epoch finishes.

If the owners of the repository can comment on how the numbers from published papers are obtained with respect to this threading issue, that would be super helpful! My best guess right now is that many training threads are run in parallel, such that training updates roughly keep up with the single eval thread, but knowing exactly what the paradigm is would help for replication.

(in response to your linked thread on this topic:)
The result isn't too odd; the performance improves initially, but as the training data is iterated on over and over, the model begins to overfit to the training data, so it generalizes more poorly to the held out evaluation data. A common strategy is to just keep the model that has the highest eval performance across many epochs of training, which is what the eval checkpoint code is set up to do (checkpoint and save a model only if its eval performance is higher than what has been seen so far).