ImageNet result

[AojunZhou]

I noticed your imagenet result is 62.2% (top1), can you share your training log for me or more detail training setting?

[jaxony]

@gngdb do you have training logs to share?

[AojunZhou]

@jaxony Thanks, Do you know how to replicate the paper result about 1x with group 3 in Pytroch.

[gngdb]

Sorry, didn't keep logs the first time. I'm running it now trying to match the settings from the paper and I'll write the full training logs to a file. Have to make some changes to the imagenet training script so I'm thinking I should just open another pull request with that included?

Also, the pretrained model in the repo at the moment is groups=3 and multiplier 1x as far as I know; those were the default settings.

[AojunZhou]

@gngdb thanks for your kind reply, I try to implement the groups=3 and multiplier 1x, I got only 59.8% validation accuracy after 60 epochs

[AojunZhou]

@gngdb Hi , I forgot to remind you that the the original paper have updated at 7 Dec 2017, https://arxiv.org/abs/1707.01083 , you can find better accuracy in new version, maybe because of larger learning rate and larger batch size and more training time, Good luck!

[windid]

Hi, all
I've trained imagenet for 3 times using this implement, following is my experiment settings and final result:

1).
init learning rate: 0.1
batch size: 256
learning rate divided by 10 for every 30 epochs
epochs: 100
groups:3
transforms:
transforms.Compose([ transforms.RandomResizedCrop(224), transforms.ColorJitter(brightness=0, contrast=0.5, saturation=0.5, hue=0), transforms.RandomHorizontalFlip(), transforms.ToTensor(), normalize, ])
Top1: 65.2%, Top5: 86.0%

2).
init learning rate: 0.2
batch size: 384
learning rate divided by 10 for every 30 epochs
epochs: 100
groups:3
transforms:
transforms.Compose([ transforms.RandomResizedCrop(224), transforms.RandomHorizontalFlip(), transforms.ToTensor(), normalize, ])
Top1: 64.8%

3).
init learning rate: 0.1
batch size: 256
learning rate divided by 10 for every 30 epochs
epochs: 100
groups: 8
transforms:
transforms.Compose([ transforms.RandomResizedCrop(224), transforms.ColorJitter(brightness=0.2, contrast=0.4, saturation=0.4, hue=0), transforms.RandomHorizontalFlip(), transforms.ToTensor(), normalize, ])

Top1: 64.8%, Top5:85.7%

I have only 2 gtx 1080, due to the limitation of gpu memory, I can only use batch size 256(groups=8)/384(groups=3), not the same as the paper.
Notice that the group8 arch's training speed is much slower than group3 with more memory consumption.

[AojunZhou]

@windid Hi, Thanks, do you use @jaxony 's code directly? what about weight decay?

[windid]

@Zhouaojun Yes, I use this implement directly, weight decay is set to 4e-5 as mentioned in paper.

[AojunZhou]

@windid ok, I will try your first implementation, with init learning rate: 0.5, batch size: 1024, 4 GPU.

[gngdb]

Can you fit this implementation with batch size 1024 on 4 GPUs with groups=8? I've tried, and it's too big. I was going to run it with a batch size of 512, but there wasn't enough time available on the shared server.

[AojunZhou]

Yes, My single GPU with 24G memory.

[gngdb]

I've now run groups=8 following as close as I could to the training settings described in the paper. Unfortunately, didn't match performance of the paper. Top 1 is 63.372%, Top 5 is 84.661%. Almost certainly because I was using a batch size of 512 instead of 1024. But, at least I've got training logs to share now. The script I ran is here: https://github.com/gngdb/ShuffleNet/blob/master/imagenet/train.py

And the command I ran the script with, along with the full log of training, is here: https://raw.githubusercontent.com/gngdb/ShuffleNet/master/imagenet/training.log

The server was restarted at some point during training, so I had to restart from a checkpoint, which you can see in the logs.

[bowenc0221]

@gngdb Did you use the linear learning rate decay as indicated in the paper? PyTorch does not have an implementation of that learning rate decay.

BTW, you also need to train for 256 epochs to match the 3e5 iterations in the paper.

[bailvwangzi]

@bowenc0221
for param_group in optimizer.param_groups:
new_lr = args.lr - (float(minibatch_index)*args.lr)/3e5
param_group['lr'] = new_lr

[gngdb]

Yes, it is a linear learning rate decay; the comment in this function is just wrong: https://github.com/gngdb/ShuffleNet/blob/master/imagenet/train.py#L297-L301