Skip to content

Commit 361fd0f

Browse files
authored
Update README.md
1 parent 9de2ec5 commit 361fd0f

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
* AGC w/ default clipping factor `--clip-grad .01 --clip-mode agc`
88
* PyTorch global norm of 1.0 (old behaviour, always norm), `--clip-grad 1.0`
99
* PyTorch value clipping of 10, `--clip-grad 10. --clip-mode value`
10-
* AGC performance is definitely sensitive to the clipping factor. More experimentation needed to determine good values for smaller batch sizes and optimizers besides those in paper. So far I've found .001-.005 is necessary for stable RMSProp training.
10+
* AGC performance is definitely sensitive to the clipping factor. More experimentation needed to determine good values for smaller batch sizes and optimizers besides those in paper. So far I've found .001-.005 is necessary for stable RMSProp training w/ NFNet/NF-ResNet.
1111

1212
### Feb 12, 2021
1313
* Update Normalization-Free nets to include new NFNet-F (https://arxiv.org/abs/2102.06171) model defs

0 commit comments

Comments
 (0)