This material is evolving. If you have questions or suggestions feel free to reach out to me.
Since our goal here is to put the learned things into practice, it will make your life easier if you have a autoregressive transformer implementation that you are familiar with.
Resources
μP is mainly about initialising your model in the right way and ensuring that it stays that way. Therefore, it probably cannot hurt to revise backpropagation. I recommend to briefly go over it based on your favourite material, e.g., François Fleuret’s little book does a great job succinctly introducing everything you need.
I guess this one is almost needless to say and I am not going to be able to enumerate everything that you need. Hopefully you know enough such that you can look up the rest on the fly. If you want to give yourself a formal refresher probably this machine learning math book can do a good job. Central limit theorem and law of large numbers are not covered by that book but also relevant.
https://blog.eleuther.ai/mutransfer/
Meetings are held every Tuesday at 17:00-18:00 CET. Please come prepared having read the assigned material for each session.
Zoom link: https://epfl.zoom.us/my/wendler?pwd=UlpxNWFTSlhuN3Ftc0tIMC9kWTZsZz09
The discussion leader will guide the group through the reading material or through their coding solution.
EDIT (28.10.): CET timezone had a shift, to keep the US slot fixed we moved to 16:00-17:00 in CET.