Hey all, just stumbled upon the preprint and wanted to say hello. I’ve been working on a similar project designed for the even smaller 1 to 10m parameter range using the tiny stories dataset. Whereas you seem to be focusing on actually getting some useable performance, I’m designing mine to facilitate novel architecture experiments with practically no compute before scaling up to a real test.
Anyways, since your license is not up yet I was wondering how loose you plan on making it as I’m an amateur and I’d guess I’ll find some of your code worth copying & editing. Ofc will link back to you
Here are links to the repo and a YouTube video about it. Also let me know if one of you would like to come on the YouTube channel for an interview, my audience would love a project like yours
https://github.com/evintunador/templateGPT
https://youtu.be/s9kQvDsWnbc
Hey all, just stumbled upon the preprint and wanted to say hello. I’ve been working on a similar project designed for the even smaller 1 to 10m parameter range using the tiny stories dataset. Whereas you seem to be focusing on actually getting some useable performance, I’m designing mine to facilitate novel architecture experiments with practically no compute before scaling up to a real test.
Anyways, since your license is not up yet I was wondering how loose you plan on making it as I’m an amateur and I’d guess I’ll find some of your code worth copying & editing. Ofc will link back to you
Here are links to the repo and a YouTube video about it. Also let me know if one of you would like to come on the YouTube channel for an interview, my audience would love a project like yours
https://github.com/evintunador/templateGPT
https://youtu.be/s9kQvDsWnbc