Ronen Elden and Yuanzhi Li, 2023

paper: https://arxiv.org/pdf/2305.07759.pdf

unoganized notes

• tinystories is an effort to facilitate the creation of LMs that are significantly smaller than the state of the art (<10M params) that have coherent reasoning capabilities
• the authors speculate that small language models (SLMs) struggle, not because of inability to comprehend natural langugae, but because of the excessive breadth of information stored in the data used for training