Hugging Face's logo
nomadicsynth's picture

nomadicsynth PRO

nomadicsynth
6 3 29
Kaliman-1981's profile picture tegridydev's profile picture RicodeFe81's profile picture
·

AI & ML interests

architecture research, knowledge discovery

Recent Activity

reacted to jiteshkprasad's post with 😎 4 days ago
We tested 27 small open language models, all under 400M parameters, in the same way. Same harness, same settings, and the model versions pinned. I think small models are really hard to compare from their own cards, so we put the whole thing out. Board: https://huggingface.co/spaces/udgamlabs/small-model-board Dataset: https://huggingface.co/datasets/udgamlabs/small-model-board-scores Write-up on the Udgam Labs blog: https://udgamlabs.com/blog/small-model-board The most surprising part for me was MMLU. In the usual A, B, C, D format, just answering "D" every time gets 26.9%, and 23 of these 27 models score at or below that. So at this size the letter format basically tests if a model can pick a letter, not what it knows. When we asked the same questions in cloze format, where the answer text itself is scored, all 23 did better. Our own models are in there like everyone else. Laghu-Base is 5th of 19 models up to 200M, and Gargi-Base is 7.3 points behind SmolLM2-360M. So we are not on top, and that is fine. What I wanted was a fair comparison. Do you compare small models on letter or cloze MMLU?
liked a dataset about 1 month ago
secemp9/arxiv-complete
upvoted a paper about 1 month ago
AccessEval: Benchmarking Disability Bias in Large Language Models
View all activity

Organizations

Blog-explorers's profile picture Neon Cortex's profile picture