Hugging Face's logo
Rao's picture
🔄 In a Training Loop

Rao

RakshithFury
3 7 11
DVLX's profile picture Praful932's profile picture xtools-at's profile picture
·

AI & ML interests

None yet

Recent Activity

liked a model about 4 hours ago
RecursiveSI/flint
reacted to ifaposto's post with 🔥 about 4 hours ago
Self-distillation methods usually rely on an Exponential Moving Average (EMA) teachers to get synced and stable critics. However, this doesn't scale well for large models since holding both teacher and student (trained) on GPU increases latency, zero3 communication, etc. We can mitigate this with LoRA finetuning: the backbone is shared & frozen, and we need to maintain two light adapters: one for the student (trained) and one for the teacher (EMA of the student adapter). Getting this to work cleanly under TRL and ZeRO-3 took some fixes, so I packaged it as a plug-and-play repo. It supports: -> robust SDFT implementation that runs under ZeRO-3 -> ZeRO-3 + LoRA integration with adapter switching for teacher/student passes -> Saving and resuming EMA teacher checkpoint -> working deepspeed configs that fix memory spikes when GPU memory is tight I tested on Qwen3.5 and ToolAlpaca. Repo: https://github.com/ifiaposto/sdft_zero3 Feedback/ more examples/ issues welcome!
repliedto jiteshkprasad's post 5 days ago
Last week we released Gargi, a 384M English language model trained from scratch on about 17B tokens, roughly 1,000× less training data than Qwen2.5-0.5B (~18T tokens, as published by its authors). What we measured (same harness for every model: lm-evaluation-harness, 0-shot, greedy, each model's start token): - Base model vs Qwen2.5-0.5B: ARC-Easy 62.8 vs 64.7, ARC-Challenge 32.3 vs 32.4, WinoGrande 54.6 vs 56.1. - Chat model: thinking on or off; GSM8K 36.0 with thinking on. - Long inputs: needle-in-a-haystack recall 100% at 8K, 93% at 16K and 90% at 20K tokens, so use it up to about 16K. Where it falls short: knowledge and everyday common sense still scale with data. MMLU is 25.5 vs 47.3, and it trails on HellaSwag and PIQA. It is not a maths specialist, and it is not safety-tuned: put a moderation layer in front of any public use. It runs on a laptop: GGUF (Q8_0 at 410 MB, Q4_K_M, f16) and MLX builds. `ollama run hf.co/udgamlabs/Gargi-GGUF:Q8_0` Apache-2.0 · weights: https://huggingface.co/udgamlabs/Gargi try it: https://udgamlabs.com/models/gargi code: https://github.com/udgamlabs/gargi Named after Gargi Vachaknavi, the Vedic philosopher remembered for asking the hardest questions.
View all activity

Organizations

Axionic Labs's profile picture