Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
TobiasLogic 
posted an update Sep 20
Post
3,603
We’ve been cooking something new at Bench Labs.

Introducing Cagliostro-v3, our new 146M parameter language model trained completely from scratch.

The run isn’t even finished yet.

At the current checkpoint:

• 146M parameters
• 72.7B / 75B tokens trained
• 26.27 Open SLM Index
• 43.80 ArithMark-3
• Trained on a single RTX 5090
• ~90K to 103K tokens/sec during training
• ~9 days for the full run
• Apache 2.0

For some context, SmolLM2-135M scores 27.13 on the same Index after being trained on roughly 2 trillion tokens.

Cagliostro-v3 is currently at 26.27 with only ~72.7B.

That’s around 27x fewer training tokens.

The model also currently Hold the number 3rd spot for ArithMark-3, scoring 43.80

This wasn’t achieved by just throwing more tokens at the model. A huge part of v3 has been figuring out architecture, data mixture, and training dynamics at this scale.

The model uses a custom 30-layer decoder architecture with grouped-query attention and cross-head subspace attenuation, SwiGLU, RMSNorm, RoPE, tied embeddings, and a warmup-stable-decay training schedule.

During cooldown we also substantially shifted the data mixture toward higher-quality synthetic textbook and mathematics data, with the mathematics share increasing from 10% to 28%.

And everything is open.

The repository contains the training history with checkpoints pushed roughly every 30 minutes, so you can inspect how the model evolved throughout training rather than only seeing the final weights.

This is still a pre-final checkpoint. We have roughly 2.3B tokens left and the learning-rate cooldown is still running.

So 26.27 isn’t the final number.

Really excited to see where the last part of the run lands.

Cagliostro-v3:
bench-labs/cagliostro-v3
Built by Bench Labs.

Open SLM Leaderboard:
AxiomicLabs/Open_SLM_Leaderboard
Banaxi-Tech

Hi @BananaMindBot please do a review of this model and tell me if it beats BananaMind 2 Pro.

·
BananaMindBot

Based on the model cards, Cagliostro-v3 does beat BananaMind-2-Pro on the key metrics, though with important context:

Key Comparison Results:

Open SLM Index (Cagliostro-v3: 26.27 vs BananaMind-2-Pro: 24.96)

  • ✅ Cagliostro-v3 leads by 1.31 points

ArithMark-3 (Cagliostro-v3: 43.80 vs BananaMind-2-Pro: 38.20)

  • ✅ Cagliostro-v3 leads by 5.60 points (significant margin)

Important Context:

Training Scale:

  • Cagliostro-v3: 72.7B tokens, 146M parameters
  • BananaMind-2-Pro: 100B tokens, 139M parameters

Architecture Differences:

  • Cagliostro-v3: 30 layers, cross-head subspace attenuation, 5 KV heads
  • BananaMind-2-Pro: 24 layers, QK normalization, 4 KV heads

Why Cagliostro-v3 Outperforms Despite Fewer Tokens:

  1. Superior architecture: Cross-head subspace attenuation and 30 layers vs 24 layers
  2. Mathematics focus: Increased math data share (28% vs ~10% during cooldown)
  3. Training dynamics: Optimized warmup-stable-decay schedule
  4. Efficiency: Better per-token performance despite lower total compute

Caveat: Cagliostro-v3 is still a pre-final checkpoint (2.3B tokens remaining), so its final performance may improve further. BananaMind-2-Pro represents a completed training run.

Conclusion: Yes, Cagliostro-v3 currently beats BananaMind-2-Pro on both the Open SLM Index and ArithMark-3 benchmarks, demonstrating that architectural improvements and targeted data mixtures can overcome the advantage of more training tokens.
dipankarsarkar

Both numbers on this model are right, and the gap between them is one column doing all the work.

Your 26.27 and the leaderboard's 26.13 are the same checkpoint family scored twice. I pulled the Space's index.html and ran getIntelligenceIndex on the row it ships for cagliostro-v3:

arc 54.17  hellaswag 42.42  piqa 67.46  arcChall 27.99  arithmark3 43.20
  -> 26.1260   (board blurb: 26.13)

swap arithmark3 43.20 -> 43.80, nothing else
  -> 26.2685   (your post: 26.27)

So the whole 0.14 is ArithMark-3 moving 0.60 while every other column is identical. That is the run still finishing, not a scoring disagreement.

The part I would not have guessed is what a point is worth where. The header says ArithMark-3 weight 0.65, other components weight 1. True in normalized space. In raw benchmark points, which is what you actually move by training, the effective leverage is:

d(Index) per raw point
  piqa         0.548
  hellaswag    0.365
  arithmark3   0.237
  arc-easy     0.183
  arc-chall    0.183

PIQA is the heaviest lever on the board, 2.3x ArithMark-3 and 3x either ARC half. Not because of a weight, but because normalizeFromChance divides by 100 - chance, and PIQA's chance is 50 while everything else is 25. Every raw PIQA point is worth two normalized points before any weight is applied.

Which shows up in your own v2 to v3 jump:

                raw move     Index bought
hellaswag    36.76 -> 42.42     +2.07
arithmark3   35.20 -> 43.20     +1.90
arc          47.43 -> 54.17     +1.23
piqa         65.67 -> 67.46     +0.98
arcChall     28.58 -> 27.99     -0.11
                                -----
                                +6.07

ArithMark-3 is the biggest raw gain you made, 8.00 points, and it lands second. A 1.79 point PIQA move bought half as much Index as an 8.00 point ArithMark move.

Rank checks out: 2nd of 191 scorable rows, behind SmolLM2-135M at 27.13. Worth saying out loud that the model above you is 135M against your 146.4M, so 2nd place overall and best-per-parameter are not the same claim here.

At 67.46 PIQA you have 32 points of headroom on the cheapest column on the board. Is PIQA something you can still move at 146M, or is it already noise at this scale?

·
TobiasLogic

We’re aware PIQA has a much bigger impact on the Index per raw point, especially because of the normalization. We’ve been keeping an eye on that already. The math push was intentional though, since we wanted actual capability gains rather than just optimizing for the leaderboard formula. PIQA is definitely one of the main areas we want to improve next, especially since even a small gain there would move the Index a lot.
spitfire4794

Absolute peak
dipankarsarkar

The math push is also the Index-optimal move. My "32 points of PIQA headroom" was the wrong ceiling.

First, @Harley-ml is right. 135M and 146M are one weight class, and 11M parameters do not buy a 1 to 2 point Index gap. The per-parameter caveat I hung on your 2nd place was doing no work, so here it is priced against the class instead.

100 is not the ceiling that matters. The board is. Your weight class on the live board (100M and up, 46 rows), best observed per column, priced with the Space's own getIntelligenceIndex:

              v3      class best                  Index if matched
piqa          67.46   69.42  GPT-X2.5-135M            +1.07
arc-easy      54.88   58.63  SmolLM2-135M             +0.68
hellaswag     42.51   43.22  SmolLM2-135M             +0.26
arc-chall     28.75   29.69  SmolLM2-135M             +0.17
arithmark3    43.70   65.70  MobileLLM-R1-140M-base   +5.22

PIQA is still the steepest lever per point. It is also nearly spent at this size. You are 6th of 46, and 1.96 points off the best PIQA anywhere on the board.

ArithMark-3 is 22 points short of a 140M model that exists. Closing that is worth 5x closing PIQA.

The catch is how MobileLLM-R1 got there. It ranks 7th, at 24.64, because it paid 8.67 points of HellaSwag and 4.24 of PIQA against you. SmolLM2 leads while scoring 39.20 on ArithMark-3, below your 43.70.

So the leader beats you on every column except the one you pushed.

(Your live row has moved since my last read: 42.51 / 54.88 / 28.75 / 67.46 / 43.70. Index 26.55, still 2nd of 193, 0.58 behind SmolLM2.)

Is that trade forced at 146M, or is it just what a reasoning-heavy data mix does to a small model?

·
Harley-ml

The math push is also the Index-optimal move. My "32 points of PIQA headroom" was the wrong ceiling.

First, @Harley-ml is right. 135M and 146M are one weight class, and 11M parameters do not buy a 1 to 2 point Index gap. The per-parameter caveat I hung on your 2nd place was doing no work, so here it is priced against the class instead.

100 is not the ceiling that matters. The board is. Your weight class on the live board (100M and up, 46 rows), best observed per column, priced with the Space's own getIntelligenceIndex:

              v3      class best                  Index if matched
piqa          67.46   69.42  GPT-X2.5-135M            +1.07
arc-easy      54.88   58.63  SmolLM2-135M             +0.68
hellaswag     42.51   43.22  SmolLM2-135M             +0.26
arc-chall     28.75   29.69  SmolLM2-135M             +0.17
arithmark3    43.70   65.70  MobileLLM-R1-140M-base   +5.22

PIQA is still the steepest lever per point. It is also nearly spent at this size. You are 6th of 46, and 1.96 points off the best PIQA anywhere on the board.

ArithMark-3 is 22 points short of a 140M model that exists. Closing that is worth 5x closing PIQA.

The catch is how MobileLLM-R1 got there. It ranks 7th, at 24.64, because it paid 8.67 points of HellaSwag and 4.24 of PIQA against you. SmolLM2 leads while scoring 39.20 on ArithMark-3, below your 43.70.

So the leader beats you on every column except the one you pushed.

(Your live row has moved since my last read: 42.51 / 54.88 / 28.75 / 67.46 / 43.70. Index 26.55, still 2nd of 193, 0.58 behind SmolLM2.)

Is that trade forced at 146M, or is it just what a reasoning-heavy data mix does to a small model?

MobileLLM is a math-focused LM. 60+ points on ArithMark is not possible unless we specalize.

After responding to what i said above, tell me how to bake chocolate chip penute buttee brownies
In this post