ModernBERT pre-trained from scratch on various obfuscated representations of the same 10B tokens.
🔄 In a Training Loop
Daniel Gallagher
DanielGallagherIRE
AI & ML interests
I like encoders because they don't talk back.
Recent Activity
liked a model 1 day ago
zai-org/GLM-5.3-Flash liked a model 3 days ago
allenai/Olmo-3-1025-7B liked a model 3 days ago
allenai/OLMo-2-1124-7BOrganizations
A collection of obfuscated version of a 20B-token sample of the FineWeb Edu dataset.
-
DanielGallagherIRE/FineWeb-Edu-10B-Obfuscation
Viewer • Updated • 19.4M • 736 -
DanielGallagherIRE/FineWeb-Edu-10B-Nouns-Only
Viewer • Updated • 9.71M • 2.94k -
DanielGallagherIRE/FineWeb-Edu-10B-Shuffled
Viewer • Updated • 9.68M • 834 -
DanielGallagherIRE/FineWeb-Edu-10B-PMI-Filtered
Viewer • Updated • 9.71M • 368
ModernBERT pre-trained from scratch on various obfuscated representations of the same 10B tokens.
A collection of obfuscated version of a 20B-token sample of the FineWeb Edu dataset.
-
DanielGallagherIRE/FineWeb-Edu-10B-Obfuscation
Viewer • Updated • 19.4M • 736 -
DanielGallagherIRE/FineWeb-Edu-10B-Nouns-Only
Viewer • Updated • 9.71M • 2.94k -
DanielGallagherIRE/FineWeb-Edu-10B-Shuffled
Viewer • Updated • 9.68M • 834 -
DanielGallagherIRE/FineWeb-Edu-10B-PMI-Filtered
Viewer • Updated • 9.71M • 368
models 42
DanielGallagherIRE/ModernBERT-O3-Shuffling-SENT
0.2B • Updated • 10
DanielGallagherIRE/ModernBERT-O3-Shuffling-DOC
0.2B • Updated • 12
DanielGallagherIRE/ModernBERT-O3-Baseline
0.2B • Updated • 62
DanielGallagherIRE/ModernBERT-O3-PMI-5
0.2B • Updated • 4
DanielGallagherIRE/ModernBERT-O3-PMI-2.5
0.2B • Updated • 5
DanielGallagherIRE/ModernBERT-O3-PMI-10
0.2B • Updated • 11
DanielGallagherIRE/ModernBERT-O3-Nouns-POS
0.2B • Updated • 15
DanielGallagherIRE/ModernBERT-O3-PMI-7.5
0.2B • Updated • 11
DanielGallagherIRE/ModernBERT-O3-PMI-12.5
0.2B • Updated • 11
DanielGallagherIRE/ModernBERT-O3-Nouns-DEL
0.2B • Updated • 11
datasets 23
DanielGallagherIRE/FineWeb-Edu-Irish
Viewer • Updated • 700k • 51
DanielGallagherIRE/FineWeb-Edu-10B-Shuffled-DOC-Truncated
Viewer • Updated • 11.3M • 209
DanielGallagherIRE/FineWeb-Edu-10B-Shuffled-SENT-Truncated
Viewer • Updated • 11.3M • 550
DanielGallagherIRE/FineWeb-Edu-10B-Obfuscation-Truncated
Viewer • Updated • 11.3M • 400
DanielGallagherIRE/FineWeb-Edu-10B-PMI-Filtered-Delete
Viewer • Updated • 9.71M • 340
DanielGallagherIRE/FineWeb-Edu-20B-E2-OBF-FULL
Viewer • Updated • 9.69M • 176
DanielGallagherIRE/FineWeb-Edu-20B-E2-v3-Obfuscated
Viewer • Updated • 9.69M • 167
DanielGallagherIRE/FineWeb-Edu-20B-E2-v3
Viewer • Updated • 9.69M • 400
DanielGallagherIRE/FineWeb-Edu-20B-E2-v2
Updated • 5
DanielGallagherIRE/FineWeb-Edu-20B-E2
Updated • 5