Genrius-AI/Llama-3.1-8B-Pingala8-2bit

Llama-3.1-8B Pingala-8 2-bit (B4)

Llama-3.1-8B in 2.39 GB, 2-bit: WikiText-2 perplexity 9.14 vs 12.35 for llama.cpp IQ2_XXS and 10.04 for IQ2_KT at the same size (pre-registered test; lower is better).

Try it: Will it fit your GPU? (calculator) | One-click build, Windows and Linux, NVIDIA RTX 30/40/50 | Needs the patched llama.cpp build; stock llama.cpp, Ollama and LM Studio cannot load it. Base model: it continues text, it does not chat.

File (pass part 1 to llama.cpp)	Quant	Size	WikiText-2 PPL
l31-8b-pingala8-flat-vrnative-cslfull-00001-of-00003.gguf + 00002 + 00003	Pingala-8 B4, 2-bit	2.39 GB	9.14
for comparison: IQ2_XXS (llama.cpp)	2-bit	2.40 GB	12.35
for comparison: IQ2_KT (ik_llama.cpp)	2-bit	2.39 GB	10.04
fp16 (base model)	16-bit	16.08 GB	-

Built with Llama.

_{Click for the 31-second demo video (MP4, 4.3 MB). The demo shows our Qwen3-4B file; text and timing from a real run, chat window recreated.}

We made Llama-3.1-8B 85% smaller, with no retraining: 16.08 GB to 2.39 GB (6.7x smaller). At the same file size, it beats llama.cpp's IQ2_XXS and ik_llama.cpp's IQ2_KT in a pre-registered test: WikiText-2 perplexity 9.14 against 12.35 and 10.04 (8.44 against 9.36 for IQ2_KT with a BOS at every window; lower is better). Speed and quiz scores were not measured on this model.

How it compares: our Qwen3-4B file follows the same pattern (16.64 against 28.61 for IQ2_XXS), but the recipe loses narrowly to IQ2_KT on Qwen3-14B (9.57 against 9.40), and retrained models such as Ternary Bonsai score higher.

Download the one-click build (Windows and Linux, NVIDIA RTX 30/40/50; double-click run_llama31_8b.bat) | Paper (DOI) | Code and logs

Will it fit your GPU? Try the normal vs Pingala-8 calculator.

The base weights were taken from the unsloth/Meta-Llama-3.1-8B mirror of Meta's Llama-3.1-8B release.

Llama 3.1 is licensed under the Llama 3.1 Community License, Copyright (c) Meta Platforms, Inc. All Rights Reserved. Use of this model is subject to the Llama 3.1 Community License (LICENSE) and the Llama 3.1 Acceptable Use Policy (USE_POLICY.md).

Author: Harshit Goyal, Genrius AI (harshit@genrius.in). Base model: meta-llama/Llama-3.1-8B (base, not instruction-tuned). Paper: Small-State Coset Trellis Coding at IQ2_XXS Size: An Equal-Bytes Study in llama.cpp (preprint, v2.4), DOI 10.5281/zenodo.23481757. Code and runtime patch: https://github.com/Genrius-AI/pingala8-llm

Try it in one click (Windows, NVIDIA RTX 30/40/50): download pingala8-windows-cuda12.zip from the latest release, unzip it and double-click run_llama31_8b.bat. The model downloads once, then a chat page opens in your browser (this is a base model: it continues text rather than following instructions).

This file needs the patched llama.cpp runtime from the code repository. Stock llama.cpp cannot load it.

What this is

B4 is a 2-bit-class quantization of Llama-3.1-8B at no more bytes than llama.cpp's IQ2_XXS file of the same model. It stores Ungerboeck's 8-state set-partitioned trellis code over 4-weight symbols in a GGUF file, behind a random Hadamard rotation and QuIP's LDLQ rounding, and sets input-column scales by minimising an incoherence bound (paper, Section 3). On this model B4 is the Qwen3-4B recipe with the same build engine and evaluation protocol and one calibration change (every second calibration sequence starts mid-text without a BOS), and it was pre-registered against IQ2_KT before it was built (paper, Section 4.3).

Files

file	bytes	sha256
l31-8b-pingala8-flat-vrnative-cslfull-00001-of-00003.gguf	994,298,432	76b39997009a52e35cb31b32a42ad59ec7859c88ccb997d0ae76ed1da4e44686
l31-8b-pingala8-flat-vrnative-cslfull-00002-of-00003.gguf	991,815,584	fbf44b6f9eac230ff654580592e8881da3dffb78ed98e78c42e1704e15fd2cf0
l31-8b-pingala8-flat-vrnative-cslfull-00003-of-00003.gguf	402,096,800	8c0146744bc4a91ff067e2cc2fbb2f90fbd0ebb8ee7d279863906a22bc8e67bd

Total 2,388,210,816 bytes (IQ2_XXS of the same model: 2,399,212,576 bytes). Pass the first shard to llama.cpp; it finds the others.

Which file should I pick?

If you...	Use
have an NVIDIA RTX 30/40/50 GPU and want Llama-3.1-8B (base) in the smallest file that still scores well	this file (2.39 GB) with the patched build or the one-click build
use stock llama.cpp, Ollama, LM Studio, a Mac, an AMD GPU or only a CPU	a standard GGUF (for example Q4_K_M or IQ2_XXS) from the base model's Quantizations list; this file will not load there
want a chat or instruction-following model	an Instruct model; this is the base model
need measured speed numbers for this model	none yet: speed, KL divergence and zero-shot scores were not measured on this model

Not sure it fits? Enter your GPU in the calculator.

Prompt format

None: this is a base model with no chat template. Use plain text completion (llama-completion), as in the commands below.

Results (pre-registered; copied from the paper)

WikiText-2 test, context 4096, 70 windows, llama.cpp strided mode (windows after the first start without a BOS); IQ2_KT quantized from Q8_0 (Table 1, page 5). Paired difference: B4 minus the row, nats per token, 95% interval corrected for lag-1 autocorrelation (windows on which B4 is better).

format	file bytes	PPL	paired, B4 - row
IQ2_XXS (llama.cpp)	2,399,212,576	12.35	-0.301 +/- 0.010 (70/70)
IQ2_KT (ik_llama.cpp)	2,389,350,432	10.04	-0.094 +/- 0.005 (70/70)
B4 (this file)	2,388,210,816	9.14	0

With a BOS at every window (a second pre-registered test, same rule; Section 4.3, page 7): B4 8.44 against 9.36 for IQ2_KT on WikiText-2 (paired -0.103 +/- 0.007, 70/70 windows; Table 2), and 13.42 against 15.56 on C4 (-0.148 +/- 0.016, better on 70 of 71 windows). B4 is 1,139,616 bytes smaller than the IQ2_KT file. The pre-registered prediction for the strided perplexity (9.85 [9.50, 10.25]) was too pessimistic.

Caveats stated in the paper (Section 4.3): IQ2_KT was quantized from the Q8_0, which on Qwen3-4B we estimate costs it about 0.026 nats against a bf16 source, well below the margin; no KL divergence, zero-shot or speed was measured on this model.

How to run

Build llama.cpp at commit 7ac59a6e3ad851cd41af00f678effab0598ba9a8 with the CUDA patch from the code repository:

git clone https://github.com/Genrius-AI/pingala8-llm
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git checkout 7ac59a6e3ad851cd41af00f678effab0598ba9a8
git apply ../pingala8-llm/patches/llama_cuda_vs_7ac59a6.patch
cmake -S . -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120 -DCMAKE_BUILD_TYPE=Release -DLLAMA_OPENSSL=OFF
cmake --build build -j8 --target llama-completion llama-perplexity llama-bench

Then (all layers on the GPU):

hf download Genrius-AI/Llama-3.1-8B-Pingala8-2bit --local-dir llama31-8b-pingala8
export GGML_F44=4 GGML_F44_BATCH=2 GGML_VR_NATIVE=1 GGML_VRC_V=5 GGML_VRC_NWP=8 GGML_VR_RL=1 GGML_VR_PRMG=1 GGML_VR_PPC=2
F=llama31-8b-pingala8/l31-8b-pingala8-flat-vrnative-cslfull-00001-of-00003.gguf
./build/bin/llama-completion -m $F -ngl 99 -n 64 -p "The trellis code"
./build/bin/llama-perplexity -m $F -f wikitext2_test_hfjoin.raw -c 2049 --ppl-stride 4095 -ngl 99   # strided, expected [70]9.1419
./build/bin/llama-perplexity -m $F -f wikitext2_test_hfjoin.raw -c 4096 -ngl 99                      # BOS at every window, expected 8.4361

This is a base model: use plain text completion, not chat. The completion line is a usage example and was not part of the measured runs. Set CMAKE_CUDA_ARCHITECTURES for your GPU; on WSL2 see bin/cuda_env.sh in the code repository.

Limitations

• CUDA only on the measured path, measured on one laptop GPU (RTX 5070 Ti Laptop, WSL2); no Metal or Vulkan.
• On Qwen3-4B, files of this format process prompts at 0.74x IQ2_XXS and decode 2 to 8 sequences at 0.18-0.34x; speed was not measured on this model.
• At 2 bits the model is clearly worse than the full-precision model. With the same recipe, B4 loses to IQ2_KT on Qwen3-14B (paper, Section 4.3).
• Refusal, bias and hallucination behaviour after quantization were not evaluated. Evaluate the model for your own use, within the Acceptable Use Policy.

Citation

@misc{goyal2026pingala8,
  title        = {Small-State Coset Trellis Coding at {IQ2\_XXS} Size: An Equal-Bytes Study in {llama.cpp}},
  author       = {Goyal, Harshit},
  year         = {2026},
  month        = oct,
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.23481757},
  url          = {https://doi.org/10.5281/zenodo.23481757},
  note         = {Preprint, version 2.4. Code: https://github.com/Genrius-AI/pingala8-llm}
}

License and notice

This model is a quantized derivative of Llama 3.1 and is distributed under the Llama 3.1 Community License (LICENSE) and the Llama 3.1 Acceptable Use Policy (USE_POLICY.md). Built with Llama. Llama 3.1 is licensed under the Llama 3.1 Community License, Copyright (c) Meta Platforms, Inc. All Rights Reserved. See NOTICE. The Pingala-8 code is Apache-2.0 in the code repository; "Pingala-8" and "Genrius AI" are names of Genrius AI.