This update provides significant performance improvements for BitNet inference on CPU through paralleled kernel implementations, native I2_S GEMM/GEMV support, configurable tiling block size and embedding quantization.

• Parallel Weight & Activation Computation
  Implemented parallel processing of weights and activations in the W2A8 vet_dot kernel, achieving improved throughput on both x86 and ARM architectures.

• Native I2_S GEMM & GEMV Support
  Integrated I2_S GEMM and GEMV operations into ggml library, making them fully compatible with the llama.cpp architecture. This enables seamless integration with existing inference pipelines.

• Configurable Tiling & Parallelism
  Introduced configurable GEMM & GEMV block sizes and parallelism levels, allowing performance fine-tuning for different CPU architectures.

• Embedding Quantization
  Added support for embedding layer quantization with Q6_K format, reducing memory footprint and improving inference speed while maintaining high accuracy.

The include/gemm-config.h file controls kernel behavior:

#define ROW_BLOCK_SIZE 4
#define COL_BLOCK_SIZE 128
#define PARALLEL_SIZE 4

Modify these values based on your CPU cache size and architecture for optimal performance. Users can fine-tune performance on their machine through include/gemm-config.h.

To use embedding quantization for additional speedup:

Using setup_env.py:

python setup_env.py --quant-embd

This automatically converts embeddings to Q6_K format.

Manual conversion:

build/bin/llama-quantize --token-embedding-type Q6_K models/BitNet-b1.58-2B-4T/ggml-model-f32.gguf models/BitNet-b1.58-2B-4T/ggml-model-i2_s-embed-q6_k.gguf I2_S 1 1

The kernel implements two parallelization strategies:

• Weight Parallel: Processes multiple weight rows/columns in a single kernel call, reducing kernel launch overhead.

• Activation Parallel: Built on top of weight parallel, amortizes the I2_S weight unpacking cost across multiple activation elements.

Recommendation: For I2_S quantization format, activation parallel is recommended due to the unpack operation benefits. The current kernel defaults to activation parallel.

Kernel Performance Comparison:

Integrated I2_S quantization format into llama.cpp's compute graph:

• GEMV Operations: Optimized matrix-vector multiplication for token generation.
• GEMM Operations: Efficient matrix-matrix multiplication for prompt processing.
• Tiling Strategy: Configurable block sizes for optimal cache utilization.

Fine-tuning kernel parameters for optimal performance on specific hardware:

Example Configuration (x86, AMD EPYC 7V13):

• Method: Activation Parallel
• Threads: 8
• Workload: 128 prompt tokens (pp128)

Fine-tuning Parameters:

• Parallelism Degree: [2, 4, 8]
• Row Block Size: [2, 4, 8, 16, 32]
• Column Block Size: [32, 64, 128, 256, 512, 1024]

Fine-tuning Results:

Shows throughput (tokens/s) for various configurations.

Optimal Configuration: Under this setup (x86, 8 threads, pp128), the best performance is achieved with parallelism degree = 4, row block size = 4, and column block size = 128.

Evaluated multiple embedding quantization formats to balance memory usage, model quality, and inference speed:

Perplexity Comparison:

Inference Speed Comparison:

Token generation throughput (tg128) for different embedding quantization types.

Recommendation: Based on comprehensive evaluation of memory footprint, perplexity preservation, and inference speed, Q6_K is selected as the optimal embedding quantization format.

Comparison of optimized parallel kernels vs. original implementation:

Test Configuration:

• Model: BitNet-b1.58-2B-4T
• Hardware: AMD EPYC 7V13
• Threads: 1 / 2 / 4 / 8 / 12 / 16
• Test: 128 prompt tokens (pp128) + 128 generated tokens (tg128)
• Method: Activation Parallel

Test Configuration:

• Model: BitNet-b1.58-2B-4T
• Hardware: Intel i7-13800H
• Threads: 1 / 2 / 4 / 6
• Test: 128 prompt tokens (pp128) + 128 generated tokens (tg128)
• Method: Activation Parallel

Test Configuration:

• Model: BitNet-b1.58-2B-4T
• Hardware: Cobalt 100
• Threads: 1 / 2 / 4 / 8
• Test: 128 prompt tokens (pp128) + 128 generated tokens (tg128)
• Method: Activation Parallel

• src/ggml-bitnet-mad.cpp: Parallel kernel implementations
• 3rdparty/llama.cpp/ggml/src/ggml.c: GEMM/GEMV integration
• include/gemm-config.h: Configuration file

• ✅ x86-64 with AVX2
• ✅ ARM with NEON
• ✅ ARM with DOTPROD extension