gemm_kernels_q5_0.c File Reference

GEMM/GEMV kernels with Q5_0 quantized weights. More...

#include <stdint.h>
#include <stddef.h>
#include <string.h>
#include <stdio.h>
#include "ckernel_quant.h"
#include "ck_features.h"

Go to the source code of this file.

Functions
void	dequant_q5_0_block (const block_q5_0 *block, float *output)
	Dequantize a single Q5_0 block to FP32. More...
void	dequant_q5_0_row (const void *src, float *dst, size_t n_elements)
	Dequantize Q5_0 row (multiple blocks) More...
float	dot_q5_0 (const void *w_q5_0, const float *x, int K)
void	gemm_nt_q5_0 (const float *A, const void *B, const float *bias, float *C, int M, int N, int K)
void	gemm_nt_q5_0_q8_0 (const void *A_q8, const void *B_q5, const float *bias, float *C, int M, int N, int K)
	Batch GEMM with Q5_0 weights and Q8_0 activations for prefill. More...
void	gemm_nt_q5_0_ref (const float *A, const void *B, const float *bias, float *C, int M, int N, int K)
	GEMM with transposed Q5_0 weights: C = A @ B^T. More...
void	gemm_nt_q5_0_sse_v2 (const float *A, const void *B, const float *bias, float *C, int M, int N, int K)
void	gemm_q5_0 (float *Y, const void *W, const float *X, int M, int N, int K)
	Matrix-matrix multiply with Q5_0 weights. More...
void	gemm_q5_0_backward (float *dX, const void *W, const float *dY, int M, int N, int K)
	Batched backward pass. More...
void	gemv_q5_0 (float *y, const void *W, const float *x, int M, int K)
	Auto-dispatch GEMV for Q5_0 weights based on CPU features. More...
void	gemv_q5_0_backward (float *dX, const void *W, const float *dY, int M, int K)
	Auto-dispatch backward. More...
void	gemv_q5_0_backward_ref (float *dX, const void *W, const float *dY, int M, int K)
	Backward pass: compute input gradient. More...
void	gemv_q5_0_parallel (float *y, const void *W, const float *x, int M, int K, int ith, int nth)
	Parallel reference GEMV for Q5_0 × FP32. More...
void	gemv_q5_0_parallel_simd (float *y, const void *W, const float *x, int M, int K, int ith, int nth)
	Parallel SIMD GEMV for Q5_0 × FP32 with prefetching. More...
void	gemv_q5_0_q8_0 (float *y, const void *W, const void *x_q8, int M, int K)
	Matrix-vector multiply with Q5_0 weights and Q8_0 input. More...
void	gemv_q5_0_q8_0_parallel_simd (float *y, const void *W, const void *x_q8, int M, int K, int ith, int nth)
	Parallel SIMD GEMV for Q5_0 x Q8_0 with prefetching. More...
void	gemv_q5_0_ref (float *y, const void *W, const float *x, int M, int K)
	Matrix-vector multiply with Q5_0 weights (scalar reference) More...
void	vec_dot_q5_0_q8_0 (int n, float *s, const void *vx, const void *vy)
	Auto-dispatch quantized dot product Q5_0 x Q8_0. More...
void	vec_dot_q5_0_q8_0_ref (int n, float *s, const void *vx, const void *vy)
	Quantized dot product: Q5_0 weights x Q8_0 input (scalar reference) More...

Detailed Description

GEMM/GEMV kernels with Q5_0 quantized weights.

CK-ENGINE KERNEL RULES:

1. NO malloc/free - memory via bump allocator, pointers passed in
2. NO OpenMP - parallelization at orchestrator/codegen layer
3. API must define: inputs, outputs, workspace, and memory layouts
4. Pure computation - deterministic, no side effects

After changes: make test && make llamacpp-parity-full

Q5_0 Format:

• 32 weights per block
• 1 FP16 scale per block
• Low 4-bits stored like Q4_0 (16 bytes)
• High 1-bit packed separately (4 bytes)
• 22 bytes per 32 weights = 5.5 bits/weight

Dequantization: w = scale * (q5 - 16) where q5 = low4bit | (highbit << 4), giving values 0-31, then subtract 16 for signed -16 to +15

Operations: Forward: Y = W @ X (W is Q5_0, X and Y are FP32) Backward: dX = W^T @ dY (gradient w.r.t. input)

Definition in file gemm_kernels_q5_0.c.

Function Documentation

dequant_q5_0_block()

void dequant_q5_0_block	(	const block_q5_0 *	block,
		float *	output
	)

Dequantize a single Q5_0 block to FP32.

Parameters

block	Pointer to Q5_0 block (22 bytes)
output	Output FP32 array (32 floats)

Definition at line 161 of file dequant_kernels.c.

{
    const float d = GGML_FP16_TO_FP32(block->d);
 
    /* Get high bits as a 32-bit integer */
    uint32_t qh;
    memcpy(&qh, block->qh, sizeof(qh));
 
    /* llama.cpp Q5_0 layout:
     * - Weight j uses: low nibble of qs[j], high bit from qh bit j
     * - Weight j+16 uses: high nibble of qs[j], high bit from qh bit (j+12)
     */
    for (int j = 0; j < QK5_0 / 2; j++) {
        const uint8_t packed = block->qs[j];
 
        /* Extract low 4 bits for two weights */
        const int lo = (packed & 0x0F);
        const int hi = (packed >> 4);
 
        /* Extract high bits from qh - matches llama.cpp exactly */
        const int xh_0 = ((qh >> (j + 0)) << 4) & 0x10;
        const int xh_1 = ((qh >> (j + 12))) & 0x10;
 
        /* Combine: 5-bit value, range 0-31, then subtract 16 */
        const int q0 = (lo | xh_0) - 16;
        const int q1 = (hi | xh_1) - 16;
 
        output[j] = d * (float)q0;
        output[j + 16] = d * (float)q1;
    }
}

References block_q5_0::d, GGML_FP16_TO_FP32, block_q5_0::qh, QK5_0, and block_q5_0::qs.

Referenced by dequant_q5_0_row().

dequant_q5_0_row()

void dequant_q5_0_row	(	const void *	src,
		float *	dst,
		size_t	n_elements
	)

Dequantize Q5_0 row (multiple blocks)

Definition at line 196 of file dequant_kernels.c.

{
    const block_q5_0 *blocks = (const block_q5_0 *)src;
    const size_t n_blocks = n_elements / QK5_0;
 
    for (size_t b = 0; b < n_blocks; b++) {
        dequant_q5_0_block(&blocks[b], &dst[b * QK5_0]);
    }
}

Referenced by dequant_row().

dot_q5_0()

float dot_q5_0	(	const void *	w_q5_0,
		const float *	x,
		int	K
	)

Definition at line 870 of file gemm_kernels_q5_0.c.

{
    float result;
    gemv_q5_0(&result, w_q5_0, x, 1, K);
    return result;
}

References gemv_q5_0().

gemm_nt_q5_0()

void gemm_nt_q5_0	(	const float *	A,
		const void *	B,
		const float *	bias,
		float *	C,
		int	M,
		int	N,
		int	K
	)

Definition at line 831 of file gemm_kernels_q5_0.c.

{
    /* For decode (M=1), use direct GEMV which has AVX optimization */
    if (M == 1) {
        /* gemm_q5_0 expects column-major output, but we need row-major
         * So we call gemv_q5_0 directly for each output element */
        gemv_q5_0(C, B, A, N, K);
        if (bias) {
            for (int n = 0; n < N; n++) {
                C[n] += bias[n];
            }
        }
        return;
    }
 
    /* For prefill (M>1), use GEMM which dispatches to GEMV with AVX/AVX512 */
    /* gemm_q5_0 produces Y as [batch x M_out]. Here:
     *   batch = M (tokens)
     *   M_out = N (output channels) */
    gemm_q5_0(C, B, A, /*M_out=*/N, /*N_batch=*/M, K);
 
    if (bias) {
        for (int m = 0; m < M; m++) {
            float *row = C + (size_t)m * (size_t)N;
            for (int n = 0; n < N; n++) {
                row[n] += bias[n];
            }
        }
    }
}

References C, gemm_q5_0(), and gemv_q5_0().

Referenced by ck_gemm_nt_quant(), qwen2_0_5b_decode_layer_0_decode(), qwen2_0_5b_decode_layer_10_decode(), qwen2_0_5b_decode_layer_11_decode(), qwen2_0_5b_decode_layer_12_decode(), qwen2_0_5b_decode_layer_13_decode(), qwen2_0_5b_decode_layer_14_decode(), qwen2_0_5b_decode_layer_15_decode(), qwen2_0_5b_decode_layer_16_decode(), qwen2_0_5b_decode_layer_17_decode(), qwen2_0_5b_decode_layer_18_decode(), qwen2_0_5b_decode_layer_19_decode(), qwen2_0_5b_decode_layer_1_decode(), qwen2_0_5b_decode_layer_20_decode(), qwen2_0_5b_decode_layer_21_decode(), qwen2_0_5b_decode_layer_22_decode(), qwen2_0_5b_decode_layer_23_decode(), qwen2_0_5b_decode_layer_2_decode(), qwen2_0_5b_decode_layer_3_decode(), qwen2_0_5b_decode_layer_4_decode(), qwen2_0_5b_decode_layer_5_decode(), qwen2_0_5b_decode_layer_6_decode(), qwen2_0_5b_decode_layer_7_decode(), qwen2_0_5b_decode_layer_8_decode(), and qwen2_0_5b_decode_layer_9_decode().

gemm_nt_q5_0_q8_0()

void gemm_nt_q5_0_q8_0	(	const void *	A_q8,
		const void *	B_q5,
		const float *	bias,
		float *	C,
		int	M,
		int	N,
		int	K
	)

Batch GEMM with Q5_0 weights and Q8_0 activations for prefill.

Computes C = A @ B^T + bias where: A: [M x K] Q8_0 quantized activations (M tokens, K features) B: [N x K] Q5_0 quantized weights (N outputs, K features) C: [M x N] FP32 output

This is the INT8 batch kernel for prefill, using pre-quantized activations to avoid FP32->Q8_0 conversion overhead per operation.

Parameters

A_q8	Input activations in Q8_0 format [M rows of K/32 blocks each]
B_q5	Weights in Q5_0 format [N rows of K/32 blocks each]
bias	Optional bias vector [N], NULL if not used
C	Output matrix [M x N], row-major FP32
M	Batch size (number of tokens)
N	Output dimension (number of output features)
K	Input dimension (must be multiple of 32)

Definition at line 1617 of file gemm_kernels_q5_0.c.

{
    const block_q5_0 *weights = (const block_q5_0 *)B_q5;
    const block_q8_0 *inputs = (const block_q8_0 *)A_q8;
    const int blocks_per_row = K / QK5_0;
 
    for (int m = 0; m < M; m++) {
        const block_q8_0 *input_row = &inputs[m * blocks_per_row];
 
        for (int n = 0; n < N; n++) {
            const block_q5_0 *weight_row = &weights[n * blocks_per_row];
            float *out = &C[m * N + n];
 
            /* Dispatches to vec_dot_q5_0_q8_0_avx (2x block unrolled) on AVX */
            vec_dot_q5_0_q8_0(K, out, weight_row, input_row);
 
            if (bias) {
                *out += bias[n];
            }
        }
    }
}

References C, QK5_0, and vec_dot_q5_0_q8_0().

Referenced by ck_test_gemm_q5_0(), gemm_nt_q8_0_dispatch(), and gemm_nt_q8_0_mlp_dispatch().

gemm_nt_q5_0_ref()

void gemm_nt_q5_0_ref	(	const float *	A,
		const void *	B,
		const float *	bias,
		float *	C,
		int	M,
		int	N,
		int	K
	)

GEMM with transposed Q5_0 weights: C = A @ B^T.

Parameters

A	Input activations [M x K], row-major FP32
B	Weight matrix in Q5_0 format [N x K], row-major quantized
bias	Optional bias [N], NULL if not used
C	Output [M x N], row-major FP32
M	Batch size (number of tokens)
N	Output dimension (number of rows in B)
K	Input dimension

Definition at line 788 of file gemm_kernels_q5_0.c.

{
    const block_q5_0 *blocks = (const block_q5_0 *)B;
    const int blocks_per_row = K / QK5_0;
 
    for (int m = 0; m < M; m++) {
        const float *a_row = &A[m * K];
 
        for (int n = 0; n < N; n++) {
            float sum = 0.0f;
 
            for (int b = 0; b < blocks_per_row; b++) {
                const block_q5_0 *block = &blocks[n * blocks_per_row + b];
                const float d = CK_FP16_TO_FP32(block->d);
                const float *ap = &a_row[b * QK5_0];
 
                uint32_t qh;
                memcpy(&qh, block->qh, sizeof(qh));
 
                /* llama.cpp Q5_0 layout - note j+12 for second weight high bit */
                for (int j = 0; j < QK5_0 / 2; j++) {
                    const uint8_t packed = block->qs[j];
                    const int lo = (packed & 0x0F);
                    const int hi = (packed >> 4);
                    const int xh_0 = ((qh >> (j + 0)) << 4) & 0x10;
                    const int xh_1 = ((qh >> (j + 12))) & 0x10;
                    const int q0 = (lo | xh_0) - 16;
                    const int q1 = (hi | xh_1) - 16;
 
                    sum += d * (float)q0 * ap[j];
                    sum += d * (float)q1 * ap[j + 16];
                }
            }
 
            C[m * N + n] = sum + (bias ? bias[n] : 0.0f);
        }
    }
}

References C, CK_FP16_TO_FP32, block_q5_0::d, block_q5_0::qh, QK5_0, and block_q5_0::qs.

Referenced by gemm_nt_q5_0_sse_v2().

gemm_nt_q5_0_sse_v2()

void gemm_nt_q5_0_sse_v2	(	const float *	A,
		const void *	B,
		const float *	bias,
		float *	C,
		int	M,
		int	N,
		int	K
	)

Definition at line 77 of file gemm_kernels_q5_0_sse_v2.c.

{
    if (K % QK_K != 0) {
        gemm_nt_q5_0_ref(A, B, bias, C, M, N, K);
        return;
    }
 
    size_t q8_size = (K / QK_K) * sizeof(block_q8_K);
    block_q8_K *A_q8 = (block_q8_K *)alloca(q8_size);
 
    const block_q5_0 *weights = (const block_q5_0 *)B;
    const int blocks_per_row = K / 32;
 
    for (int m = 0; m < M; m++) {
        quantize_row_q8_k(&A[m * K], A_q8, K);
 
        for (int n = 0; n < N; n++) {
            float sumf = 0.0f;
            const block_q5_0 *w_row = weights + n * blocks_per_row;
 
            for (int b = 0; b < blocks_per_row; b++) {
                int q8_block_idx = (b * 32) / QK_K;
                int q8_offset = (b * 32) % QK_K;
                sumf += dot_q5_0_q8_k_32_sse(&w_row[b], &A_q8[q8_block_idx], q8_offset);
            }
 
            C[m * N + n] = sumf + (bias ? bias[n] : 0.0f);
        }
    }
}

gemm_q5_0()

void gemm_q5_0	(	float *	Y,
		const void *	W,
		const float *	X,
		int	M,
		int	N,
		int	K
	)

Matrix-matrix multiply with Q5_0 weights.

Definition at line 682 of file gemm_kernels_q5_0.c.

{
    for (int n = 0; n < N; n++) {
        gemv_q5_0(&Y[n * M], W, &X[n * K], M, K);
    }
}

References gemv_q5_0().

Referenced by gemm_nt_q5_0().

gemm_q5_0_backward()

void gemm_q5_0_backward	(	float *	dX,
		const void *	W,
		const float *	dY,
		int	M,
		int	N,
		int	K
	)

Batched backward pass.

Definition at line 762 of file gemm_kernels_q5_0.c.

{
    for (int n = 0; n < N; n++) {
        gemv_q5_0_backward(&dX[n * K], W, &dY[n * M], M, K);
    }
}

References gemv_q5_0_backward().

gemv_q5_0()

void gemv_q5_0	(	float *	y,
		const void *	W,
		const float *	x,
		int	M,
		int	K
	)

Auto-dispatch GEMV for Q5_0 weights based on CPU features.

Dispatch priority (best available):

1. AVX-512 (512-bit vectors) - Intel Skylake-X+
2. AVX2+FMA (256-bit vectors) - Intel Haswell+
3. AVX (256-bit vectors) - Intel Sandy Bridge+
4. SSE4.1 (128-bit vectors) - Intel Nehalem+
5. Reference (scalar) - Fallback

Uses ck_features.h for standardized feature detection.

Parameters

y	Output vector [M]
W	Weight matrix in Q5_0 format [M x K]
x	Input vector [K]
M	Number of output rows
K	Number of input columns (hidden dimension)

Definition at line 547 of file gemm_kernels_q5_0.c.

{
// Dispatch order: AVX512 > AVX2 > AVX > SSE > ref
#if defined(__AVX512F__)
    gemv_q5_0_avx512(y, W, x, M, K);
#elif defined(__AVX2__)
    gemv_q5_0_avx2(y, W, x, M, K);
#elif defined(__AVX__)
    gemv_q5_0_avx(y, W, x, M, K);
#elif defined(__SSE4_1__)
    gemv_q5_0_sse_v2(y, W, x, M, K);
#else
    gemv_q5_0_ref(y, W, x, M, K);
#endif
}

References gemv_q5_0_ref().

Referenced by dot_q5_0(), gemm_nt_q5_0(), and gemm_q5_0().

gemv_q5_0_backward()

void gemv_q5_0_backward	(	float *	dX,
		const void *	W,
		const float *	dY,
		int	M,
		int	K
	)

Auto-dispatch backward.

Definition at line 751 of file gemm_kernels_q5_0.c.

{
    gemv_q5_0_backward_ref(dX, W, dY, M, K);
}

References gemv_q5_0_backward_ref().

Referenced by gemm_q5_0_backward().

gemv_q5_0_backward_ref()

void gemv_q5_0_backward_ref	(	float *	dX,
		const void *	W,
		const float *	dY,
		int	M,
		int	K
	)

Backward pass: compute input gradient.

Parameters

dX	Output gradient w.r.t. input [K]
W	Weight matrix in Q5_0 format [M x K]
dY	Gradient w.r.t. output [M]
M	Number of output rows
K	Number of columns (input dimension)

Definition at line 705 of file gemm_kernels_q5_0.c.

{
    const block_q5_0 *blocks = (const block_q5_0 *)W;
    const int blocks_per_row = K / QK5_0;
 
    /* Zero output gradient */
    memset(dX, 0, K * sizeof(float));
 
    /* Accumulate: dX += W^T @ dY */
    for (int row = 0; row < M; row++) {
        const float dy = dY[row];
 
        for (int b = 0; b < blocks_per_row; b++) {
            const block_q5_0 *block = &blocks[row * blocks_per_row + b];
            const float d = CK_FP16_TO_FP32(block->d);
            float *dxp = &dX[b * QK5_0];
 
            /* Get high bits */
            uint32_t qh;
            memcpy(&qh, block->qh, sizeof(qh));
 
            /* llama.cpp Q5_0 layout - note j+12 for second weight high bit */
            for (int j = 0; j < QK5_0 / 2; j++) {
                const uint8_t packed = block->qs[j];
 
                /* Extract and reconstruct 5-bit values */
                const int lo = (packed & 0x0F);
                const int hi = (packed >> 4);
                const int xh_0 = ((qh >> (j + 0)) << 4) & 0x10;
                const int xh_1 = ((qh >> (j + 12))) & 0x10;
                const int q0 = (lo | xh_0) - 16;
                const int q1 = (hi | xh_1) - 16;
 
                dxp[j] += d * (float)q0 * dy;
                dxp[j + 16] += d * (float)q1 * dy;
            }
        }
    }
}

References CK_FP16_TO_FP32, block_q5_0::d, block_q5_0::qh, QK5_0, and block_q5_0::qs.

Referenced by gemv_q5_0_backward().

gemv_q5_0_parallel()

void gemv_q5_0_parallel	(	float *	y,
		const void *	W,
		const float *	x,
		int	M,
		int	K,
		int	ith,
		int	nth
	)

Parallel reference GEMV for Q5_0 × FP32.

Definition at line 576 of file gemm_kernels_q5_0.c.

{
    if (!y || !W || !x || M <= 0 || K <= 0) return;
    if (ith < 0 || nth <= 0 || ith >= nth) return;
 
    const int dr = (M + nth - 1) / nth;
    const int r0 = dr * ith;
    const int r1 = (r0 + dr < M) ? (r0 + dr) : M;
 
    if (r0 >= M) return;
 
    const block_q5_0 *blocks = (const block_q5_0 *)W;
    const int blocks_per_row = K / QK5_0;
 
    for (int row = r0; row < r1; row++) {
        float sum = 0.0f;
        for (int b = 0; b < blocks_per_row; b++) {
            const block_q5_0 *block = &blocks[row * blocks_per_row + b];
            const float d = CK_FP16_TO_FP32(block->d);
            const float *xp = &x[b * QK5_0];
 
            uint32_t qh;
            memcpy(&qh, block->qh, sizeof(qh));
 
            for (int j = 0; j < QK5_0 / 2; j++) {
                const uint8_t packed = block->qs[j];
                const int lo = (packed & 0x0F);
                const int hi = (packed >> 4);
                const int xh_0 = ((qh >> (j + 0)) << 4) & 0x10;
                const int xh_1 = ((qh >> (j + 12))) & 0x10;
                const int w0 = (lo | xh_0) - 16;
                const int w1 = (hi | xh_1) - 16;
                sum += d * (w0 * xp[j] + w1 * xp[j + QK5_0/2]);
            }
        }
        y[row] = sum;
    }
}

References CK_FP16_TO_FP32, block_q5_0::d, block_q5_0::qh, QK5_0, and block_q5_0::qs.

Referenced by gemv_q5_0_parallel_simd().

gemv_q5_0_parallel_simd()

void gemv_q5_0_parallel_simd	(	float *	y,
		const void *	W,
		const float *	x,
		int	M,
		int	K,
		int	ith,
		int	nth
	)

Parallel SIMD GEMV for Q5_0 × FP32 with prefetching.

Definition at line 622 of file gemm_kernels_q5_0.c.

{
    if (!y || !W || !x || M <= 0 || K <= 0) return;
    if (ith < 0 || nth <= 0 || ith >= nth) return;
 
    const int dr = (M + nth - 1) / nth;
    const int r0 = dr * ith;
    const int r1 = (r0 + dr < M) ? (r0 + dr) : M;
 
    if (r0 >= M) return;
 
    const block_q5_0 *blocks = (const block_q5_0 *)W;
    const int blocks_per_row = K / QK5_0;
 
#if defined(__AVX__) || defined(__SSE4_1__)
    /* Prefetch first few rows */
    const int PREFETCH_ROWS = 4;
    for (int p = 0; p < PREFETCH_ROWS && r0 + p < r1; ++p) {
        const char *row_ptr = (const char *)(blocks + (r0 + p) * blocks_per_row);
        _mm_prefetch(row_ptr, _MM_HINT_T0);
        _mm_prefetch(row_ptr + 64, _MM_HINT_T0);
    }
 
    for (int row = r0; row < r1; ++row) {
        /* Prefetch rows ahead */
        if (row + PREFETCH_ROWS < r1) {
            const char *prefetch_ptr = (const char *)(blocks + (row + PREFETCH_ROWS) * blocks_per_row);
            _mm_prefetch(prefetch_ptr, _MM_HINT_T0);
            _mm_prefetch(prefetch_ptr + 64, _MM_HINT_T0);
        }
 
        /* Use SIMD dot product for this row */
#if defined(__AVX512F__)
        /* Call single-row AVX512 implementation */
        gemv_q5_0_avx512(&y[row], (const char *)blocks + row * blocks_per_row * sizeof(block_q5_0), x, 1, K);
#elif defined(__AVX2__)
        gemv_q5_0_avx2(&y[row], (const char *)blocks + row * blocks_per_row * sizeof(block_q5_0), x, 1, K);
#elif defined(__AVX__)
        gemv_q5_0_avx(&y[row], (const char *)blocks + row * blocks_per_row * sizeof(block_q5_0), x, 1, K);
#else
        gemv_q5_0_sse_v2(&y[row], (const char *)blocks + row * blocks_per_row * sizeof(block_q5_0), x, 1, K);
#endif
    }
#else
    /* Fallback to reference parallel */
    gemv_q5_0_parallel(y, W, x, M, K, ith, nth);
#endif
}

References gemv_q5_0_parallel(), and QK5_0.

gemv_q5_0_q8_0()

void gemv_q5_0_q8_0	(	float *	y,
		const void *	W,
		const void *	x_q8,
		int	M,
		int	K
	)

Matrix-vector multiply with Q5_0 weights and Q8_0 input.

Parameters

y	Output vector [M]
W	Weight matrix in Q5_0 format [M x K]
x_q8	Input vector in Q8_0 format [K]
M	Number of output rows
K	Number of columns (must be multiple of 32)

Definition at line 1529 of file gemm_kernels_q5_0.c.

{
    const block_q5_0 *w_blocks = (const block_q5_0 *)W;
    const block_q8_0 *x_blocks = (const block_q8_0 *)x_q8;
    const int blocks_per_row = K / QK5_0;
 
    for (int row = 0; row < M; row++) {
        vec_dot_q5_0_q8_0(K, &y[row],
                          &w_blocks[row * blocks_per_row],
                          x_blocks);
    }
}

References QK5_0, and vec_dot_q5_0_q8_0().

Referenced by ck_test_gemv_q5_0(), and ck_test_gemv_q5_0_q8_0().

gemv_q5_0_q8_0_parallel_simd()

void gemv_q5_0_q8_0_parallel_simd	(	float *	y,
		const void *	W,
		const void *	x_q8,
		int	M,
		int	K,
		int	ith,
		int	nth
	)

Parallel SIMD GEMV for Q5_0 x Q8_0 with prefetching.

Each thread processes rows [r0, r1) where r0 = ith * ceil(M/nth). Uses vec_dot_q5_0_q8_0 dispatch (auto-selects AVX512/AVX/SSE/scalar).

Definition at line 1551 of file gemm_kernels_q5_0.c.

{
    if (!y || !W || !x_q8 || M <= 0 || K <= 0) return;
    if (ith < 0 || nth <= 0 || ith >= nth) return;
 
    const int dr = (M + nth - 1) / nth;
    const int r0 = dr * ith;
    const int r1 = (r0 + dr < M) ? (r0 + dr) : M;
 
    if (r0 >= M) return;
 
    const block_q5_0 *w_blocks = (const block_q5_0 *)W;
    const block_q8_0 *x_blocks = (const block_q8_0 *)x_q8;
    const int blocks_per_row = K / QK5_0;
 
#if defined(__AVX__) || defined(__SSE4_1__)
    const int PREFETCH_ROWS = 4;
    for (int p = 0; p < PREFETCH_ROWS && r0 + p < r1; ++p) {
        const char *row_ptr = (const char *)(w_blocks + (r0 + p) * blocks_per_row);
        _mm_prefetch(row_ptr, _MM_HINT_T0);
        _mm_prefetch(row_ptr + 64, _MM_HINT_T0);
    }
 
    for (int row = r0; row < r1; ++row) {
        if (row + PREFETCH_ROWS < r1) {
            const char *pf = (const char *)(w_blocks + (row + PREFETCH_ROWS) * blocks_per_row);
            _mm_prefetch(pf, _MM_HINT_T0);
            _mm_prefetch(pf + 64, _MM_HINT_T0);
        }
 
        vec_dot_q5_0_q8_0(K, &y[row],
                          &w_blocks[row * blocks_per_row],
                          x_blocks);
    }
#else
    for (int row = r0; row < r1; row++) {
        vec_dot_q5_0_q8_0(K, &y[row],
                          &w_blocks[row * blocks_per_row],
                          x_blocks);
    }
#endif
}

References QK5_0, and vec_dot_q5_0_q8_0().

gemv_q5_0_ref()

void gemv_q5_0_ref	(	float *	y,
		const void *	W,
		const float *	x,
		int	M,
		int	K
	)

Matrix-vector multiply with Q5_0 weights (scalar reference)

Parameters

y	Output vector [M]
W	Weight matrix in Q5_0 format [M x K]
x	Input vector [K]
M	Number of output rows
K	Number of columns (must be multiple of 32)

Definition at line 64 of file gemm_kernels_q5_0.c.

{
    const block_q5_0 *blocks = (const block_q5_0 *)W;
    const int blocks_per_row = K / QK5_0;
 
    for (int row = 0; row < M; row++) {
        float sum = 0.0f;
 
        for (int b = 0; b < blocks_per_row; b++) {
            const block_q5_0 *block = &blocks[row * blocks_per_row + b];
            const float d = CK_FP16_TO_FP32(block->d);
            const float *xp = &x[b * QK5_0];
 
            /* Get high bits as 32-bit integer */
            uint32_t qh;
            memcpy(&qh, block->qh, sizeof(qh));
 
            /* llama.cpp Q5_0 layout:
             * - Weight j uses: low nibble of qs[j], high bit from qh bit j
             * - Weight j+16 uses: high nibble of qs[j], high bit from qh bit (j+12)
             * Note: j+12 not j+16 for the high bit of the second weight!
             */
            for (int j = 0; j < QK5_0 / 2; j++) {
                const uint8_t packed = block->qs[j];
 
                /* Extract nibbles */
                const int lo = (packed & 0x0F);
                const int hi = (packed >> 4);
 
                /* Extract high bits - matches llama.cpp exactly */
                const int xh_0 = ((qh >> (j + 0)) << 4) & 0x10;
                const int xh_1 = ((qh >> (j + 12))) & 0x10;
 
                /* Combine to 5-bit signed value */
                const int q0 = (lo | xh_0) - 16;
                const int q1 = (hi | xh_1) - 16;
 
                /* Weights at indices j and j+16 */
                sum += d * (float)q0 * xp[j];
                sum += d * (float)q1 * xp[j + 16];
            }
        }
 
        y[row] = sum;
    }
}

References CK_FP16_TO_FP32, block_q5_0::d, block_q5_0::qh, QK5_0, and block_q5_0::qs.

Referenced by gemv_q5_0().

vec_dot_q5_0_q8_0()

void vec_dot_q5_0_q8_0	(	int	n,
		float *	s,
		const void *	vx,
		const void *	vy
	)

Auto-dispatch quantized dot product Q5_0 x Q8_0.

Dispatch priority:

1. AVX512 (best performance on modern Intel/AMD)
2. AVX (256-bit float ops, works on Sandy/Ivy Bridge and newer)
3. SSSE3 (128-bit fallback)
4. Reference scalar (last resort)

Definition at line 1498 of file gemm_kernels_q5_0.c.

{
#if defined(__AVX512F__)
    vec_dot_q5_0_q8_0_avx512(n, s, vx, vy);
#elif defined(__AVX__)
    /* AVX for 256-bit float ops (works on Ivy Bridge and newer) */
    vec_dot_q5_0_q8_0_avx(n, s, vx, vy);
#elif defined(__SSSE3__)
    /* SSSE3 - most efficient on older CPUs */
    vec_dot_q5_0_q8_0_sse(n, s, vx, vy);
#else
    vec_dot_q5_0_q8_0_ref(n, s, vx, vy);
#endif
}

References vec_dot_q5_0_q8_0_ref().

Referenced by ck_test_vec_dot_q5_0_q8_0(), gemm_nt_q5_0_q8_0(), gemv_fused_q5_0_bias_parallel_omp(), gemv_q5_0_from_fp32(), gemv_q5_0_q8_0(), gemv_q5_0_q8_0_parallel_omp(), gemv_q5_0_q8_0_parallel_simd(), mega_fused_attention_decode_q5_0(), mega_fused_attention_decode_q5_0_parallel_simd(), and out_proj_head_major_q5_0_q8_0().

vec_dot_q5_0_q8_0_ref()

void vec_dot_q5_0_q8_0_ref	(	int	n,
		float *	s,
		const void *	vx,
		const void *	vy
	)

Quantized dot product: Q5_0 weights x Q8_0 input (scalar reference)

Parameters

n	Number of elements (must be multiple of 32)
s	Output: scalar dot product result
vx	Q5_0 quantized weights
vy	Q8_0 quantized input

Definition at line 899 of file gemm_kernels_q5_0.c.

{
    const int qk = QK5_0;  /* 32 */
    const int nb = n / qk;
 
    const block_q5_0 *x = (const block_q5_0 *)vx;
    const block_q8_0 *y = (const block_q8_0 *)vy;
 
    float sumf = 0.0f;
 
    for (int ib = 0; ib < nb; ib++) {
        /* Load high bits for this block */
        uint32_t qh;
        memcpy(&qh, x[ib].qh, sizeof(qh));
 
        int sumi0 = 0;
        int sumi1 = 0;
 
        for (int j = 0; j < qk / 2; j++) {
            /* Extract high bits - matches llama.cpp exactly */
            const uint8_t xh_0 = ((qh & (1u << (j + 0))) >> (j + 0)) << 4;
            const uint8_t xh_1 = ((qh & (1u << (j + 16))) >> (j + 12));
 
            /* Reconstruct 5-bit signed values (-16 to +15) */
            const int32_t x0 = (int8_t)(((x[ib].qs[j] & 0x0F) | xh_0) - 16);
            const int32_t x1 = (int8_t)(((x[ib].qs[j] >> 4) | xh_1) - 16);
 
            /* Integer dot product with Q8_0 values */
            sumi0 += x0 * y[ib].qs[j];
            sumi1 += x1 * y[ib].qs[j + qk / 2];
        }
 
        int sumi = sumi0 + sumi1;
        sumf += (CK_FP16_TO_FP32(x[ib].d) * CK_FP16_TO_FP32(y[ib].d)) * sumi;
    }
 
    *s = sumf;
}

References CK_FP16_TO_FP32, QK5_0, and block_q8_0::qs.

Referenced by vec_dot_q5_0_q8_0().