gemm_kernels_q5_1.c File Reference

GEMM/GEMV kernels with Q5_1 quantized weights. More...

#include <stdint.h>
#include <stddef.h>
#include <string.h>
#include "ckernel_quant.h"

Go to the source code of this file.

Functions
float	dot_q5_1 (const void *w_q5_1, const float *x, int K)
void	gemm_nt_q5_1 (const float *A, const void *B, const float *bias, float *C, int M, int N, int K)
	GEMM with transposed Q5_1 weights: C = A @ B^T. More...
void	gemm_q5_1 (float *Y, const void *W, const float *X, int M, int N, int K)
	Matrix-matrix multiply with Q5_1 weights. More...
void	gemm_q5_1_backward (float *dX, const void *W, const float *dY, int M, int N, int K)
	Batched backward pass. More...
void	gemv_q5_1 (float *y, const void *W, const float *x, int M, int K)
	Auto-dispatch GEMV. More...
void	gemv_q5_1_backward (float *dX, const void *W, const float *dY, int M, int K)
	Auto-dispatch backward. More...
void	gemv_q5_1_backward_ref (float *dX, const void *W, const float *dY, int M, int K)
	Backward pass: compute input gradient. More...
void	gemv_q5_1_ref (float *y, const void *W, const float *x, int M, int K)
	Matrix-vector multiply with Q5_1 weights (scalar reference) More...

Detailed Description

GEMM/GEMV kernels with Q5_1 quantized weights.

CK-ENGINE KERNEL RULES:

1. NO malloc/free - memory via bump allocator, pointers passed in
2. NO OpenMP - parallelization at orchestrator/codegen layer
3. API must define: inputs, outputs, workspace, and memory layouts
4. Pure computation - deterministic, no side effects

After changes: make test && make llamacpp-parity-full

Q5_1 Format:

• 32 weights per block
• 1 FP16 scale (d) per block
• 1 FP16 minimum (m) per block
• Low 4-bits stored like Q4_1 (16 bytes)
• High 1-bit packed separately (4 bytes)
• 24 bytes per 32 weights = 6.0 bits/weight

Dequantization: w = d * q5 + m where q5 = low4bit | (highbit << 4), giving values 0-31

Operations: Forward: Y = W @ X (W is Q5_1, X and Y are FP32) Backward: dX = W^T @ dY (gradient w.r.t. input)

Definition in file gemm_kernels_q5_1.c.

Function Documentation

dot_q5_1()

float dot_q5_1	(	const void *	w_q5_1,
		const float *	x,
		int	K
	)

Definition at line 357 of file gemm_kernels_q5_1.c.

{
    float result;
    gemv_q5_1(&result, w_q5_1, x, 1, K);
    return result;
}

References gemv_q5_1().

gemm_nt_q5_1()

void gemm_nt_q5_1	(	const float *	A,
		const void *	B,
		const float *	bias,
		float *	C,
		int	M,
		int	N,
		int	K
	)

GEMM with transposed Q5_1 weights: C = A @ B^T.

Parameters

A	Input activations [M x K], row-major FP32
B	Weight matrix in Q5_1 format [N x K], row-major quantized
bias	Optional bias [N], NULL if not used
C	Output [M x N], row-major FP32
M	Batch size (number of tokens)
N	Output dimension
K	Input dimension

Definition at line 309 of file gemm_kernels_q5_1.c.

{
    const block_q5_1 *blocks = (const block_q5_1 *)B;
    const int blocks_per_row = K / QK5_1;
 
    for (int m = 0; m < M; m++) {
        const float *a_row = &A[m * K];
 
        for (int n = 0; n < N; n++) {
            float sum = 0.0f;
 
            for (int b = 0; b < blocks_per_row; b++) {
                const block_q5_1 *block = &blocks[n * blocks_per_row + b];
                const float d = CK_FP16_TO_FP32(block->d);
                const float min = CK_FP16_TO_FP32(block->m);
                const float *ap = &a_row[b * QK5_1];
 
                uint32_t qh;
                memcpy(&qh, block->qh, sizeof(qh));
 
                /* First 16 weights: low nibbles, high bits from qh[0:15] */
                for (int j = 0; j < QK5_1 / 2; j++) {
                    const int lo = (block->qs[j] & 0x0F);
                    const int hi = ((qh >> j) & 1) << 4;
                    sum += (d * (float)(lo | hi) + min) * ap[j];
                }
 
                /* Second 16 weights: high nibbles, high bits from qh[16:31] */
                for (int j = 0; j < QK5_1 / 2; j++) {
                    const int lo = (block->qs[j] >> 4);
                    const int hi = ((qh >> (j + 16)) & 1) << 4;
                    sum += (d * (float)(lo | hi) + min) * ap[j + QK5_1 / 2];
                }
            }
 
            C[m * N + n] = sum + (bias ? bias[n] : 0.0f);
        }
    }
}

References C, CK_FP16_TO_FP32, block_q5_1::d, block_q5_1::m, block_q5_1::qh, QK5_1, and block_q5_1::qs.

Referenced by ck_gemm_nt_quant(), and ck_test_gemm_q5_1().

gemm_q5_1()

void gemm_q5_1	(	float *	Y,
		const void *	W,
		const float *	X,
		int	M,
		int	N,
		int	K
	)

Matrix-matrix multiply with Q5_1 weights.

Definition at line 203 of file gemm_kernels_q5_1.c.

{
    for (int n = 0; n < N; n++) {
        gemv_q5_1(&Y[n * M], W, &X[n * K], M, K);
    }
}

References gemv_q5_1().

gemm_q5_1_backward()

void gemm_q5_1_backward	(	float *	dX,
		const void *	W,
		const float *	dY,
		int	M,
		int	N,
		int	K
	)

Batched backward pass.

Definition at line 284 of file gemm_kernels_q5_1.c.

{
    for (int n = 0; n < N; n++) {
        gemv_q5_1_backward(&dX[n * K], W, &dY[n * M], M, K);
    }
}

References gemv_q5_1_backward().

gemv_q5_1()

void gemv_q5_1	(	float *	y,
		const void *	W,
		const float *	x,
		int	M,
		int	K
	)

Auto-dispatch GEMV.

Definition at line 184 of file gemm_kernels_q5_1.c.

{
#ifdef __AVX512F__
    gemv_q5_1_avx512(y, W, x, M, K);
#else
    gemv_q5_1_ref(y, W, x, M, K);
#endif
}

References gemv_q5_1_ref().

Referenced by ck_test_gemv_q5_1(), dot_q5_1(), and gemm_q5_1().

gemv_q5_1_backward()

void gemv_q5_1_backward	(	float *	dX,
		const void *	W,
		const float *	dY,
		int	M,
		int	K
	)

Auto-dispatch backward.

Definition at line 273 of file gemm_kernels_q5_1.c.

{
    gemv_q5_1_backward_ref(dX, W, dY, M, K);
}

References gemv_q5_1_backward_ref().

Referenced by gemm_q5_1_backward().

gemv_q5_1_backward_ref()

void gemv_q5_1_backward_ref	(	float *	dX,
		const void *	W,
		const float *	dY,
		int	M,
		int	K
	)

Backward pass: compute input gradient.

Parameters

dX	Output gradient w.r.t. input [K]
W	Weight matrix in Q5_1 format [M x K]
dY	Gradient w.r.t. output [M]
M	Number of output rows
K	Number of columns (input dimension)

Definition at line 226 of file gemm_kernels_q5_1.c.

{
    const block_q5_1 *blocks = (const block_q5_1 *)W;
    const int blocks_per_row = K / QK5_1;
 
    /* Zero output gradient */
    memset(dX, 0, K * sizeof(float));
 
    /* Accumulate: dX += W^T @ dY */
    for (int row = 0; row < M; row++) {
        const float dy = dY[row];
 
        for (int b = 0; b < blocks_per_row; b++) {
            const block_q5_1 *block = &blocks[row * blocks_per_row + b];
            const float d = CK_FP16_TO_FP32(block->d);
            const float m = CK_FP16_TO_FP32(block->m);
            float *dxp = &dX[b * QK5_1];
 
            /* Get high bits */
            uint32_t qh;
            memcpy(&qh, block->qh, sizeof(qh));
 
            /* First 16 weights: low nibbles, high bits from qh[0:15] */
            for (int j = 0; j < QK5_1 / 2; j++) {
                const int lo = (block->qs[j] & 0x0F);
                const int hi = ((qh >> j) & 1) << 4;
                const float w = d * (float)(lo | hi) + m;
                dxp[j] += w * dy;
            }
 
            /* Second 16 weights: high nibbles, high bits from qh[16:31] */
            for (int j = 0; j < QK5_1 / 2; j++) {
                const int lo = (block->qs[j] >> 4);
                const int hi = ((qh >> (j + 16)) & 1) << 4;
                const float w = d * (float)(lo | hi) + m;
                dxp[j + QK5_1 / 2] += w * dy;
            }
        }
    }
}

References CK_FP16_TO_FP32, block_q5_1::d, block_q5_1::m, block_q5_1::qh, QK5_1, and block_q5_1::qs.

Referenced by gemv_q5_1_backward().

gemv_q5_1_ref()

void gemv_q5_1_ref	(	float *	y,
		const void *	W,
		const float *	x,
		int	M,
		int	K
	)

Matrix-vector multiply with Q5_1 weights (scalar reference)

Parameters

y	Output vector [M]
W	Weight matrix in Q5_1 format [M x K]
x	Input vector [K]
M	Number of output rows
K	Number of columns (must be multiple of 32)

Definition at line 52 of file gemm_kernels_q5_1.c.

{
    const block_q5_1 *blocks = (const block_q5_1 *)W;
    const int blocks_per_row = K / QK5_1;
 
    for (int row = 0; row < M; row++) {
        float sum = 0.0f;
 
        for (int b = 0; b < blocks_per_row; b++) {
            const block_q5_1 *block = &blocks[row * blocks_per_row + b];
            const float d = CK_FP16_TO_FP32(block->d);
            const float m = CK_FP16_TO_FP32(block->m);
            const float *xp = &x[b * QK5_1];
 
            /* Get high bits as 32-bit integer */
            uint32_t qh;
            memcpy(&qh, block->qh, sizeof(qh));
 
            /* GGML Q5_1 layout: weights 0-15 from LOW nibbles, 16-31 from HIGH nibbles.
             * High bits: bits 0-15 of qh → first half, bits 16-31 → second half.
             * NOT interleaved like Q4_0/Q4_1. */
 
            /* First 16 weights: low nibbles of qs[j], high bit from qh bits 0-15 */
            for (int j = 0; j < QK5_1 / 2; j++) {
                const int lo = (block->qs[j] & 0x0F);
                const int hi = ((qh >> j) & 1) << 4;
                const float w = d * (float)(lo | hi) + m;
                sum += w * xp[j];
            }
 
            /* Second 16 weights: high nibbles of qs[j], high bit from qh bits 16-31 */
            for (int j = 0; j < QK5_1 / 2; j++) {
                const int lo = (block->qs[j] >> 4);
                const int hi = ((qh >> (j + 16)) & 1) << 4;
                const float w = d * (float)(lo | hi) + m;
                sum += w * xp[j + QK5_1 / 2];
            }
        }
 
        y[row] = sum;
    }
}

References CK_FP16_TO_FP32, block_q5_1::d, block_q5_1::m, block_q5_1::qh, QK5_1, and block_q5_1::qs.

Referenced by gemv_q5_1().