gemm_kernels_q4_1.c File Reference

GEMM/GEMV kernels with Q4_1 quantized weights. More...

#include <stdint.h>
#include <stddef.h>
#include <string.h>
#include "ckernel_quant.h"

Go to the source code of this file.

Functions
float	dot_q4_1 (const void *w_q4_1, const float *x, int K)
void	gemm_nt_q4_1 (const float *A, const void *B, const float *bias, float *C, int M, int N, int K)
	GEMM with transposed Q4_1 weights: C = A @ B^T. More...
void	gemm_q4_1 (float *Y, const void *W, const float *X, int M, int N, int K)
	Matrix-matrix multiply with Q4_1 weights. More...
void	gemm_q4_1_backward (float *dX, const void *W, const float *dY, int M, int N, int K)
	Batched backward pass. More...
void	gemv_q4_1 (float *y, const void *W, const float *x, int M, int K)
	Auto-dispatch GEMV. More...
void	gemv_q4_1_backward (float *dX, const void *W, const float *dY, int M, int K)
	Auto-dispatch backward. More...
void	gemv_q4_1_backward_ref (float *dX, const void *W, const float *dY, int M, int K)
	Backward pass: compute input gradient. More...
void	gemv_q4_1_ref (float *y, const void *W, const float *x, int M, int K)
	Matrix-vector multiply with Q4_1 weights (scalar reference) More...

Detailed Description

GEMM/GEMV kernels with Q4_1 quantized weights.

CK-ENGINE KERNEL RULES:

1. NO malloc/free - memory via bump allocator, pointers passed in
2. NO OpenMP - parallelization at orchestrator/codegen layer
3. API must define: inputs, outputs, workspace, and memory layouts
4. Pure computation - deterministic, no side effects

After changes: make test && make llamacpp-parity-full

Q4_1 Format:

• 32 weights per block
• 1 FP16 scale (d) per block
• 1 FP16 minimum (m) per block
• 20 bytes per 32 weights = 5.0 bits/weight

Dequantization: w = d * q + m where q is the 4-bit unsigned value (0-15)

Operations: Forward: Y = W @ X (W is Q4_1, X and Y are FP32) Backward: dX = W^T @ dY (gradient w.r.t. input)

Definition in file gemm_kernels_q4_1.c.

Function Documentation

dot_q4_1()

float dot_q4_1	(	const void *	w_q4_1,
		const float *	x,
		int	K
	)

Definition at line 299 of file gemm_kernels_q4_1.c.

{
    float result;
    gemv_q4_1(&result, w_q4_1, x, 1, K);
    return result;
}

References gemv_q4_1().

gemm_nt_q4_1()

void gemm_nt_q4_1	(	const float *	A,
		const void *	B,
		const float *	bias,
		float *	C,
		int	M,
		int	N,
		int	K
	)

GEMM with transposed Q4_1 weights: C = A @ B^T.

Parameters

A	Input activations [M x K], row-major FP32
B	Weight matrix in Q4_1 format [N x K], row-major quantized
bias	Optional bias [N], NULL if not used
C	Output [M x N], row-major FP32
M	Batch size (number of tokens)
N	Output dimension
K	Input dimension

Definition at line 256 of file gemm_kernels_q4_1.c.

{
    const block_q4_1 *blocks = (const block_q4_1 *)B;
    const int blocks_per_row = K / QK4_1;
 
    for (int m = 0; m < M; m++) {
        const float *a_row = &A[m * K];
 
        for (int n = 0; n < N; n++) {
            float sum = 0.0f;
 
            for (int b = 0; b < blocks_per_row; b++) {
                const block_q4_1 *block = &blocks[n * blocks_per_row + b];
                const float d = CK_FP16_TO_FP32(block->d);
                const float min = CK_FP16_TO_FP32(block->m);
                const float *ap = &a_row[b * QK4_1];
 
                for (int i = 0; i < QK4_1 / 2; i++) {
                    const uint8_t packed = block->qs[i];
                    const int q0 = (packed & 0x0F);
                    const int q1 = (packed >> 4);
 
                    const float w0 = d * (float)q0 + min;
                    const float w1 = d * (float)q1 + min;
 
                    sum += w0 * ap[2 * i + 0];
                    sum += w1 * ap[2 * i + 1];
                }
            }
 
            C[m * N + n] = sum + (bias ? bias[n] : 0.0f);
        }
    }
}

References C, CK_FP16_TO_FP32, block_q4_1::d, block_q4_1::m, QK4_1, and block_q4_1::qs.

Referenced by ck_gemm_nt_quant().

gemm_q4_1()

void gemm_q4_1	(	float *	Y,
		const void *	W,
		const float *	X,
		int	M,
		int	N,
		int	K
	)

Matrix-matrix multiply with Q4_1 weights.

Definition at line 158 of file gemm_kernels_q4_1.c.

{
    for (int n = 0; n < N; n++) {
        gemv_q4_1(&Y[n * M], W, &X[n * K], M, K);
    }
}

References gemv_q4_1().

gemm_q4_1_backward()

void gemm_q4_1_backward	(	float *	dX,
		const void *	W,
		const float *	dY,
		int	M,
		int	N,
		int	K
	)

Batched backward pass.

Definition at line 231 of file gemm_kernels_q4_1.c.

{
    for (int n = 0; n < N; n++) {
        gemv_q4_1_backward(&dX[n * K], W, &dY[n * M], M, K);
    }
}

References gemv_q4_1_backward().

gemv_q4_1()

void gemv_q4_1	(	float *	y,
		const void *	W,
		const float *	x,
		int	M,
		int	K
	)

Auto-dispatch GEMV.

Definition at line 139 of file gemm_kernels_q4_1.c.

{
#ifdef __AVX512F__
    gemv_q4_1_avx512(y, W, x, M, K);
#else
    gemv_q4_1_ref(y, W, x, M, K);
#endif
}

References gemv_q4_1_ref().

Referenced by dot_q4_1(), and gemm_q4_1().

gemv_q4_1_backward()

void gemv_q4_1_backward	(	float *	dX,
		const void *	W,
		const float *	dY,
		int	M,
		int	K
	)

Auto-dispatch backward.

Definition at line 220 of file gemm_kernels_q4_1.c.

{
    gemv_q4_1_backward_ref(dX, W, dY, M, K);
}

References gemv_q4_1_backward_ref().

Referenced by gemm_q4_1_backward().

gemv_q4_1_backward_ref()

void gemv_q4_1_backward_ref	(	float *	dX,
		const void *	W,
		const float *	dY,
		int	M,
		int	K
	)

Backward pass: compute input gradient.

Parameters

dX	Output gradient w.r.t. input [K]
W	Weight matrix in Q4_1 format [M x K]
dY	Gradient w.r.t. output [M]
M	Number of output rows
K	Number of columns (input dimension)

Definition at line 181 of file gemm_kernels_q4_1.c.

{
    const block_q4_1 *blocks = (const block_q4_1 *)W;
    const int blocks_per_row = K / QK4_1;
 
    /* Zero output gradient */
    memset(dX, 0, K * sizeof(float));
 
    /* Accumulate: dX += W^T @ dY */
    for (int row = 0; row < M; row++) {
        const float dy = dY[row];
 
        for (int b = 0; b < blocks_per_row; b++) {
            const block_q4_1 *block = &blocks[row * blocks_per_row + b];
            const float d = CK_FP16_TO_FP32(block->d);
            const float m = CK_FP16_TO_FP32(block->m);
            float *dxp = &dX[b * QK4_1];
 
            for (int i = 0; i < QK4_1 / 2; i++) {
                const uint8_t packed = block->qs[i];
                const int q0 = (packed & 0x0F);
                const int q1 = (packed >> 4);
 
                const float w0 = d * (float)q0 + m;
                const float w1 = d * (float)q1 + m;
 
                dxp[2*i + 0] += w0 * dy;
                dxp[2*i + 1] += w1 * dy;
            }
        }
    }
}

References CK_FP16_TO_FP32, block_q4_1::d, block_q4_1::m, QK4_1, and block_q4_1::qs.

Referenced by gemv_q4_1_backward().

gemv_q4_1_ref()

void gemv_q4_1_ref	(	float *	y,
		const void *	W,
		const float *	x,
		int	M,
		int	K
	)

Matrix-vector multiply with Q4_1 weights (scalar reference)

Parameters

y	Output vector [M]
W	Weight matrix in Q4_1 format [M x K]
x	Input vector [K]
M	Number of output rows
K	Number of columns (must be multiple of 32)

Definition at line 50 of file gemm_kernels_q4_1.c.

{
    const block_q4_1 *blocks = (const block_q4_1 *)W;
    const int blocks_per_row = K / QK4_1;
 
    for (int row = 0; row < M; row++) {
        float sum = 0.0f;
 
        for (int b = 0; b < blocks_per_row; b++) {
            const block_q4_1 *block = &blocks[row * blocks_per_row + b];
            const float d = CK_FP16_TO_FP32(block->d);
            const float m = CK_FP16_TO_FP32(block->m);
            const float *xp = &x[b * QK4_1];
 
            for (int i = 0; i < QK4_1 / 2; i++) {
                const uint8_t packed = block->qs[i];
                const int q0 = (packed & 0x0F);
                const int q1 = (packed >> 4);
 
                /* Dequantize: w = d * q + m */
                const float w0 = d * (float)q0 + m;
                const float w1 = d * (float)q1 + m;
 
                sum += w0 * xp[2*i + 0];
                sum += w1 * xp[2*i + 1];
            }
        }
 
        y[row] = sum;
    }
}

References CK_FP16_TO_FP32, block_q4_1::d, block_q4_1::m, QK4_1, and block_q4_1::qs.

Referenced by gemv_q4_1().