gemm_kernels_q4_0.c File Reference

GEMM/GEMV kernels with Q4_0 quantized weights. More...

#include <stdint.h>
#include <stddef.h>
#include <string.h>
#include "ckernel_quant.h"

Go to the source code of this file.

Functions
float	dot_q4_0 (const void *w_q4_0, const float *x, int K)
void	gemm_nt_q4_0 (const float *A, const void *B, const float *bias, float *C, int M, int N, int K)
	Matrix-matrix multiply: C[M,N] = A[M,K] @ B[N,K]^T + bias.
void	gemm_q4_0 (float *Y, const void *W, const float *X, int M, int N, int K)
	Matrix-matrix multiply with Q4_0 weights.
void	gemm_q4_0_backward (float *dX, const void *W, const float *dY, int M, int N, int K)
	Batched backward pass.
void	gemv_q4_0 (float *y, const void *W, const float *x, int M, int K)
	Auto-dispatch GEMV.
void	gemv_q4_0_backward (float *dX, const void *W, const float *dY, int M, int K)
	Auto-dispatch backward.
void	gemv_q4_0_backward_ref (float *dX, const void *W, const float *dY, int M, int K)
	Backward pass: compute input gradient.
void	gemv_q4_0_ref (float *y, const void *W, const float *x, int M, int K)
	Matrix-vector multiply with Q4_0 weights (scalar reference)

Detailed Description

GEMM/GEMV kernels with Q4_0 quantized weights.

CK-ENGINE KERNEL RULES:

1. NO malloc/free - memory via bump allocator, pointers passed in
2. NO OpenMP - parallelization at orchestrator/codegen layer
3. API must define: inputs, outputs, workspace, and memory layouts
4. Pure computation - deterministic, no side effects

After changes: make test && make llamacpp-parity-full

Q4_0 Format:

• 32 weights per block
• 1 FP16 scale per block
• 18 bytes per 32 weights = 4.5 bits/weight

Operations: Forward: Y = W @ X (W is Q4_0, X and Y are FP32) Backward: dX = W^T @ dY (gradient w.r.t. input)

Note: Weight gradients are not computed for quantized weights. For fine-tuning, use LoRA adapters which maintain FP32 gradients separately.

Definition in file gemm_kernels_q4_0.c.

Function Documentation

dot_q4_0()

float dot_q4_0	(	const void *	w_q4_0,
		const float *	x,
		int	K
	)

Definition at line 347 of file gemm_kernels_q4_0.c.

{
    float result;
    gemv_q4_0(&result, w_q4_0, x, 1, K);
    return result;
}

References gemv_q4_0().

gemm_nt_q4_0()

void gemm_nt_q4_0	(	const float *	A,
		const void *	B,
		const float *	bias,
		float *	C,
		int	M,
		int	N,
		int	K
	)

Matrix-matrix multiply: C[M,N] = A[M,K] @ B[N,K]^T + bias.

Parameters

A	Input matrix [M x K], row-major FP32
B	Weight matrix in Q4_0 format, [N x K] stored row-major
bias	Optional bias [N], NULL if not used
C	Output [M x N], row-major FP32
M	Batch size (number of tokens)
N	Output dimension (number of rows in B)
K	Input dimension

Definition at line 176 of file gemm_kernels_q4_0.c.

{
    const block_q4_0 *blocks = (const block_q4_0 *)B;
    const int blocks_per_row = K / QK4_0;
 
    for (int m = 0; m < M; m++) {
        const float *a_row = &A[m * K];
 
        for (int n = 0; n < N; n++) {
            float sum = 0.0f;
 
            for (int b = 0; b < blocks_per_row; b++) {
                const block_q4_0 *block = &blocks[n * blocks_per_row + b];
                const float d = CK_FP16_TO_FP32(block->d);
                const float *ap = &a_row[b * QK4_0];
 
                for (int i = 0; i < QK4_0 / 2; i++) {
                    const uint8_t packed = block->qs[i];
                    const int q0 = (packed & 0x0F) - 8;
                    const int q1 = (packed >> 4) - 8;
 
                    sum += d * (float)q0 * ap[2 * i + 0];
                    sum += d * (float)q1 * ap[2 * i + 1];
                }
            }
 
            C[m * N + n] = sum + (bias ? bias[n] : 0.0f);
        }
    }
}

References C, CK_FP16_TO_FP32, block_q4_0::d, QK4_0, and block_q4_0::qs.

Referenced by ck_gemm_nt_quant().

gemm_q4_0()

void gemm_q4_0	(	float *	Y,
		const void *	W,
		const float *	X,
		int	M,
		int	N,
		int	K
	)

Matrix-matrix multiply with Q4_0 weights.

Definition at line 151 of file gemm_kernels_q4_0.c.

{
    for (int n = 0; n < N; n++) {
        gemv_q4_0(&Y[n * M], W, &X[n * K], M, K);
    }
}

References gemv_q4_0().

gemm_q4_0_backward()

void gemm_q4_0_backward	(	float *	dX,
		const void *	W,
		const float *	dY,
		int	M,
		int	N,
		int	K
	)

Batched backward pass.

Definition at line 333 of file gemm_kernels_q4_0.c.

{
    for (int n = 0; n < N; n++) {
        gemv_q4_0_backward(&dX[n * K], W, &dY[n * M], M, K);
    }
}

References gemv_q4_0_backward().

gemv_q4_0()

void gemv_q4_0	(	float *	y,
		const void *	W,
		const float *	x,
		int	M,
		int	K
	)

Auto-dispatch GEMV.

Definition at line 132 of file gemm_kernels_q4_0.c.

{
#ifdef __AVX512F__
    gemv_q4_0_avx512(y, W, x, M, K);
#else
    gemv_q4_0_ref(y, W, x, M, K);
#endif
}

References gemv_q4_0_ref().

Referenced by dot_q4_0(), and gemm_q4_0().

gemv_q4_0_backward()

void gemv_q4_0_backward	(	float *	dX,
		const void *	W,
		const float *	dY,
		int	M,
		int	K
	)

Auto-dispatch backward.

Definition at line 318 of file gemm_kernels_q4_0.c.

{
#ifdef __AVX512F__
    gemv_q4_0_backward_avx512(dX, W, dY, M, K);
#else
    gemv_q4_0_backward_ref(dX, W, dY, M, K);
#endif
}

References gemv_q4_0_backward_ref().

Referenced by gemm_q4_0_backward().

gemv_q4_0_backward_ref()

void gemv_q4_0_backward_ref	(	float *	dX,
		const void *	W,
		const float *	dY,
		int	M,
		int	K
	)

Backward pass: compute input gradient.

Parameters

dX	Output gradient w.r.t. input [K]
W	Weight matrix in Q4_0 format [M x K]
dY	Gradient w.r.t. output [M]
M	Number of output rows
K	Number of columns (input dimension)

Definition at line 230 of file gemm_kernels_q4_0.c.

{
    const block_q4_0 *blocks = (const block_q4_0 *)W;
    const int blocks_per_row = K / QK4_0;
 
    /* Zero output gradient */
    memset(dX, 0, K * sizeof(float));
 
    /* Accumulate: dX += W^T @ dY */
    for (int row = 0; row < M; row++) {
        const float dy = dY[row];
 
        for (int b = 0; b < blocks_per_row; b++) {
            const block_q4_0 *block = &blocks[row * blocks_per_row + b];
            const float d = CK_FP16_TO_FP32(block->d);
            float *dxp = &dX[b * QK4_0];
 
            for (int i = 0; i < QK4_0 / 2; i++) {
                const uint8_t packed = block->qs[i];
                const int8_t q0 = (packed & 0x0F) - 8;
                const int8_t q1 = (packed >> 4) - 8;
 
                dxp[2*i + 0] += d * (float)q0 * dy;
                dxp[2*i + 1] += d * (float)q1 * dy;
            }
        }
    }
}

References CK_FP16_TO_FP32, block_q4_0::d, QK4_0, and block_q4_0::qs.

Referenced by gemv_q4_0_backward().

gemv_q4_0_ref()

void gemv_q4_0_ref	(	float *	y,
		const void *	W,
		const float *	x,
		int	M,
		int	K
	)

Matrix-vector multiply with Q4_0 weights (scalar reference)

Parameters

y	Output vector [M]
W	Weight matrix in Q4_0 format [M x K]
x	Input vector [K]
M	Number of output rows
K	Number of columns (must be multiple of 32)

Definition at line 49 of file gemm_kernels_q4_0.c.

{
    const block_q4_0 *blocks = (const block_q4_0 *)W;
    const int blocks_per_row = K / QK4_0;
 
    for (int row = 0; row < M; row++) {
        float sum = 0.0f;
 
        for (int b = 0; b < blocks_per_row; b++) {
            const block_q4_0 *block = &blocks[row * blocks_per_row + b];
            const float d = CK_FP16_TO_FP32(block->d);
            const float *xp = &x[b * QK4_0];
 
            for (int i = 0; i < QK4_0 / 2; i++) {
                const uint8_t packed = block->qs[i];
                const int8_t q0 = (packed & 0x0F) - 8;
                const int8_t q1 = (packed >> 4) - 8;
 
                sum += d * (float)q0 * xp[2*i + 0];
                sum += d * (float)q1 * xp[2*i + 1];
            }
        }
 
        y[row] = sum;
    }
}

References CK_FP16_TO_FP32, block_q4_0::d, QK4_0, and block_q4_0::qs.

Referenced by gemv_q4_0().