← Back to C-Kernel-Engine Docs Doxygen Source Documentation
CKTokenizerToken Struct Reference

#include <tokenizer.h>

Data Fields

int32_t id
 
bool is_special
 
float score
 

Detailed Description

Definition at line 88 of file tokenizer.h.

Field Documentation

◆ id

int32_t CKTokenizerToken::id

Definition at line 89 of file tokenizer.h.

◆ is_special

bool CKTokenizerToken::is_special

Definition at line 91 of file tokenizer.h.

◆ score

float CKTokenizerToken::score

Definition at line 90 of file tokenizer.h.


The documentation for this struct was generated from the following file: