CKTokenizerToken Struct Reference

#include <tokenizer.h>

Data Fields
int32_t	id
bool	is_special
float	score

Detailed Description

Definition at line 88 of file tokenizer.h.

Field Documentation

id

int32_t CKTokenizerToken::id

Definition at line 89 of file tokenizer.h.

is_special

bool CKTokenizerToken::is_special

Definition at line 91 of file tokenizer.h.

score

float CKTokenizerToken::score

Definition at line 90 of file tokenizer.h.

---

The documentation for this struct was generated from the following file:

• tokenizer.h