CKVocabEntry Struct Reference

#include <ck_tokenizer.h>

Data Fields
int32_t	id
struct CKVocabEntry *	next
char *	token
int	token_len

Detailed Description

Definition at line 55 of file ck_tokenizer.h.

Field Documentation

id

int32_t CKVocabEntry::id

Definition at line 58 of file ck_tokenizer.h.

Referenced by ck_tokenizer_add_token(), ck_tokenizer_load(), and ck_tokenizer_load_binary().

next

struct CKVocabEntry* CKVocabEntry::next

Definition at line 59 of file ck_tokenizer.h.

Referenced by ck_tokenizer_add_token(), ck_tokenizer_load(), ck_tokenizer_load_binary(), and ck_tokenizer_lookup().

token

char* CKVocabEntry::token

Definition at line 56 of file ck_tokenizer.h.

Referenced by ck_tokenizer_add_token(), ck_tokenizer_load(), and ck_tokenizer_load_binary().

token_len

int CKVocabEntry::token_len

Definition at line 57 of file ck_tokenizer.h.

Referenced by ck_tokenizer_add_token(), ck_tokenizer_load(), and ck_tokenizer_load_binary().

---

The documentation for this struct was generated from the following file:

• ck_tokenizer.h