CKBPEConfig Struct Reference

#include <true_bpe.h>

Data Fields
bool	add_bos
bool	add_eos
bool	byte_fallback
CKBPEPretokenizer	pretokenizer
CKSpacePrefixStyle	space_prefix_style

Detailed Description

Definition at line 59 of file true_bpe.h.

Field Documentation

add_bos

bool CKBPEConfig::add_bos

Definition at line 60 of file true_bpe.h.

add_eos

bool CKBPEConfig::add_eos

Definition at line 61 of file true_bpe.h.

byte_fallback

bool CKBPEConfig::byte_fallback

Definition at line 62 of file true_bpe.h.

pretokenizer

CKBPEPretokenizer CKBPEConfig::pretokenizer

Definition at line 64 of file true_bpe.h.

space_prefix_style

CKSpacePrefixStyle CKBPEConfig::space_prefix_style

Definition at line 63 of file true_bpe.h.

---

The documentation for this struct was generated from the following file:

• true_bpe.h