Tokenizer
| Function | Description |
|---|---|
fromBpe config | Create from a BPE config. |
fromSentencePiece config | Create from a SentencePiece config. |
fromTiktoken config | Create from a Tiktoken config. |
fromWordPiece config | Create from a WordPiece config. |
wrap inner | Wrap an existing Microsoft.ML.Tokenizers.Tokenizer instance. |
fromBpe
fromBpe configCreate from a BPE config.
Parameters
config:BpeConfig
Returns Tokenizer
fromSentencePiece
fromSentencePiece configCreate from a SentencePiece config.
Parameters
config:SentencePieceConfig
Returns Tokenizer
fromTiktoken
fromTiktoken configCreate from a Tiktoken config.
Parameters
config:TiktokenConfig
Returns Tokenizer
fromWordPiece
fromWordPiece configCreate from a WordPiece config.
Parameters
config:WordPieceConfig
Returns Tokenizer
wrap
wrap innerWrap an existing Microsoft.ML.Tokenizers.Tokenizer instance.
Parameters
inner:Tokenizer
Returns Tokenizer