Tokenizer
Microsoft.ML.Tokenizers adapters and factories.
| Function | Description |
|---|---|
fromBert config | Create a BERT tokenizer. Encoding adds [CLS]/[SEP] when AddSpecialTokens is true. |
fromBpe config | Create a BPE tokenizer. |
fromFunctions encode decode | Create a tokenizer from encode and decode functions. |
fromSentencePiece config | Create a SentencePiece tokenizer. |
fromTiktoken config | Create a Tiktoken tokenizer. |
fromWordPiece config | Create a WordPiece tokenizer. |
inner tokenizer | Return the wrapped Microsoft.ML.Tokenizers instance. |
Tokenizers created with fromFunctions have no backend; this call fails. | |
wrap inner | Wrap a Microsoft.ML.Tokenizers tokenizer. |
fromBert
fromBert configCreate a BERT tokenizer. Encoding adds [CLS]/[SEP] when AddSpecialTokens is true.
Parameters
config:BertConfig
Returns Tokenizer
fromBpe
fromBpe configCreate a BPE tokenizer.
Parameters
config:BpeConfig
Returns Tokenizer
fromFunctions
fromFunctions encode decodeCreate a tokenizer from encode and decode functions.
Parameters
encode:string -> int64 listdecode:int64 list -> string
Returns Tokenizer
fromSentencePiece
fromSentencePiece configCreate a SentencePiece tokenizer.
Parameters
config:SentencePieceConfig
Returns Tokenizer
fromTiktoken
fromTiktoken configCreate a Tiktoken tokenizer.
Parameters
config:TiktokenConfig
Returns Tokenizer
fromWordPiece
fromWordPiece configCreate a WordPiece tokenizer.
Parameters
config:WordPieceConfig
Returns Tokenizer
inner
inner tokenizerReturn the wrapped Microsoft.ML.Tokenizers instance.
Tokenizers created with fromFunctions have no backend; this call fails.
Parameters
tokenizer:Tokenizer
Returns Tokenizer
wrap
wrap innerWrap a Microsoft.ML.Tokenizers tokenizer.
Parameters
inner:Tokenizer
Returns Tokenizer