Text Processing

Toro.Text bridges Microsoft.ML.Tokenizers and Toro tensors. It provides an F#-idiomatic Tokenizer wrapper and an Encode module for converting text to tensors.

open Toro
open Toro.Text

Tokenizer

Tokenizer wraps a Microsoft.ML.Tokenizers.Tokenizer instance with a simple API:

let tokenizer = Tokenizer.fromTiktoken (TiktokenConfig.create "gpt-4o")

let ids = tokenizer.encode "hello world"       // int list
let text = tokenizer.decode ids                 // "hello world"
let count = tokenizer.countTokens "hello world" // int

Tokenizer Types

Each tokenizer type has a dedicated config record. Use create with required parameters, then override optional fields with record update syntax. See the Tokenizer API reference for config details.

Tiktoken (OpenAI)

// Simple — just the model name
let tok = Tokenizer.fromTiktoken (TiktokenConfig.create "gpt-4o")

// With extra special tokens
let tok = Tokenizer.fromTiktoken
            { TiktokenConfig.create "gpt-4o" with
                ExtraSpecialTokens = [ "<|pad|>", 100300 ] }

WordPiece (BERT)

let tok = Tokenizer.fromWordPiece
            { WordPieceConfig.create "vocab.txt" with
                SpecialTokens = [ "[CLS]", 101; "[SEP]", 102; "[PAD]", 0 ]
                PreTokenizer = Regex @"\w+|[^\w\s]+"
                Normalizer = LowerCase }

BPE

let tok = Tokenizer.fromBpe (BpeConfig.create "vocab.json" "merges.txt")

SentencePiece

let tok = Tokenizer.fromSentencePiece (SentencePieceConfig.create "model.spm")

Normalizer and PreTokenizer

Common configurations are covered by Toro-native DU types:

type TextNormalizer =
    | NoNormalizer
    | LowerCase
    | CustomNormalizer of Microsoft.ML.Tokenizers.Normalizer

type TextPreTokenizer =
    | DefaultPreTokenizer
    | Regex of pattern: string
    | CustomPreTokenizer of Microsoft.ML.Tokenizers.PreTokenizer

For advanced cases, use the Custom* constructors as escape hatches.

Escape Hatch

Access the underlying Microsoft.ML.Tokenizers.Tokenizer via .Inner, or use Tokenizer.wrap to wrap a manually-constructed instance:

let inner: Microsoft.ML.Tokenizers.Tokenizer = tokenizer.Inner
let tok = Tokenizer.wrap someCustomInstance

Encode

The Encode module converts tokenized text directly to Toro tensors.

toTensor

Encode a single text to a 1-D int64 tensor of fixed length:

let tok = Tokenizer.fromTiktoken (TiktokenConfig.create "gpt-4o")
let t = Encode.toTensor tok "hello world" 16 0 Cpu
// t: [16] int64, padded with 0s

batch

Encode multiple texts to a padded [B, L] tensor and an attention mask:

let tok = Tokenizer.fromTiktoken (TiktokenConfig.create "gpt-4o")
let texts = [ "hello world"; "good morning" ]
let struct (ids, mask) = Encode.batch tok texts 16 0 Cpu
// ids:  [2, 16] int64
// mask: [2, 16] — 1 at token positions, 0 at padding

attentionMask

Generate a mask from an existing token tensor:

let tokens = Tensor.ofArray ([| 101L; 7592L; 0L; 0L |], Cpu)
let mask = Encode.attentionMask tokens 0
// mask: [4] = [1, 1, 0, 0]