Encode
Text-to-tensor encoding utilities.
| Function | Description |
|---|---|
attentionMask tokens padId | Generate an attention mask: 1 where token differs from padId, 0 at pad positions. |
batch tokenizer texts maxLen padId device | Encode a batch of texts to a padded [B, L] tensor and an attention mask. |
toTensor tokenizer text maxLen padId device | Encode a single text to a 1-D token ID tensor of length maxLen. |
attentionMask
attentionMask tokens padIdGenerate an attention mask: 1 where token differs from padId, 0 at pad positions.
Parameters
tokens:TensorpadId:int
Returns Tensor
batch
batch tokenizer texts maxLen padId deviceEncode a batch of texts to a padded [B, L] tensor and an attention mask.
Parameters
tokenizer:Tokenizertexts:string listmaxLen:intpadId:intdevice:Device
Returns Tensor * Tensor
toTensor
toTensor tokenizer text maxLen padId deviceEncode a single text to a 1-D token ID tensor of length maxLen.
Parameters
tokenizer:Tokenizertext:stringmaxLen:intpadId:intdevice:Device
Returns Tensor