Core Concepts

Toro's API is designed for concise method chaining. Operations throw exceptions on failure, just like PyTorch. This page covers the design choices that shape the API.

Direct API

Toro operations throw exceptions on failure. Shape mismatches, dtype incompatibilities, and device errors surface as .NET exceptions. This enables concise method chaining without Result unwrapping:

let x = Tensor.randn ([ 4; 2 ], F32, Cpu)
let w = Tensor.randn ([ 2; 1 ], F32, Cpu)
let pred = x.matmul w
let shifted = pred + 1.0
let loss = shifted.meanAll ()
loss.backward ()

Records as Models

Toro models are plain F# records. There is no base class to inherit. Each record defines a forward member that composes its layers:

type Features = {
    Conv1: Conv2d; Bn1: BatchNorm; Pool: MaxPool2d
} with
    member this.forward(train, x) =
        x |> this.Conv1.forward |> this.Bn1.forwardT train |> _.relu() |> this.Pool.forward

type MyModel = { Features: Features; Fc: Linear } with
    member this.forward(train, x) =
        x |> this.Features.forward(train) |> _.flatten(1, -1) |> this.Fc.forward

Records nest naturally. Model.trainableVars walks the entire structure by reflection and collects every Tensor with RequiresGrad = true:

let vars = Model.trainableVars myModel
let opt = AdamW.createWithLr 0.001 vars

See Neural Networks for the full model composition guide.

Disabling Gradients

Use Toro.noGrad to disable gradient tracking during evaluation:

Toro.noGrad (fun () ->
    let pred = model.forward testX
    let loss = Loss.crossEntropy pred testY
    printfn "Test loss: %.4f" (loss.item ())
)

Toro.noGrad wraps a function call in a torch.no_grad() scope. This reduces memory use and speeds up inference.

Memory Management with scoped { }

TorchSharp tensors hold native memory that the .NET garbage collector cannot see. In training loops, intermediate tensors accumulate and can cause out-of-memory errors.

The scoped { } CE solves this. It wraps the block in a torch.NewDisposeScope(). All intermediate tensors created inside the block are disposed when the scope exits:

for batch in trainLoader do
    scoped {
        let x, y = batch
        let logits = model.forward x
        let loss = Loss.crossEntropy logits y
        loss.backward ()
        opt.step ()
    }
    // All intermediate tensors (x, logits, loss, ...) are freed here

Auto-keep

When a scoped { } block returns a value containing tensors, those tensors are automatically kept alive past the scope. This works for single tensors, tuples, records, discriminated unions (including Option), lists, and arrays:

// Single Tensor
let mask = scoped {
    // ... intermediate work ...
    Tensor(filled.masked_fill (mask, negInf))
}

// Tuple
let k, v = scoped { newK, newV }

// Option
let maybeT = scoped { Some tensor }

// Record
let out = scoped { { Attn = a; Weights = w } }

Tensor.keep

Tensor.keep moves a tensor out of the current dispose scope. Use it when a component (such as a cache) needs to retain a tensor past the caller's scope:

let cached = Tensor.keep newK  // survives the caller's dispose scope

scoped { } return values are auto-kept, so Tensor.keep is not needed for return values.

Guidelines

Use scoped { } in any hot loop that creates tensors: training batches, optimizer steps, and inference loops. Pre-existing tensors (model parameters, optimizer state) are not affected by the scope.

DType and Device

Every tensor carries a DType (element type) and a Device (storage location). DType is a discriminated union with cases such as F32 (default), F64, I64, and Bool. Device is either Cpu or Cuda n. See the DType and Device API references for the full case lists.