Keyboard shortcuts

Press or to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Data model and lifecycle

One type, two modes

Tensor {
    data:  Vec<f64>,
    shape: Vec<usize>,
    dynamic: Option<Box<DynamicTensor>>,   // #[cfg(feature = "dynamic")] only
}

There is one tensor type, with two modes. Every Tensor is at least a numeric Vec<f64> with a shape. With the dynamic feature enabled, it can additionally hold a DynamicTensor — heterogeneous, Element-typed storage — behind the same Tensor handle.

The dynamic field does not exist without the dynamic feature. On the default feature set, Tensor is exactly the numeric pair (data, shape); there is no hidden mode to reach. Everything below that mentions DynamicTensor, Element, or ViewKind applies only when dynamic is enabled.

The lifecycle

A value’s path from raw input to a computed result crosses exactly one gate:

StageWhat happensAPI
ingestRead CSV/JSON into heterogeneous storageTable (matten-data), or core’s from_csv_dynamic / from_json_dynamic
cleanFill or select values — still dynamic, still Element-typedfill_none, selection methods
convertThe single gate: numeric-only from here ontry_numeric() — fails on any Text/None element
computeArithmetic, reductions, matmul — numeric onlythe core Tensor API

try_numeric() is the one place a heterogeneous value either becomes a plain numeric Tensor or is rejected. Nothing upstream of it is numeric; nothing downstream of it is anything else.

Table (matten-data) is a separate type in a companion crate, not a Tensor variant and not part of core. It reaches core through NumericTable::to_tensor() -> Result<matten::Tensor, _>.

The types involved

DynamicTensor { storage: Arc<Vec<Element>>, shape: Vec<usize>, len: usize, view: ViewKind }
ViewKind       Contiguous { offset: usize } | Indexed(Vec<usize>)
Element        Float(f64) | Int(i64) | Text(Arc<str>) | Bool(bool) | None

Table (matten-data) { headers: Vec<String>, rows: Vec<Vec<CellValue>> }
CellValue      Text(String) | Float(f64) | Int(i64) | Bool(bool) | Missing

Table and CellValue are a companion-crate representation for tabular input, not part of core’s Tensor/Element model — to_tensor() is the only bridge between the two.

The storage state machine

Dynamic storage is copy-on-write (RFC-012): a tensor either owns its storage uniquely, or shares it with other tensors via Arc, and moves between the two.

stateDiagram-v2
    [*] --> ContiguousUnique
    ContiguousUnique --> IndexedShared: slice()
    IndexedShared --> ContiguousUnique: get_element_mut() (materialize)
    ContiguousUnique --> ContiguousUnique: get_element_mut() (no-op, already unique)

Two consequences of this follow directly, and matter independently:

  • A slice retains its source’s entire allocation for as long as the slice lives — even after the source tensor is dropped. A one-element slice of a 100,000-element tensor keeps all 100,000 elements in memory. See Slicing (RFC-102 §8.1).
  • Mutating a slice releases that allocation, as a side effect. The first write through get_element_mut() materializes a fresh, uniquely-owned copy and detaches from whatever the tensor was sharing — an incidental escape hatch from the retention cost above, arriving from an unrelated operation. See Dynamic feature (RFC-104 §6.1).