Dynamic feature (Element model)

The dynamic feature enables heterogeneous dynamic tensors. Enable it in Cargo.toml:

matten = { version = "0.46.0", features = ["dynamic"] }

matten is not a dataframe library. The dynamic feature is for ingesting and cleaning messy PoC data before converting to numeric tensors or handing off to a specialised crate.

The lifecycle is deliberately staged:

messy JSON / CSV / Elements
        |
        v
dynamic Tensor<Element>
        |
        | inspect: schema_summary, count_none, none_mask, numeric_mask
        v
dynamic Tensor<Element> with known issues
        |
        | clean: fill_none, forward_fill_none
        v
dynamic Tensor<Element> ready for policy
        |
        | convert: try_numeric / try_numeric_with
        v
numeric Tensor<f64>
        |
        v
ordinary matten computation

The important boundary is the conversion step. Arithmetic, slicing, reshape, and reductions belong after try_numeric(), not before it.

Element variants

use matten::Element;

Element::Float(1.5)            // IEEE 754 f64
Element::Int(42)               // i64
Element::text("active")        // UTF-8 text (Arc<str> internally)
Element::Bool(true)            // boolean
Element::None                  // missing / null

size_of::<Element>() == 24 bytes on 64-bit targets (all text representations give the same size; Arc<str> was chosen for cheap clone in CoW slices).

Constructing dynamic tensors

use matten::{Element, Tensor};

let t = Tensor::from_elements(
    vec![
        Element::Float(1.0), Element::text("ok"), Element::Bool(true),
        Element::Int(2),     Element::None,        Element::Bool(false),
    ],
    &[2, 3],
);

// Boundary-safe variant:
let t = Tensor::try_from_elements(data, &[2, 3])?;

Element predicates and coercion

#![allow(unused)]
fn main() {
use matten::Element;
Element::None.is_none();         // true
Element::Float(1.0).is_numeric(); // true
Element::Int(42).is_numeric();   // true
Element::Bool(true).is_numeric(); // false — no silent bool coercion

Element::Float(1.5).try_as_f64();  // Some(1.5)
Element::Int(7).try_as_f64();      // Some(7.0)
Element::text("3").try_as_f64();   // None — no silent text coercion
Element::None.try_as_f64();        // None
assert!(Element::None.is_none());
assert!(Element::Float(1.0).is_numeric());
assert!(Element::Int(42).is_numeric());
assert!(!Element::Bool(true).is_numeric());
assert_eq!(Element::Float(1.5).try_as_f64(), Some(1.5));
assert_eq!(Element::Int(7).try_as_f64(), Some(7.0));
assert_eq!(Element::text("3").try_as_f64(), None);
assert_eq!(Element::None.try_as_f64(), None);
}

Coercion policy (RFC-011 §11)

Use fill_none or explicit conversion helpers to clean data before arithmetic.

Accessing elements

t.get_element(&[0, 1])  // Option<Element> — None if out of bounds
t.is_dynamic()          // true for dynamic tensors
t.to_elements()         // Vec<Element> in row-major order

Mutable element access (RFC-104)

get_element_mut mirrors get_element, returning Option<&mut Element> instead of a copy. The caller reads, changes, or replaces the variant — the library never chooses one, so writing a 42.0 into what was an Int column raises no coercion question; it simply becomes a Float because that is what was written.

#![allow(unused)]
fn main() {
#[cfg(feature = "dynamic")] {
use matten::{Element, Tensor};

let mut t = Tensor::from_elements(vec![Element::Int(1), Element::Int(2)], &[2]);
*t.get_element_mut(&[1]).unwrap() = Element::text("two");
assert_eq!(t.get_element(&[1]), Some(Element::text("two")));
assert_eq!(t.get_element_mut(&[9]), None); // out of bounds
}
}

If this tensor’s storage is shared — for example it is a slice — the first write materializes it: a fresh, uniquely owned copy is made and the tensor detaches from whatever it was sharing with, so the write can never reach a shared parent. This is a no-op when the storage is already contiguous and unique — a second write on an already-detached tensor does not reallocate.

Worth knowing: materializing a slice releases the source’s allocation it was otherwise keeping alive for as long as the slice lived — the retention cost documented in Slicing gets an incidental escape hatch here, as a side effect of an unrelated operation (mutation), not a feature built for that purpose.

Missing-value utilities

use matten::{Element, Tensor};

let t = Tensor::from_elements(
    vec![Element::Float(1.0), Element::None, Element::Float(3.0), Element::None],
    &[4],
);

// Count None values
t.count_none()          // 2

// Boolean-like mask: 1.0 where None, 0.0 elsewhere (numeric f64 tensor)
let mask = t.none_mask();   // [0.0, 1.0, 0.0, 1.0]
// RFC-011 named alias:
let mask = t.is_none_mask(); // identical result

// Constant fill
let filled = t.fill_none(Element::Float(0.0)); // [1.0, 0.0, 3.0, 0.0]

// Forward-fill: carry last non-None value forward (fallback for leading None)
let t2 = Tensor::from_elements(
    vec![Element::None, Element::Float(1.0), Element::None, Element::Float(4.0)],
    &[4],
);
let fwd = t2.forward_fill_none(Element::Float(-1.0));
// [-1.0, 1.0, 1.0, 4.0]  (leading None takes fallback)

// Sum skipping None (panics on non-numeric non-None elements)
t.sum_skip_none()  // 4.0  (1.0 + 3.0, None values skipped)

Masks make readiness visible without changing the data:

dynamic values:     [ Float(1.0), None, Text("x"), Int(4) ]

none_mask():        [    0.0,     1.0,     0.0,    0.0 ]
numeric_mask():     [    1.0,     0.0,     0.0,    1.0 ]

meaning:
  none_mask    = where missing values are
  numeric_mask = which values strict try_numeric() can accept

Parsing mixed data

#![allow(unused)]
fn main() {
// JSON: null→None, booleans→Bool, strings→Text, integers→Int, floats→Float
#[cfg(feature = "json")]
let t = Tensor::from_json_dynamic(r#"[[1, "active", true], [2, null, false]]"#)?;

// CSV: empty field→None, "true"/"false"→Bool, integers→Int, floats→Float, rest→Text
#[cfg(feature = "csv")]
let t = Tensor::from_csv_dynamic("1,active,true\n2,,false\n")?;
}

Slicing (RFC-102)

slice() and slice_str() work on dynamic tensors, returning a dynamic tensor. Slicing selects positions — it does not interpret Element values, so heterogeneity is irrelevant and Text/None/Bool survive a slice unchanged alongside Int/Float. Storage is shared, not copied (Arc::clone), the same copy-on-write model fill_none and the other element-producing methods above already use internally:

#![allow(unused)]
fn main() {
#[cfg(feature = "dynamic")] {
use matten::{Element, Tensor};

let t = Tensor::from_elements((0..6).map(Element::Int).collect(), &[2, 3]);
let row = t.slice().index(0).all().build().unwrap();
assert!(row.is_dynamic());
assert_eq!(row.get_element(&[1]), Some(Element::Int(1)));
}
}

See Slicing for the full contract.

Current limitations (guard model)

In the current release, many numeric operations reject dynamic tensors with a clear matten unsupported error message. You must convert to a numeric tensor first using try_numeric().

Guarded (will panic or return Err):

• reshape, flatten, transpose, swap_axes
• all arithmetic operators and reductions
• dot / matmul
• as_slice, to_vec, into_vec, get, get_flat
• Serialize / serde

The underlying Arc-based CoW storage (DynamicTensor) is implemented internally; slicing (above) is its first public use. reshape is not yet wired to it.

// Correct pattern: ingest → clean → convert → arithmetic
let raw = Tensor::from_csv_dynamic("1.0,2.0\n3.0,4.0\n")?;
let filled  = raw.fill_none(Element::Float(0.0));
let numeric: Tensor = filled.try_numeric()?; // convert to numeric
let result = &numeric * 2.0;                 // numeric arithmetic

Workflow pattern

#![allow(unused)]
fn main() {
use matten::{Element, Tensor};

fn process_messy_csv(input: &str) -> Result<Tensor, Box<dyn std::error::Error>> {
    // 1. Ingest as dynamic
    let raw = Tensor::from_csv_dynamic(input)?;

    // 2. Fill missing values
    let clean = raw.fill_none(Element::Float(0.0));

    // 3. Convert to numeric tensor for arithmetic
    let numeric = clean.try_numeric()?;

    // 4. Use numeric arithmetic, reductions, matmul...
    Ok(numeric)
}
}

For a dirty row, the same workflow looks like this:

input row:       1.0, "", "active", 4

dynamic parse:   Float(1.0)  None  Text("active")  Int(4)

inspect:         none_mask    -> [0, 1, 0, 0]
                 numeric_mask -> [1, 0, 0, 1]

clean:           fill_none(0.0)
                 Float(1.0)  Float(0.0)  Text("active")  Int(4)

convert:         strict try_numeric() still rejects Text("active")
                 allow_text_parse() only helps text that actually parses as f64

Limitations

• No dataframe joins, group-by, pivot, or query operations.
• No date/time dtype.
• No categorical dtype.
• No silent text-to-number or bool-to-number coercion.
• Batched matmul on dynamic tensors requires try_numeric first.
• For large datasets, consider specialised crates (polars, ndarray).