matten-data — table to Tensor

matten-data is a small, production-ready (RFC-085) companion crate for the boring step between a small table-like input (such as a CSV) and a numeric [matten::Tensor]. It is a conversion helper, not a dataframe library or query engine.

For joins, group-by, lazy queries, or datetime handling, use Polars, DataFusion, or Pandas. matten-data deliberately does none of those. Row-count-bounded batched CSV reading is available behind the optional streaming feature (RFC-082) — see below.

Install

[dependencies]
matten = "0.46.0"
matten-data = "0.46.0"

Both crates share one lock-step family version (RFC-030); maturity is a per-crate Status label, not a separate version number.

Quickstart

use matten::Tensor;
use matten_data::Table;

fn main() -> Result<(), matten_data::MattenDataError> {
let csv = "region,sales,cost\nnorth,100,40\nsouth,150,\neast,120,55";

let tensor: Tensor = Table::from_csv_str(csv)?
    .select_columns(["sales", "cost"])? // choose columns by name, in this order
    .fill_missing(0.0)?                  // the missing south/cost becomes 0.0
    .try_numeric()?                      // strict, explicit conversion to f64
    .to_tensor()?;                       // a normal [rows, columns] Tensor

assert_eq!(tensor.shape(), &[3, 2]);
Ok(())
}

The data path is intentionally explicit:

CSV text
  |
  v
Table
  headers: region, sales, cost
  rows:    3
  |
  | select_columns(["sales", "cost"])
  v
Table
  headers: sales, cost
  rows:    3
  |
  | fill_missing(0.0)
  v
Table
  missing cost cell is now an explicit numeric value
  |
  | try_numeric()
  v
NumericTable
  all selected cells are f64-compatible
  |
  | to_tensor()
  v
Tensor shape [3, 2]
  rows    = CSV data rows
  columns = selected columns, in requested order

The example suite

The numbered tutorial suite teaches one step at a time; csv_to_tensor is a single comprehensive overview.

cargo run -p matten-data --example data_00_quickstart
cargo run -p matten-data --example data_06_visual_readiness_summary

Output Tensor shape

to_tensor produces a tensor of shape [rows, columns], where rows are the data rows (the header is not a row) and columns are the selected columns in the order you requested them. The data is row-major: row 0’s values come first, then row 1’s, and so on. Once converted, the result is an ordinary matten::Tensor — every core operation applies.

For the quickstart input, selecting sales and cost gives:

source rows

row 0: region=north
row 1: region=south
row 2: region=east

selected columns: sales, cost

selected table

row 0: sales=100  cost=40
row 1: sales=150  cost=0      (filled explicitly)
row 2: sales=120  cost=55

Tensor shape [3, 2]

[ 100   40 ]
[ 150    0 ]
[ 120   55 ]

flat row-major data:

[100, 40, 150, 0, 120, 55]

Missing-value policy

Missing cells are never silently turned into 0. A missing value that reaches numeric conversion is a precise MissingValue { column, row } error (the row is the 1-based CSV line number). You decide what a missing value means by calling fill_missing with an explicit value before converting.

The policy is visible in the workflow:

missing cell present
        |
        | try_numeric()
        v
MissingValue error

missing cell present
        |
        | fill_missing(value)
        v
explicit value present
        |
        | try_numeric()
        v
NumericTable

Numeric conversion policy

Conversion is strict and explicit (try_numeric then to_tensor): integers and floats become f64; booleans and non-numeric text are rejected (they are never coerced to 1/0); and a remaining missing cell is rejected. This keeps the boundary between “table-like text” and “numbers” honest and visible.

Limitations

matten-data has no joins, group-by, pivot, query DSL, lazy execution, indexing/loc/iloc, rolling/window operations, datetime engine, or categorical dtype system. It is for small, application-validated or trusted data. When you need those capabilities, reach for a dataframe/query engine (Polars, DataFusion, Pandas) instead.

Streaming (optional, streaming feature)

CsvBatchReader (RFC-082) reads a CSV file in row-count-bounded Table batches, off by default behind the streaming feature — the csv feature is implied, no new dependency. This is a memory strategy, not a dataframe engine: no schema evolution, no lenient/skip-malformed mode, no streaming numeric conversion, and no async. A batch is exactly a Table; concatenating every batch reproduces Table::from_csv_path’s output for ordinary input. Two malformed-input edge cases deliberately diverge from Table::from_csv_path (a file that is blank but not empty — one that trims to nothing but contains at least one whitespace character other than a line terminator, such as a space or tab — and invalid UTF-8) — see CsvBatchReader’s own documentation for the exact behavior. matten-data’s production-ready promotion (RFC-085) covers this feature too: stable in what it does, but its scope may still grow (RFC-082 §5 defers nine further items, including async and resumability).

[dependencies]
matten-data = { version = "0.46.0", features = ["streaming"] }

use matten_data::CsvBatchReader;

fn main() -> Result<(), matten_data::MattenDataError> {
let path = std::path::Path::new("large.csv");
let mut reader = CsvBatchReader::open(path, 10_000)?;
while let Some(batch) = reader.next_batch()? {
    // process one Table batch at a time
}
Ok(())
}

Status and maturity

Production-ready (0.46.x family, RFC-085). The table-to-Tensor API is mostly stable but pre-1.0; pin the release explicitly. The crate’s scope is locked and enforced in CI (RFC-042), and core matten never depends on it (RFC-022).