Statistics (core-lite)

Core matten provides exactly four statistics reductions (RFC-040), alongside the mean/mean_axis already in Reductions and matrix multiplication:

var / std — population variance / std over all elements.
var_axis / std_axis — the same along one axis.

Anything with significant statistical policy — quantile, percentile, histogram, covariance, correlation, z-score, sample variance — is out of core scope (a possible future matten-stats companion). matten is a family-car PoC library, not a statistics package.

Population variance, not sample variance

All four use population variance (ddof = 0):

mean = sum(x) / n
var  = sum((x - mean)^2) / n
std  = sqrt(var)

There is no sample-variance (ddof = 1) variant, no var_with_ddof, and no nanvar/nanstd in core. A single-element tensor has variance 0.0. A two-pass algorithm is used (mean first, then squared deviations) to avoid the avoidable cancellation of the naive one-pass E[x^2] - E[x]^2.

NaN propagates: any NaN element yields NaN (per-slice for the axis variants), consistent with the other f64 reductions. Use try_numeric() to convert a dynamic tensor first; the statistics methods reject dynamic tensors.

var / std

Tensor::var(&self) -> f64
Tensor::std(&self) -> f64
Tensor::try_var(&self) -> Result<f64, MattenError>
Tensor::try_std(&self) -> Result<f64, MattenError>

[1, 2, 3, 4]  ->  mean 2.5,  var 1.25,  std sqrt(1.25) ≈ 1.118

The try_* forms return MattenError::Unsupported on a dynamic tensor. They also guard the empty-tensor case with MattenError::InvalidArgument (RFC-105) — reachable via slicing to a zero-sized shape and, since RFC-111, directly from any constructor as well.

var_axis / std_axis

Tensor::var_axis(&self, axis: usize) -> Tensor
Tensor::std_axis(&self, axis: usize) -> Tensor
Tensor::try_var_axis(&self, axis: usize) -> Result<Tensor, MattenError>
Tensor::try_std_axis(&self, axis: usize) -> Result<Tensor, MattenError>

The reduced axis is removed from the output shape (no keepdims), matching the existing axis reductions (mean_axis, sum_axis):

[[1, 2, 3], [4, 5, 6]]  var_axis(0)  ->  [2.25, 2.25, 2.25]   // shape [3], per column
[[1, 2, 3], [4, 5, 6]]  var_axis(1)  ->  [2/3, 2/3]            // shape [2], per row

The shape behavior is the same for var_axis and std_axis; std_axis just takes the square root of each variance:

input shape [2, 3]
axes         0  1

var_axis(0): collapse rows, keep columns

[ 1  2  3 ]
[ 4  5  6 ]  ->  [ var([1,4])  var([2,5])  var([3,6]) ]
             ->  [    2.25        2.25        2.25    ]   shape [3]

var_axis(1): collapse columns, keep rows

[ 1  2  3 ]
[ 4  5  6 ]  ->  [ var([1,2,3])  var([4,5,6]) ]
             ->  [     2/3            2/3     ]           shape [2]

The try_* forms return MattenError::Shape if axis >= rank, or MattenError::Unsupported on a dynamic tensor.

Empty reduced axis (RFC-110): var_axis/try_var_axis and std_axis/try_std_axis additionally return MattenError::InvalidArgument (or, for the panicking forms, a panic carrying that message) when the reduced axis has length 0 — variance and standard deviation of nothing are undefined, not NaN. A zero-length axis that survives the reduction (a different axis than the one being reduced) is unaffected and still returns Ok with an empty result.

Out of scope for core

sample variance (ddof = 1)    quantile        percentile
histogram                     covariance      correlation
z-score                       nanvar/nanstd   statistical tests

These stay out of core; RFC-040 §9’s gate for a future companion was cleared once at least three clearly-useful, well-scoped APIs were accepted, and that companion — matten-stats — now exists (RFC-078, expanded by RFC-083; see below). Some (z-score) overlap with matten-mlprep and must not be duplicated there.

The `matten-stats` companion (RFC-078, RFC-083, RFC-084, RFC-090)

matten-stats is a separate, production-ready candidate companion crate (promoted in RFC-084 once its surface settled) computing statistical summaries that core deliberately excludes: covariance, covariance_population, correlation, quantile, skewness, kurtosis, and histogram. A summary is returned as f64 where it is scalar, and as a small owned struct (Histogram) where it is inherently vector-valued — matten-stats never returns a Tensor (RFC-090 §5 amended the original Tensor -> f64 framing to make room for histogram’s non-scalar result). Its estimator conventions differ per function, matching what each function’s name is expected to mean in the wider ecosystem (NumPy/SciPy/R/pandas):

covariance             sample,     ddof = 1        (NumPy/R/pandas `cov`/`corrcoef` default)
covariance_population  population, ddof = 0        (explicit in the name; no default to choose)
correlation            ddof-invariant                 (the n - 1 factors cancel)
skewness               g1,  uncorrected                (SciPy `skew(bias=True)` default; NOT pandas' `.skew()`)
kurtosis               g2,  uncorrected, EXCESS         (SciPy `kurtosis(fisher=True, bias=True)` default; NOT pandas' `.kurt()`)

kurtosis reports excess kurtosis (Fisher’s definition): a normal distribution scores 0.0, not 3.0. pandas’ .skew()/.kurt() bias-correct and so return a different number than skewness/kurtosis for the same input — this divergence is deliberate, not an oversight. See the crate’s own README for the full API and error model.

`histogram` — bin count is the caller’s choice (RFC-090)

pub struct Histogram {
    pub counts: Vec<usize>,
    pub edges: Vec<f64>,
}

pub fn histogram(x: &Tensor, bins: usize) -> Result<Histogram, MattenStatsError>;

There is no automatic bin-count rule — not Sturges, not Freedman-Diaconis, not Scott, no "auto". bins is required because bin count is a genuine analytical choice; a histogram whose bin count was picked for the caller teaches the wrong lesson (RFC-090 §4.1).

The range is always [min(x), max(x)]; there is no range parameter. The last bin is closed at the top ([edges[bins - 1], edges[bins]]), unlike every other bin ([edges[i], edges[i + 1]), half-open) — otherwise the maximum value would fall in no bin and silently vanish from the counts. A constant input errors (ZeroVariance) rather than widening the range the way NumPy does ((v - 0.5, v + 0.5)); that 0.5 comes from nowhere in the data.

Example

See 16_variance_std.rs for a runnable walkthrough of core’s var/std, and histogram.rs (cargo run -p matten-stats --example stats_histogram) for matten-stats’s histogram.

Keyboard shortcuts

matten