A perfect smoother; A discrete time version of spline smoothing for equally spaced data

Overview

Whittaker Smoother

Aka Whittaker-Henderson, Whittaker-Eilers Smoother is known as the perfect smoother. Its a discrete-time version of spline smoothing for equally spaced data. It minimizes the functional

$$\sum_{i=0}^n (z_i - y_i)^2 + \lambda \sum_{i=0}^n (\delta ^p z)_i ^2 $$

where y are the datapoints, z is the smoothed function, and $\delta^2 z$ is the pth derivative of $z_i$ , which is evaluated numerically. A penalty is imposed on nonsmooth functions, with higher values of $\lambda$ increasing the penalty and leading to a smoother output.

The smoothed output can be obtained by solving the linear system $$x = (W + \lambda * D^T D )^{-1} W y $$ Where W is the weight matrix (Identity matrix in practice) and D is the difference matrix (See difference_matrix for its construction).

Examples

Here we see the wood dataset smoothed whith both order 2 and 3. wood_2 wood_3

Comparison to Moving Averages and Convolution Kernels

Compared to a moving average smoother, this method does not suffer from a group-delay.

Compared to a convolution kernel such as the savitzky-golay filter, the values at the edge are well defined and don't need to be interpolated. The savitzky-golay filter does have a nice flat passband, but suffers from unsatisfactory high-frequency noise, which is not sufficiently suppressed. This is a particular problem when the derivative of the data is of importance.

Usage

To use this smoother in you project, add this to your Cargo.toml:

[dependencies]
whittaker_smoother = "0.1"

Now you can use the smoothing function as such:

use whittaker_smoother::whittaker_smoother;

// Here we use the WOOD_DATASET, but this can be any series that you would like to smooth
let raw = Vec::from_iter(WOOD_DATASET.iter().map(|v| *v as f64));
let lambda = 2e4;
let order = 3;
let smoothed = whittaker_smoother(&raw, lambda, order).unwrap();

And BAM, that's it! There is you perfectly smoothed series.

Further Reading:

See the papers folder for two papers showing additional details of the method.

This implementation was inspired by A python implementation.

Potential Upgrades:

  • Add benchmarks
  • Use sparse matrices if available
  • Add function for computing the optimal lambda based on cross-validation (See eilers2003)

License

Copyright (C) 2020 <Mathis Wellmann [email protected]>

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License along with this program. If not, see https://www.gnu.org/licenses/.

GNU AGPLv3

You might also like...
Fast and simple datetime, date, time and duration parsing for rust.

speedate Fast and simple datetime, date, time and duration parsing for rust. speedate is a lax† RFC 3339 date and time parser, in other words, it pars

Compile-time stuff and other goodies for rustaceans πŸ¦€

πŸ₯― bagel: Always baked, never fried bagel is a collection of macros and other things that we frequently use at Skytable, primarily to get work done at

Rust crate: Overloaded Literals to construct your datatypes without boilerplate and with compile-time validation.

overloaded_literals   Overloaded Literals to construct your datatypes without boilerplate and with compile-time validation. Features Compile-time vali

Parses a relative time string and returns a `Duration`

humantime_to_duration A Rust crate for parsing human-readable relative time strings and converting them to a Duration. Features Parses a variety of hu

Time to dive into Rust!

Lets-Learn-Rust Time to dive into Rust! Day 1 Installation Running a Simple Rust Program Managing Projects with Cargo Basic Programming - Comments and

πŸ¦€ A Rust CLI to find the optimal time to meet given a when2meet URL

when3meet πŸ¦€ The Rust when2meet CLI Install | Usage | Contributing & Issues | Docs Built with ❀️ and πŸ¦€ by Garrett Ladley Install cargo install when3m

Construct complex structures within single call + simple compile-time meta-inheritance model with mixins.

Introduction constructivism is a Rust sample-library designed to simplify the construction of structured data by defining and manipulating sequences o

Proof-of-concept for a memory-efficient data structure for zooming billion-event traces

Proof-of-concept for a gigabyte-scale trace viewer This repo includes: A memory-efficient representation for event traces An unusually simple and memo

Data structures and algorithms for 3D geometric modeling.

geom3d Data structures and algorithms for 3D geometric modeling. Features: Bezier curve and surface B-Spline curve and surface Spin surface Sweep surf

Releases(0.1)
  • 0.1(Aug 11, 2023)

    I'm proud to present the first Rust implementation of the Whittaker smoother, which is known as the perfect smoother due to its ability to fit a discrete time version of a spline, meaning the smoothed function has a smooth p-th order derivative as well. I'm genuinely suprised how little this beautiful algorithm is know given its awesome properties. There is not even a Wikipedia page for it, which I may have to change in the future. Anyways, hope this is useful to you!

    Source code(tar.gz)
    Source code(zip)
Owner
Mathis Wellmann
Protocol engineer building on substrate @gensyn-ai. Otherwise algo-trading cryptocurrency derivatives using ML techniques.
Mathis Wellmann
A real-time data backend for browser-based applications.

DriftDB DriftDB is a real-time data backend for browser-based applications. For more information, see driftdb.com. Structure of this repo docs/: main

drifting in space 453 Feb 6, 2023
An expression based data notation, aimed at transpiling itself to any cascaded data notation.

Lala An expression oriented data notation, aimed at transpiling itself to any cascaded data notation. Lala is separated into three components: Nana, L

null 37 Mar 9, 2022
A kernel version manager for systemd-boot and AOSC OS

Your systemd-boot's best friend ever (Implemented in Rust) A kernel version manager for systemd-boot and AOSC OS Usage First initialize friend and sys

AOSC-Dev 20 Oct 9, 2022
A rust version of "the super tiny compiler"

The (Rust) super tiny compiler This project is a rust version of the super tiny compiler (the original one (JS) was created by Jamie Kyle). The output

Eduardo Stuart 6 Dec 25, 2022
Modern Rust utility library delivering modularity, performance & extras; or simply Rust version of Lodash

Lorust - API Documentation Lorust is the Rust version of Lodash, which is a modern Javascript utilty library delivering modularity, performance & extr

Imamuzzaki Abu Salam 2 Jul 9, 2023
Safe, comp time generated queries in rust

query_builder For each struct field following methods will be generated. All fields where_FIELDNAME_eq Numeric fields where_FIELDNAME_le where_FIELDNA

Amirreza Askarpour 2 Oct 31, 2021
A real-time mixer

Pagana Pagana is a real-time mixer. This project is still in early stages of development and is not ready for any kind of production use or testing. D

null 1 Nov 26, 2021
Compile time static maps for Rust

Rust-PHF Documentation Rust-PHF is a library to generate efficient lookup tables at compile time using perfect hash functions. It currently uses the C

null 1.3k Jan 1, 2023
Isn't it time to be a bit nicer to rustc?

politeness-macro Aren't we all too rude to computers? Isn't it time to bring a bit more politeness into our programming? Shouldn't we be a bit nicer t

Rin 6 Mar 11, 2022
Simple time handling in Rust

time Documentation: latest release main branch book Minimum Rust version policy The time crate is guaranteed to compile with any release of rustc from

Time 680 Dec 31, 2022