A Decimal Implementation written in pure Rust suitable for financial calculations.

Paul Mason

Last update: Jan 5, 2023

Related tags

Overview

Decimal

A Decimal implementation written in pure Rust suitable for financial calculations that require significant integral and fractional digits with no round-off errors.

The binary representation consists of a 96 bit integer number, a scaling factor used to specify the decimal fraction and a 1 bit sign. Because of this representation, trailing zeros are preserved and may be exposed when in string form. These can be truncated using the normalize or round_dp functions.

Getting started

To get started, add rust_decimal and optionally rust_decimal_macros to your Cargo.toml:

[dependencies]
rust_decimal = "1.18"
rust_decimal_macros = "1.18"

Usage

Decimal numbers can be created in a few distinct ways. The easiest and most optimal method of creating a Decimal is to use the procedural macro within the rust_decimal_macros crate:

// Procedural macros need importing directly
use rust_decimal_macros::dec;

let number = dec!(-1.23);
assert_eq!("-1.23", number.to_string());

Alternatively you can also use one of the Decimal number convenience functions:

// Using the prelude can help importing trait based functions (e.g. core::str::FromStr).
use rust_decimal::prelude::*;

// Using an integer followed by the decimal points
let scaled = Decimal::new(202, 2);
assert_eq!("2.02", scaled.to_string());

// From a string representation
let from_string = Decimal::from_str("2.02").unwrap();
assert_eq!("2.02", from_string.to_string());

// From a string representation in a different base
let from_string_base16 = Decimal::from_str_radix("ffff", 16).unwrap();
assert_eq!("65535", from_string_base16.to_string());

// Using the `Into` trait
let my_int: Decimal = 3i32.into();
assert_eq!("3", my_int.to_string());

// Using the raw decimal representation
let pi = Decimal::from_parts(1102470952, 185874565, 1703060790, false, 28);
assert_eq!("3.1415926535897932384626433832", pi.to_string());

Once you have instantiated your Decimal number you can perform calculations with it just like any other number:

use rust_decimal::prelude::*;

let amount = Decimal::from_str("25.12").unwrap();
let tax = Decimal::from_str("0.085").unwrap();
let total = amount + (amount * tax).round_dp(2);
assert_eq!(total.to_string(), "27.26");

`c-repr`

Forces Decimal to use [repr(C)]. The corresponding target layout is 128 bit aligned.

`db-postgres`

This feature enables a PostgreSQL communication module. It allows for reading and writing the Decimal type by transparently serializing/deserializing into the NUMERIC data type within PostgreSQL.

`db-tokio-postgres`

Enables the tokio postgres module allowing for async communication with PostgreSQL.

`db-diesel-postgres`

Enable diesel PostgreSQL support.

`db-diesel-mysql`

Enable diesel MySQL support.

`legacy-ops`

As of 1.10 the algorithms used to perform basic operations have changed which has benefits of significant speed improvements. To maintain backwards compatibility this can be opted out of by enabling the legacy-ops feature.

`maths`

The maths feature enables additional complex mathematical functions such as pow, ln, enf, exp etc. Documentation detailing the additional functions can be found on the MathematicalOps trait.

Please note that ln and log10 will panic on invalid input with checked_ln and checked_log10 the preferred functions to curb against this. When the maths feature was first developed the library would return 0 on invalid input. To re-enable this non-panicking behavior, please use the feature: maths-nopanic.

`rocket-traits`

Enable support for Rocket forms by implementing the FromFormField trait.

`rust-fuzz`

Enable rust-fuzz support by implementing the Arbitrary trait.

`serde-float`

Enable this so that JSON serialization of Decimal types are sent as a float instead of a string (default).

e.g. with this turned on, JSON serialization would output:

{
  "value": 1.234
}

`serde-str`

This is typically useful for bincode or csv like implementations.

Since bincode does not specify type information, we need to ensure that a type hint is provided in order to correctly be able to deserialize. Enabling this feature on its own will force deserialization to use deserialize_str instead of deserialize_any.

If, for some reason, you also have serde-float enabled then this will use deserialize_f64 as a type hint. Because converting to f64 loses precision, it's highly recommended that you do NOT enable this feature when working with bincode. That being said, this will only use 8 bytes so is slightly more efficient in terms of storage size.

`serde-arbitrary-precision`

This is used primarily with serde_json and consequently adds it as a "weak dependency". This supports the arbitrary_precision feature inside serde_json when parsing decimals.

This is recommended when parsing "float" looking data as it will prevent data loss.

`std`

Enable std library support. This is enabled by default, however in the future will be opt in. For now, to support no_std libraries, this crate can be compiled with --no-default-features.

Building

Please refer to the Build document for more information on building and testing Rust Decimal.

Minimum Rust Compiler Version

The current minimum compiler version is 1.51.0 which was released on 2021-03-25 and included support for "const generics".

Updating the minimum supported version

This library maintains support for rust compiler versions that are 5 minor versions away from the current stable rust compiler version. For example, if the current stable compiler version is 1.50.0 then we will guarantee support up to and including 1.45.0. Of note, we will only update the minimum supported version if and when required.

Comments

Support the `pow` method for `Decimal`/floating point arguments

Ion would like to remove its dependency to C, and the next step would be to remove the decimal crate. Unfortunately, not being able to raise some Decimal to a floating point power is a serious barrier to our objective.

opened by AdminXVII 34
Feature request: Conversion from float retaining excess precision
Currently this code prints 0.1:

let d = Decimal::from_f64(0.1).unwrap(); println!("{}", d);

At first that sounds like correct behavior and what one would want. But I would argue that it is in fact not the correct result, and that it should instead output 0.10000000000000000555111512313.

As we know, the token 0.1 is read by Rust into an f64 number, where it is represented by a floating-point number equal to the fraction 3602879701896397/36028797018963968. (This ratio can be obtained in Python using 0.1 .as_integer_ratio(), or in Rust by printing BigRational::from_float(0.1).) While the resulting number can be displayed as 0.1 without loss of precision when again re-read as float, and while humans often think of it as 0.1, it is the fraction that is in fact used in floating-point calculations.

This is why I would expect "0.1".parse::<Decimal>().unwrap() to differ from Decimal::from_f64(0.1).unwrap(). The former reads from decimal and is precisely 0.1, whereas the latter converts the result of the lossy conversion of the 0.1 token to floating point. The input of Decimal::from_f64(0.1) is the number corresponding to the above-mentioned fraction, which has an exact decimal representation (because its denominator is a power of 2): 0.1000000000000000055511151231257827021181583404541015625. Given the constraint of a mantissa smaller than 2**96, the closest rust_decimal::Decimal representation of this is 10000000000000000555111512313 * 10**-29, so I'd expect the Decimal to display as 0.10000000000000000555111512313.

The same applies to from_f32 where Decimal has enough precision to hold all the digits of 13421773/134217728, and should display 0.100000001490116119384765625.

Python's decimal does the equivalent conversion, except its decimal is not limited to u128, so it can accommodate the whole float:

>>> import decimal >>> decimal.Decimal(0.1) Decimal('0.1000000000000000055511151231257827021181583404541015625') >>> decimal.Decimal("0.1") Decimal('0.1')

Looking at the implementation, it's not at all obvious that the current behavior is intended. The intention seems to be to convert the number with maximum possible precision. While I'm aware that some people might consider the current behavior to be a feature, I'd still ask to change it, or at least to provide a different method for converting floats to decimals while minimizing the difference from the original number.
opened by hniksic 18
Faster serde
I had a use case where my bottleneck was due to decimal deserialization via bincode, so I though I would do some low hanging fruit optimizations.

Feedback would be much appreciated.

Overview

Adds arrrayvec as a dependency since it will replace the heap allocated vectors and strings

Removes heap allocation for both serialization & deserialization with the serde-str feature enable. This is done my removing heap allocation for the from_str impl as well as using a to_array_str impl instead of using to_string.

Adds a from_str_radix specialization when radix is 10, since its the most common case (~25% performance improvement)

Improve the to_string algorithm by making it 2 pass instead of 3 pass

Outstanding issues

Need to add tests for the to_array_str method

Im not sure the constants I used for my arrays are the right ones. Basically we wanna use the highest possible size necessary otherwise it will panic when pushing a sufficiently long decimal.

Performance improvements according to the micro benchmarks:

before: test deserialize_bincode ... bench: 6,544 ns/iter (+/- 3,308) test serialize_bincode ... bench: 15,397 ns/iter (+/- 7,748) test decimal_to_string ... bench: 7,288 ns/iter (+/- 77) test decimal_from_str ... bench: 5,684 ns/iter (+/- 180) after: test deserialize_bincode ... bench: 2,884 ns/iter (+/- 1,448) test serialize_bincode ... bench: 5,248 ns/iter (+/- 57) test decimal_to_string ... bench: 4,203 ns/iter (+/- 48) test decimal_from_str ... bench: 1,592 ns/iter (+/- 810)

I also tested against my real life application and the performance improvements where similar. I got from about 500k reads/s to 1.3M reads/s. I did not test the serialization path however.
opened by jean-airoldie 14
feat: Add `rng.gen_range` support.
It appears that the previous work done in #517 only added partial rand support. In particular, it added support for rng.gen() but not rng.gen_range(). The latter lets you provide a low and high value and get a random number in between them:

let d = rng.gen_range(dec!(1.00)..=dec!(3.00)); assert!(d <= dec!(3.00)); assert!(d >= dec!(1.00));

This PR adds such support. It approaches the problem by taking the low and high values, rescaling them to match (with the higher of the two scales) if necessary, and then farming out the actual randomness by shipping off the mantissa values to rand::distributions::uniform::UniformInt<i128>. The UniformInt struct then returns the random mantissa, which is re-decimalized and returned.

Scale is relevant here, the example above is distinct from rng.gen_range(dec!(1.0)..=dec!(3.0), because the original example has 201 valid values (1.00, 1.01, 1.02, etc.) whereas this later one has only three potential values (1.0, 2.0, 3.0).
opened by lukesneeringer 11
Add fuzz targets to exercize checked functions
Useful to catch user inputs that could abort thread execution like #376.

Adds limited CI execution through a script that can also be used for local development.

Constify some methods

Creates a new try_new method for convenience.

Adds initial constructors, add_and_sub and div_and_mul targets (more yet to come).

CI is purposely failing because things like:

let a = X; let b = Y; let c = a / b; assert_eq!(c * b, a);

Are currently being used, which brings rounding errors like:

left: `127.99999999986666197612373336` right: `128`

If it is a normal behaviour (probably is), then the above code snippet can be replaced by:

let a = X; let b = Y; let _c = a / b; let _d = a * b;
opened by c410-f3r 11

Error with PostgreSQL and high scale

I'm experiencing a hard crash reading a decimal value from a postgres table using a high-scale numeric type. The error happens with rust_decimal version 1.2.0, version 1.1.0 works fine. We can replicate the issue in all major postgres versions from 9 to 12.

My schema looks like this:

postgres=# \d "Test";
                Table "sql_load_test.Test"
 Column |      Type      | Collation | Nullable | Default
--------+----------------+-----------+----------+---------
 fl     | numeric(65,30) |           | not null | 0
 id     | integer        |           | not null |
 fl2    | numeric(65,15) |           |          |
Indexes:
    "Test_pkey" PRIMARY KEY, btree (id)

And we can reproduce this with a simple test program:

Cargo.toml:

[dependencies]
rust_decimal = { version = "1.2", features = ["postgres"] }
tokio-postgres = { version = "0.5", features = ["with-uuid-0_8", "with-chrono-0_4", "with-serde_json-1"] }
tokio = { version = "0.2", features = ["rt-core", "macros"] }

main.rs:

use rust_decimal::Decimal;
use tokio_postgres::{Error, NoTls};

#[tokio::main]
async fn main() -> Result<(), Error> {
    let (client, connection) = tokio_postgres::connect(
        "postgresql://postgres:prisma@localhost:5432/postgres",
        NoTls,
    )
    .await?;

    tokio::spawn(async move {
        if let Err(e) = connection.await {
            eprintln!("connection error: {}", e);
        }
    });

    let rows = client
        .query("SELECT fl2 from sql_load_test.\"Test\"", &[])
        .await?;

    let value: Decimal = rows[0].get(0);
    println!("WORKS: {}", value);

    let rows = client
        .query("SELECT fl from sql_load_test.\"Test\"", &[])
        .await?;

    let value: Decimal = rows[0].get(0);
    println!("CRASHES: {}", value);

    Ok(())
}

The program output is here:

env RUST_BACKTRACE=1 cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.04s
     Running `target/debug/decimal-test`
WORKS: 1.200000000000000
thread 'main' panicked at 'attempt to multiply with overflow', /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libcore/num/mod.rs:1983:27
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/libunwind.rs:88
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.40/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print_fmt
             at src/libstd/sys_common/backtrace.rs:84
   3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
             at src/libstd/sys_common/backtrace.rs:61
   4: core::fmt::write
             at src/libcore/fmt/mod.rs:1025
   5: std::io::Write::write_fmt
             at src/libstd/io/mod.rs:1426
   6: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:65
   7: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:50
   8: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:193
   9: std::panicking::default_hook
             at src/libstd/panicking.rs:210
  10: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:471
  11: rust_begin_unwind
             at src/libstd/panicking.rs:375
  12: core::panicking::panic_fmt
             at src/libcore/panicking.rs:84
  13: core::panicking::panic
             at src/libcore/panicking.rs:51
  14: core::num::<impl i64>::pow
             at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libcore/num/mod.rs:1983
  15: rust_decimal::postgres::<impl rust_decimal::decimal::Decimal>::from_postgres
             at /home/pimeys/.cargo/registry/src/github.com-1ecc6299db9ec823/rust_decimal-1.2.0/src/postgres.rs:99
  16: rust_decimal::postgres::postgres::<impl postgres_types::FromSql for rust_decimal::decimal::Decimal>::from_sql
             at /home/pimeys/.cargo/registry/src/github.com-1ecc6299db9ec823/rust_decimal-1.2.0/src/postgres.rs:507
  17: postgres_types::FromSql::from_sql_nullable
             at /home/pimeys/.cargo/registry/src/github.com-1ecc6299db9ec823/postgres-types-0.1.0/src/lib.rs:433
  18: tokio_postgres::row::Row::get_inner
             at /home/pimeys/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-postgres-0.5.1/src/row.rs:174
  19: tokio_postgres::row::Row::get
             at /home/pimeys/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-postgres-0.5.1/src/row.rs:140
  20: decimal_test::main::{{closure}}
             at src/main.rs:29
  21: <std::future::GenFuture<T> as core::future::future::Future>::poll
             at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libstd/future.rs:43
  22: tokio::runtime::basic_scheduler::BasicScheduler<P>::block_on
             at /home/pimeys/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.11/src/runtime/basic_scheduler.rs:138
  23: tokio::runtime::Runtime::block_on::{{closure}}
             at /home/pimeys/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.11/src/runtime/mod.rs:411
  24: tokio::runtime::context::enter
             at /home/pimeys/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.11/src/runtime/context.rs:72
  25: tokio::runtime::handle::Handle::enter
             at /home/pimeys/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.11/src/runtime/handle.rs:34
  26: tokio::runtime::Runtime::block_on
             at /home/pimeys/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-0.2.11/src/runtime/mod.rs:408
  27: decimal_test::main
             at src/main.rs:4
  28: std::rt::lang_start::{{closure}}
             at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libstd/rt.rs:67
  29: std::rt::lang_start_internal::{{closure}}
             at src/libstd/rt.rs:52
  30: std::panicking::try::do_call
             at src/libstd/panicking.rs:292
  31: __rust_maybe_catch_panic
             at src/libpanic_unwind/lib.rs:78
  32: std::panicking::try
             at src/libstd/panicking.rs:270
  33: std::panic::catch_unwind
             at src/libstd/panic.rs:394
  34: std::rt::lang_start_internal
             at src/libstd/rt.rs:51
  35: std::rt::lang_start
             at /rustc/5e1a799842ba6ed4a57e91f7ab9435947482f7d8/src/libstd/rt.rs:67
  36: main
  37: __libc_start_main
  38: _start
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

To save looking into the large backtrace, the bug line is this and happens in the pow function call (overflow).

opened by pimeys 11

Decimal::to_f64 produces imprecise results
I'm having some problems with your implementation of to_f64,

for example:

assert_eq!(Decimal::new(22238, 4).to_f64(), Decimal::new(22238, 4).to_string().parse::<f64>().ok());

fails, your implementation returns 2.2237999999999998 while std returns correctly 2.2238

Is there a way to make your implementation more precise?
opened by nappa85 9
Optimize `parse_str_radix_10`
Replacing #454, this PR resolves #453 and:

Added some tests

Fixed two edge cases

Simplified the logic of parse_str_radix_10

Optimized the performance with 64/128 bit integers

Speed up further with tail call optimization

Running cargo bench --bench lib_benches -- decimal_from_str

On M1 Pro, the numbers are:

master: test decimal_from_str ... bench: 566 ns/iter (+/- 21) 7f18e0d: test decimal_from_str ... bench: 88 ns/iter (+/- 21)

On Ryzen 3990x, the numbers are:

master: test decimal_from_str ... bench: 557 ns/iter (+/- 13) 432f9d6: decimal_from_str ... bench: 129 ns/iter (+/- 2)

I know this is a lot of code changes. No rush at all and feel free to ask us questions if pieces of the code seems unclear.

Have a happy new year too!
opened by chris-cantor 8
Decimal::deserialize can panic

The PR https://github.com/paupino/rust-decimal/pull/433 introduced a panic in the deserialize method: https://github.com/paupino/rust-decimal/blob/c74bd40701cce99b264af6b9af2e66d0f66251f1/src/decimal.rs#L710

This means that one cannot receive raw serialized decimals from a untrusted 3rd party. I'm also assuming that casting raw bytes to Decimal, even with the repr(C) freature enabled is UB.

Do you think it would be possible to make casting bytes to a Decimal type not UB? My specific use case is for binary serialization where serde is too slow.

opened by jean-airoldie 8
implemented an optimized to_f64()
This is a follow up of the issue here: https://github.com/paupino/rust-decimal/issues/234

I ended up implementing an alternative (IMHO simpler) idea. Hopefully the code is readable enough.

I thought about re-constructing the f64 from scratch using the Decimal's internal parts, but quickly ran into some questions that I'm not sure how to address. The gist is this:

We know a Decimal value is sign * m / 10^e, where m is a 96-bit integer, and e is at most 28, while an f64 value is sign * m / 2^e. This means we need to convert a base 10 exponent into a base 2 exponent.

So, we can rewrite the Decimal value sign * m / 10^e into sign * (m / 5^e) / 2^e. Let m' = m / 5^e. This m' will be the mantissa for the f64 value. The problem is m itself is not necessarily divisible by 5, the best we can do is approximate it with the integer that's closest to m / 5^e. So there's some loss of precision.

So, IMO, since we are losing precision here anyways, maybe it's worth it to simply use f64 arithmetic to calculate the fractional part of the Decimal value directly, and add that to the integral part of the Decimal value. Hence, the alternative algorithm below.

This implementation breaks one of the test cases in it_converts_to_f64 with the following output:

---- it_converts_to_f64 stdout ---- thread 'it_converts_to_f64' panicked at 'assertion failed: `(left == right)` left: `0.0000000000025`, right: `0.0000000000025000000000000007`', tests/decimal_tests.rs:1065:5

The test case is originally doing exact floating number comparison between the LHS and RHS. I revised it to use approximate comparison through the float-cmp crate. This shows the LHS and RHS are less than 2 ULPS apart, which means they are less than 2 units of precision apart at their current exponent.

This may or may not be acceptable for you... I'm happy to consider other routes that can potentially avoid modifying this existing test case.
opened by hengchu 8
Diesel support

Postgres conversion code is refactored to be library-independent with Diesel and rust-postgres support bolted on top of it. Tests are taken from Diesel's bigdecimal support, but I suspect that more exhaustive testing is required.

Should only be merged after #187.

opened by vorot93 8

Confusing f64 → Decimal

Example:

let f = 1652185258.8058286;
let a: Decimal = f.to_string().parse().unwrap();
let b = Decimal::try_from(f).unwrap();
let c = Decimal::from_f64_retain(f).unwrap();
println!("try_from(f64 -> str): {}", a);
println!("try_from(f64):        {}", b);
println!("from_f64_retain(f64): {}", c);

try_from(f64 -> str): 1652185258.8058286
try_from(f64):        1652185258.805829
from_f64_retain(f64): 1652185258.8058285713195800781

Can a direct conversion be improved? Or, at least, it should be noted in the documentation about such problems. Maybe the crate should provide a special feature to have more expensive (but at least without allocation and with ryu) conversion by default.

opened by loyd 2

Add parity scale codec
PR adds support of parity-scale-codec for rust-decimal.

References

https://docs.substrate.io/reference/scale-codec/

https://github.com/paritytech/parity-scale-codec
opened by Gauthamastro 0
[v2] Replace error type with ones more like std

A start towards #49, though I haven't audited panicking, just migrated the existing errors.

Notably, because the error type no longer holds String, the crate no longer has a hard requirement on alloc.

opened by CAD97 1
Decimals parse `.` in the rounding position incorrectly
https://github.com/paupino/rust-decimal/blob/4b7176177cfd3938cecb59d9358667a8a2d4ab6d/src/str.rs#L358-L364

[src\main.rs:109] Decimal::from_str("1.0000000000000000000000000000.5") = Ok( 1.0000000000000000000000000000, )

The condition here is inverted for the decimal point; it should be allowed if !point. Currently maybe_round is only called with point=true anyway.

Fixing this will reveal another case of #542 if maybe_round is ever called with point=false.
opened by CAD97 0
Decimals parsed with `_` in the rounding position round incorrectly
[src\main.rs:109] Decimal::from_str("1.0000000000000000000000000000_6") = Ok( 1.0000000000000000000000000000, )

https://github.com/paupino/rust-decimal/blob/4b7176177cfd3938cecb59d9358667a8a2d4ab6d/src/str.rs#L358-L364

This is actually somewhat interesting to fix, as we need to scan forward to the first non-_ character, and maybe_round doesn't have that information available currently.
opened by CAD97 0