Hi 👋🏼,

I get the following error when trying to install tsdownsample:

Collecting tsdownsample
  Downloading tsdownsample-0.1.0a1.tar.gz (5.5 kB)
  Installing build dependencies ...   Getting requirements to build wheel ...   Preparing metadata (pyproject.toml) ... Requirement already satisfied: pandas>=1.3 in /home/jonas/.cache/pypoetry/virtualenvs/semi-guided-speech-27YL4uf1-py3.8/lib/python3.8/site-packages (from tsdownsample) (1.4.0)
Requirement already satisfied: numpy>=1.21 in /home/jonas/.cache/pypoetry/virtualenvs/semi-guided-speech-27YL4uf1-py3.8/lib/python3.8/site-packages (from tsdownsample) (1.22.2)
Requirement already satisfied: python-dateutil>=2.8.1 in /home/jonas/.cache/pypoetry/virtualenvs/semi-guided-speech-27YL4uf1-py3.8/lib/python3.8/site-packages (from pandas>=1.3->tsdownsample) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /home/jonas/.cache/pypoetry/virtualenvs/semi-guided-speech-27YL4uf1-py3.8/lib/python3.8/site-packages (from pandas>=1.3->tsdownsample) (2021.3)
Requirement already satisfied: six>=1.5 in /home/jonas/.cache/pypoetry/virtualenvs/semi-guided-speech-27YL4uf1-py3.8/lib/python3.8/site-packages (from python-dateutil>=2.8.1->pandas>=1.3->tsdownsample) (1.16.0)
Building wheels for collected packages: tsdownsample
  Building wheel for tsdownsample (pyproject.toml) ... error: subprocess-exited-with-error
  
  × Building wheel for tsdownsample (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [27 lines of output]
      A setup.py file already exists. Using it.
      Traceback (most recent call last):
        File "/tmp/pip-install-ptqqurvn/tsdownsample_7026e70fb6cf4d72bd2b00ae792aa535/setup.py", line 28, in <module>
          from setup import *
        File "/tmp/pip-install-ptqqurvn/tsdownsample_7026e70fb6cf4d72bd2b00ae792aa535/setup.py", line 29, in <module>
          build(setup_kwargs)
      NameError: name 'build' is not defined
      Traceback (most recent call last):
        File "/home/jonas/.cache/pypoetry/virtualenvs/semi-guided-speech-27YL4uf1-py3.8/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 351, in <module>
          main()
        File "/home/jonas/.cache/pypoetry/virtualenvs/semi-guided-speech-27YL4uf1-py3.8/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 333, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/home/jonas/.cache/pypoetry/virtualenvs/semi-guided-speech-27YL4uf1-py3.8/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 249, in build_wheel
          return _build_backend().build_wheel(wheel_directory, config_settings,
        File "/tmp/pip-build-env-65vdhqp1/overlay/lib/python3.8/site-packages/poetry/core/masonry/api.py", line 56, in build_wheel
          return WheelBuilder.make_in(
        File "/tmp/pip-build-env-65vdhqp1/overlay/lib/python3.8/site-packages/poetry/core/masonry/builders/wheel.py", line 85, in make_in
          wb.build(target_dir=directory)
        File "/tmp/pip-build-env-65vdhqp1/overlay/lib/python3.8/site-packages/poetry/core/masonry/builders/wheel.py", line 118, in build
          self._build(zip_file)
        File "/tmp/pip-build-env-65vdhqp1/overlay/lib/python3.8/site-packages/poetry/core/masonry/builders/wheel.py", line 179, in _build
          self._run_build_command(setup)
        File "/tmp/pip-build-env-65vdhqp1/overlay/lib/python3.8/site-packages/poetry/core/masonry/builders/wheel.py", line 217, in _run_build_command
          subprocess.check_call(
        File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['/home/jonas/.cache/pypoetry/virtualenvs/semi-guided-speech-27YL4uf1-py3.8/bin/python', '/tmp/pip-install-ptqqurvn/tsdownsample_7026e70fb6cf4d72bd2b00ae792aa535/setup.py', 'build', '-b', '/tmp/pip-install-ptqqurvn/tsdownsample_7026e70fb6cf4d72bd2b00ae792aa535/build']' returned non-zero exit status 1.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for tsdownsample
Failed to build tsdownsample
ERROR: Could not build wheels for tsdownsample, which is required to install pyproject.toml-based projects

fyi, trying to install it via poetry gives the same error

poetry add tsdownsample
Using version ^0.1.0a1 for tsdownsample

Updating dependencies
Resolving dependencies... (5.6s)

Writing lock file

Package operations: 1 install, 2 updates, 0 removals

  • Updating typing-extensions (4.4.0 -> 4.0.1)
  • Updating pydantic (1.10.2 -> 1.8.2)
  • Installing tsdownsample (0.1.0a1): Failed

  CalledProcessError

  Command '['/home/jonas/.cache/pypoetry/virtualenvs/semi-guided-speech-27YL4uf1-py3.8/bin/python', '-m', 'pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/home/jonas/.cache/pypoetry/virtualenvs/semi-guided-speech-27YL4uf1-py3.8', '--no-deps', '/home/jonas/.cache/pypoetry/artifacts/f4/dc/c1/ed26154e04a327d7ece0b3074c32a5b05bf7057f4dd093eb77b29e0b13/tsdownsample-0.1.0a1.tar.gz']' returned non-zero exit status 1.

  at /usr/lib/python3.10/subprocess.py:526 in run
       522│             # We don't call process.wait() as .__exit__ does that for us.
       523│             raise
       524│         retcode = process.poll()
       525│         if check and retcode:
    →  526│             raise CalledProcessError(retcode, process.args,
       527│                                      output=stdout, stderr=stderr)
       528│     return CompletedProcess(process.args, retcode, stdout, stderr)
       529│ 
       530│ 

The following error occurred when trying to handle this error:


  EnvCommandError

  Command ['/home/jonas/.cache/pypoetry/virtualenvs/semi-guided-speech-27YL4uf1-py3.8/bin/python', '-m', 'pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/home/jonas/.cache/pypoetry/virtualenvs/semi-guided-speech-27YL4uf1-py3.8', '--no-deps', '/home/jonas/.cache/pypoetry/artifacts/f4/dc/c1/ed26154e04a327d7ece0b3074c32a5b05bf7057f4dd093eb77b29e0b13/tsdownsample-0.1.0a1.tar.gz'] errored with the following return code 1, and output: 
  Processing /home/jonas/.cache/pypoetry/artifacts/f4/dc/c1/ed26154e04a327d7ece0b3074c32a5b05bf7057f4dd093eb77b29e0b13/tsdownsample-0.1.0a1.tar.gz
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
    Preparing metadata (pyproject.toml): started
    Preparing metadata (pyproject.toml): finished with status 'done'
  Building wheels for collected packages: tsdownsample
    Building wheel for tsdownsample (pyproject.toml): started
    Building wheel for tsdownsample (pyproject.toml): finished with status 'error'
    error: subprocess-exited-with-error
    
    × Building wheel for tsdownsample (pyproject.toml) did not run successfully.
    │ exit code: 1
    ╰─> [27 lines of output]
        A setup.py file already exists. Using it.
        Traceback (most recent call last):
          File "/tmp/pip-req-build-ubi789br/setup.py", line 28, in <module>
            from setup import *
          File "/tmp/pip-req-build-ubi789br/setup.py", line 29, in <module>
            build(setup_kwargs)
        NameError: name 'build' is not defined
        Traceback (most recent call last):
          File "/home/jonas/.cache/pypoetry/virtualenvs/semi-guided-speech-27YL4uf1-py3.8/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 351, in <module>
            main()
          File "/home/jonas/.cache/pypoetry/virtualenvs/semi-guided-speech-27YL4uf1-py3.8/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 333, in main
            json_out['return_val'] = hook(**hook_input['kwargs'])
          File "/home/jonas/.cache/pypoetry/virtualenvs/semi-guided-speech-27YL4uf1-py3.8/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 249, in build_wheel
            return _build_backend().build_wheel(wheel_directory, config_settings,
          File "/tmp/pip-build-env-4xa2n354/overlay/lib/python3.8/site-packages/poetry/core/masonry/api.py", line 56, in build_wheel
            return WheelBuilder.make_in(
          File "/tmp/pip-build-env-4xa2n354/overlay/lib/python3.8/site-packages/poetry/core/masonry/builders/wheel.py", line 85, in make_in
            wb.build(target_dir=directory)
          File "/tmp/pip-build-env-4xa2n354/overlay/lib/python3.8/site-packages/poetry/core/masonry/builders/wheel.py", line 118, in build
            self._build(zip_file)
          File "/tmp/pip-build-env-4xa2n354/overlay/lib/python3.8/site-packages/poetry/core/masonry/builders/wheel.py", line 179, in _build
            self._run_build_command(setup)
          File "/tmp/pip-build-env-4xa2n354/overlay/lib/python3.8/site-packages/poetry/core/masonry/builders/wheel.py", line 217, in _run_build_command
            subprocess.check_call(
          File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
            raise CalledProcessError(retcode, cmd)
        subprocess.CalledProcessError: Command '['/home/jonas/.cache/pypoetry/virtualenvs/semi-guided-speech-27YL4uf1-py3.8/bin/python', '/tmp/pip-req-build-ubi789br/setup.py', 'build', '-b', '/tmp/pip-req-build-ubi789br/build']' returned non-zero exit status 1.
        [end of output]
    
    note: This error originates from a subprocess, and is likely not a problem with pip.
    ERROR: Failed building wheel for tsdownsample
  Failed to build tsdownsample
  ERROR: Could not build wheels for tsdownsample, which is required to install pyproject.toml-based projects
  

  at /usr/lib/python3.10/site-packages/poetry/utils/env.py:1476 in _run
      1472│                 output = subprocess.check_output(
      1473│                     command, stderr=subprocess.STDOUT, env=env, **kwargs
      1474│                 )
      1475│         except CalledProcessError as e:
    → 1476│             raise EnvCommandError(e, input=input_)
      1477│ 
      1478│         return decode(output)
      1479│ 
      1480│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:

The following error occurred when trying to handle this error:


  PoetryException

  Failed to install /home/jonas/.cache/pypoetry/artifacts/f4/dc/c1/ed26154e04a327d7ece0b3074c32a5b05bf7057f4dd093eb77b29e0b13/tsdownsample-0.1.0a1.tar.gz

  at /usr/lib/python3.10/site-packages/poetry/utils/pip.py:51 in pip_install
       47│ 
       48│     try:
       49│         return environment.run_pip(*args)
       50│     except EnvCommandError as e:
    →  51│         raise PoetryException(f"Failed to install {path.as_posix()}") from e
       52│ 

In this PR we transfer from pandas.Series as representation for time series to a purely np.ndarray based representation (pandas.Series its index can at this time not represent most numeric dtypes, e.g. int16)

The signature for the downsample methdod is changed to:

downsample([x], y, n_out, **kwargs)

Note the following:

• x is optional, allowing the same (memory) efficient runtime as with pd.RangeIndex
• x and y are both positional arguments (this is in line with how the matplotlib,pyplot.plot signature is)
• n_out is a mandatory keyword argument

Example:

from tsdownsample import MinMaxLTTBDownsampler
import numpy as np

# Create a time series
y = np.random.randn(10_000_000)

# Downsample to 1000 points
s_ds = MinMaxLTTBDownsampler().downsample(y, n_out=1000)

Other aspects of this PR

• [x] support x and y datetime64: view the array as np.int64
• [x] support x and y timedelta64: view the array as np.int64
• [x] support y bool: view the array as np.int8 (under the hood numpy represents a bool as 8 bits)
• [x] support y int8 & uint8
• [ ] ~~support categorical data~~ => out of scope for this PR, would mostly be some Python code converting it to an encoding (which is perhaps not the responsibility of this library...) - also not really sure how numpy would represent categorical data

Major TODOs:

• documentation :ledger: => will leave this for another PR
• do not assume fixed sampling rate => will leave this for another PR

Tested the pip install tsdownsample on an Apple silicon cpu (M2).

Instead of installing the library (by using the downloaded binary), the laptop tried to build (:cry:) the library itself..

Normal installation :arrow_down: (here we see no build, just installation)

This PR alleviates the assumption that when x has fixed sampling rate (& no gaps) for certain downsample algorithms. Should fix #10.

This PR has the following contributions:

• [x] add sequential & parallel iterator that generates equidistant x-bin idxs (aka searchsorted) in utils.rs
  • [x] use f64 for val_step as this avoids rounding errors with integer / usize x-data types !! => seems to be significantly slower for minmaxlttb (10 - 20 %)
• [x] minmax algo with x
• [x] m4 algo with x
• [x] minmax lttb with x?
• [x] :snake: update Python bindings (check n_out requirements etc.)
• [x] :warning: update Python warnings + check new assertions
• [x] :racehorse: update benchmarks
• [x] :detective: search & fix out of bounds index issues in generic code of the downsample algorithms

Algorithm changes:

• M4:
  • n_out should be multiple of 4
  • removed the code which added the last datapoint to the output => when arr.len % n_out = 0 & equally sampled (+ no gaps) => difference betweeen with x and without x implementation
• MinMax:
  • n_out should be multiple of 2
  • removed the code which added the first and last datapoint to the output => when arr.len % n_out = 0 & equally sampled (+ no gaps) => difference betweeen with x and without x implementation

Python binding changes:

• Removed f16 as supported x datatype (IMO, no 16-bit dtypes make sense as x datatype... as these at max (u16) can only represent up to 65_535 values)
• M4 requires n_out to be multiple of 4
• MinMax requires n_out to be multiple of 2

The current code assumes equally sampled x-data (bin size is expressed in number of samples - assuming there are no gaps & sampling rate does not change)

To remove this assumption we should use something like np.searchsorted to determine the start- and end-indexes of the bins (this uses the x-data values to determine the bins).

I guess for empty bins, we should also perform some cleanup?

Fixes #2 (I hope :pray:)

• [x] :robot: switch to maturin for build & release
• [x] :bug: (minor) bug fixes
• [x] :dizzy: head/tail recursion in build macro to build for all x & y dtype combinations
• [x] :heavy_check_mark: extend Python tests to test the rust build
• [x] :sparkles: add minmax_ratio to minmaxlttb bindings
• [x] :arrow_up: upgrade argminmax dependency -> now also supporting SIMD on aarch64 :rocket:

:vomiting_face: this PR hacks support for minmax_ratio in the current interface (requiring to disable some type checks from mypy.. - #5 should further address this!)

Fixes #2 (I Hope :pray:)

• [x] :bug: bug fix in switch_mod code for rust downsamplers
• [x] :detective: add first (naive) tests
• [x] :building_construction: build
• [x] :rocket: release
• [x] :pen: update readme

Should open the path to the first proper PyPi release :fire:

• [x] add tsdownsample_rs (aka the rust code) as subpackage (under tsdownsample._rust.
• [x] :bug: bug fix in area calculation of LTTB_without_x
• [x] :snake: first version of the Python bidings
• [x] :pen: lint + checking the Python & Rust code

TODO -> will create new PR for this:

• [ ] testing of Python package
• [ ] deploy script

I have been playing around with your implementation of lttb and trying to compare it to other implementations (numpy and numba) and have been impressed by the speed.

But I have noticed that the algorithm does not give the same results as your other implementation in plotly_resampler. It seems to fail around 30,000 points. Do you know if this is expected?

Looking at the above log-log plot, some extra overhead is added, around 30,000. So I'm thinking it could be because of some race condition when parallelizing the algorithm.

import numpy as np

from plotly_resampler.aggregation.algorithms.lttb_py import LTTB_core_py
from tsdownsample import MinMaxLTTBDownsampler

np.random.seed(1)

ns = [i for i in np.arange(1, 51) * 1000]
n_out = 1000
for n in ns:
    x, y = np.arange(n), np.random.randn(n).cumsum()
    py = LTTB_core_py.downsample(x, y, n_out)
    ru = MinMaxLTTBDownsampler().downsample(x, y, n_out=n_out)

    print(f"Matches with {n:>5}: {(py == ru).all()}")

• which interface to implement, decision statements of current architecture
• what to look out for
• links to existing integrations (plotly-resampler in near future 🙏🏼 )

:exclamation: :rotating_light: The current Python interface is still under development!!

Some factors that should be taken into account;

• decide whether we go for classes or methods
• flexible datatype support (pd.Series seems not ideal, cfr. the bug we uncovered @jonasvdd)
• ...

Some other TODOs;

• [x] add minmax_ratio to Python bindings of minmaxlttb algorithm (:heavy_check_mark: #4)

(continuation of issue #2 )