Dataflow system for building self-driving car and robotics applications.

ERDOS

Last update: Dec 29, 2022

Related tags

Learning Resources rust robot drone robotics ros robot-framework dataflow self-driving-car autonomous-vehicles robot-programming

Overview

ERDOS

ERDOS is a platform for developing self-driving cars and robotics applications.

Getting started

The easiest way to get ERDOS running is to use our Docker image:

docker pull erdosproject/erdos

Local installation

System requirements

ERDOS is known to work on Ubuntu 16.04, 18.04, and 20.04.

Rust installation

To develop an ERDOS application in Rust, simply include ERDOS in Cargo.toml. The latest ERDOS release is published on Crates.io and documentation is available on Crates.io.

If you'd like to contribute to ERDOS, first install Rust. Then run the following to clone the repository and build ERDOS:

rustup default nightly  # use nightly Rust toolchain
git clone https://github.com/erdos-project/erdos.git && cd erdos
cargo build

Python Installation

To develop an ERDOS application in Python, simply run pip install erdos. Documentation is available on Read the Docs.

If you'd like to contribute to ERDOS, first install Rust. Then run the following to clone the repository and build ERDOS:

rustup default nightly  # use nightly Rust toolchain
git clone https://github.com/erdos-project/erdos.git && cd erdos
python3 python/setup.py develop

Python files are available under the python/ directory, and the Python-Rust bridge interface is developed under src/python/.

If you'd like to build ERDOS for release (has better performance, but longer build times), run python3 python/setup.py install.

Running an example

python3 python/examples/simple_pipeline.py

Build parameters

By default, ERDOS supports up to 20 read streams and 10 write streams in a stream bundle (used to add callbacks across multiple streams; see callback builder in Rust docs for more details). Some applications may require bundles that support more read and write streams. This can be configured by setting the ERDOS_BUNDLE_MAX_READ_STREAMS and ERDOS_BUNDLE_MAX_WRITE_STREAMS environment variables when building.

Building ERDOS for the first time may be slow because these parameters result in generated code which dominates the build time. Subsequent builds should be much faster due to caching unless build.rs or the code generation scripts are modified. In that case, consider setting ERDOS_BUNDLE_MAX_READ_STREAMS and ERDOS_BUNDLE_MAX_WRITE_STREAMS to speed up builds.

Writing Applications

ERDOS provides Python and Rust interfaces for developing applications.

The Python interface provides easy integration with popular libraries such as tensorflow, but comes at the cost of performance (e.g. slower serialization and the lack of multithreading).

The Rust interface provides more safety guarantees (e.g. compile-time type checking) and faster performance (e.g. multithreading and zero-copy message passing). High performance, safety critical applications such as self-driving car pipelines deployed in production should use the Rust API to take full advantage of ERDOS.

ERDOS Design

ERDOS is a streaming dataflow system designed for self-driving car pipelines and robotics applications.

Components of the pipelines are implemented as operators which are connected by data streams. The set of operators and streams forms the dataflow graph, the representation of the pipline that ERDOS processes.

Applications define the dataflow graph by connecting operators to streams in the driver section of the program. Operators are typically implemented elsewhere.

ERDOS is designed for low latency. Self-driving car pipelines require end-to-end deadlines on the order of hundreds of milliseconds for safe driving. Similarly, self-driving cars typically process gigabytes per second of data on small clusters. Therefore, ERDOS is optimized to send small amounts of data (gigabytes as opposed to terabytes) as quickly as possible.

ERDOS provides determinisim through watermarks. Low watermarks are a bound on the age of messages received and operators will ignore any messages older than the most recent watermark received. By processing on watermarks, applications can avoid non-determinism from processing messages out of order.

Getting involved

If you would like to contact us, you can:

Community on Slack: Join our community on Slack for discussions about development, questions about usage, and feature requests.
Github Issues: For reporting bugs.

We always welcome contributions to ERDOS. One way to get started is to pick one of the issues tagged with good first issue -- these are usually good issues that help you familiarize yourself with the ERDOS code base. Please submit contributions using pull requests.

Comments

Added type annotations to streams.py and message.py

I added the type annotations to Streams.py and message.py and tested it from examples/simple_pipeline.py by adding and running the following code: def f() -> Stream[int]: pass Still trying to figure out if I can add type annotations to split_by_type

opened by HivaMohammadzadeh1 15

[MacOS] Python Build Fails

We started seeing the following error while linking after restructuring the project layout in #189.

The error is as follows:

note: Undefined symbols for architecture x86_64:
            "_PyList_Append", referenced from:
                pyo3::types::list::PyList::append::_$u7b$$u7b$closure$u7d$$u7d$::hb3ee2e9f284d525d in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.11.rcgu.o)
            "_PyBytes_FromStringAndSize", referenced from:
                pyo3::types::bytes::PyBytes::new::h2346bc142757f449 in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.10.rcgu.o)
            "_PyGILState_Release", referenced from:
                _$LT$pyo3..gil..GILGuard$u20$as$u20$core..ops..drop..Drop$GT$::drop::h0875490cce98b755 in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.8.rcgu.o)
            "_Py_IsInitialized", referenced from:
                pyo3::gil::GILGuard::acquire::_$u7b$$u7b$closure$u7d$$u7d$::hd3c8560dc835bafa in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.8.rcgu.o)
            "_PyExc_ValueError", referenced from:
                _$LT$pyo3..exceptions..PyValueError$u20$as$u20$pyo3..type_object..PyTypeInfo$GT$::type_object_raw::h203baf438139c41f in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.0.rcgu.o)
            "_PyEval_ThreadsInitialized", referenced from:
                pyo3::gil::GILGuard::acquire::_$u7b$$u7b$closure$u7d$$u7d$::hd3c8560dc835bafa in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.8.rcgu.o)
            "_PyExc_RuntimeError", referenced from:
                _$LT$pyo3..exceptions..PyRuntimeError$u20$as$u20$pyo3..type_object..PyTypeInfo$GT$::type_object_raw::h5d98c20758e049cf in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.0.rcgu.o)
            "_PyExc_OverflowError", referenced from:
                _$LT$pyo3..exceptions..PyOverflowError$u20$as$u20$pyo3..type_object..PyTypeInfo$GT$::type_object_raw::h1264959a788b625e in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.0.rcgu.o)
            "_PyExc_AttributeError", referenced from:
                _$LT$pyo3..exceptions..PyAttributeError$u20$as$u20$pyo3..type_object..PyTypeInfo$GT$::type_object_raw::h444faae03f915602 in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.0.rcgu.o)
            "_PyExc_BaseException", referenced from:
                _$LT$pyo3..exceptions..PyBaseException$u20$as$u20$pyo3..type_object..PyTypeInfo$GT$::type_object_raw::hf9a5e6af9086ad35 in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.0.rcgu.o)
            "_PyNumber_Index", referenced from:
                pyo3::types::num::_$LT$impl$u20$pyo3..conversion..FromPyObject$u20$for$u20$u64$GT$::extract::h98d5978ac258bae4 in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.15.rcgu.o)
            "_PySequence_Check", referenced from:
                _$LT$pyo3..types..sequence..PySequence$u20$as$u20$pyo3..conversion..PyTryFrom$GT$::try_from::h2dc65b78675b0ceb in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.15.rcgu.o)
            "_PyGILState_Ensure", referenced from:
                pyo3::gil::GILGuard::acquire_unchecked::hc686c27195a27bb3 in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.8.rcgu.o)
            "_PyCFunction_NewEx", referenced from:
                pyo3::types::function::PyCFunction::internal_new_from_pointers::h3dfba0bb2114f4ef in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.15.rcgu.o)
            "_PyErr_Print", referenced from:
                pyo3::err::panic_after_error::hf97c212b8c021511 in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.15.rcgu.o)
            "_PyErr_NormalizeException", referenced from:
                pyo3::err::PyErr::normalized::h40e8f4bd4677a4be in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.15.rcgu.o)
            "_PyErr_GivenExceptionMatches", referenced from:
                pyo3::err::PyErr::is_instance::h6d8c078c385141c4 in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.15.rcgu.o)
            "_PyErr_PrintEx", referenced from:
                pyo3::err::PyErr::take::h5624bb2bfe614f3f in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.15.rcgu.o)
                pyo3::err::PyErr::print::h94540702bdf29637 in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.15.rcgu.o)
            "_PyObject_Str", referenced from:
                pyo3::types::any::PyAny::str::hc6e5d7ce429ff947 in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.7.rcgu.o)
            "_PyObject_Repr", referenced from:
                pyo3::types::any::PyAny::repr::h2e828644f171a29c in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.7.rcgu.o)
            "_PyObject_SetAttrString", referenced from:
                pyo3::type_object::initialize_tp_dict::h85474ffa22186f1c in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.9.rcgu.o)
            "_PyLong_AsUnsignedLongLong", referenced from:
                pyo3::types::num::_$LT$impl$u20$pyo3..conversion..FromPyObject$u20$for$u20$u64$GT$::extract::h98d5978ac258bae4 in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.15.rcgu.o)
            "_PyBool_Type", referenced from:
                pyo3::ffi::boolobject::PyBool_Check::he43343da1f685b26 in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.7.rcgu.o)
            "_PyImport_AddModule", referenced from:
                pyo3::python::Python::run_code::hcdebb70890cc601b in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.3.rcgu.o)
            "_PyUnicode_FromStringAndSize", referenced from:
                pyo3::types::string::PyString::new::ha54147c354b58f93 in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.13.rcgu.o)
            "_PyObject_GenericSetDict", referenced from:
                pyo3::pyclass::push_dict_getset::h1f5b8f42adcfeda9 in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.14.rcgu.o)
            "_PyDict_New", referenced from:
                pyo3::types::dict::PyDict::new::he5971f940984fcf9 in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.6.rcgu.o)
            "_PyBaseObject_Type", referenced from:
                _$LT$pyo3..types..any..PyAny$u20$as$u20$pyo3..type_object..PyTypeInfo$GT$::type_object_raw::h8201a6fbce7cd3d2 in erdos_python.252pvga0nl19qj44.rcgu.o
                _$LT$pyo3..pyclass_init..PyNativeTypeInitializer$LT$T$GT$$u20$as$u20$pyo3..pyclass_init..PyObjectInit$LT$T$GT$$GT$::into_new_object::h8e220cd96ee46e30 in erdos_python.2ze6cujhuouzjoit.rcgu.o
                _$LT$pyo3..pycell..PyCellBase$LT$U$GT$$u20$as$u20$pyo3..pycell..PyCellLayout$LT$T$GT$$GT$::tp_dealloc::h17cb07af0edd0e70 in erdos_python.xm8924r44o4qwn0.rcgu.o
            "_PyImport_Import", referenced from:
                pyo3::types::module::PyModule::import::h12ce7f05bb31e648 in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.2.rcgu.o)
            "__Py_Dealloc", referenced from:
                pyo3::ffi::object::Py_DECREF::hd1cf0d0adcd38b6f in erdos_python.flqabfgcxz43r2.rcgu.o
                pyo3::ffi::object::Py_DECREF::h1249eddce9b41e89 in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.15.rcgu.o)
            "_PyIter_Next", referenced from:
                _$LT$$RF$pyo3..types..iterator..PyIterator$u20$as$u20$core..iter..traits..iterator..Iterator$GT$::next::h26f75ab62926198b in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.2.rcgu.o)
            "__Py_NoneStruct", referenced from:
                pyo3::ffi::object::Py_None::h4bb72b47c0c5129b in erdos_python.flqabfgcxz43r2.rcgu.o
                pyo3::ffi::object::Py_None::h32f1dc2ca1a0868c in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.15.rcgu.o)
            "_PyType_IsSubtype", referenced from:
                pyo3::ffi::object::PyObject_TypeCheck::h3da3a145cdfbe1b5 in erdos_python.flqabfgcxz43r2.rcgu.o
            "_PyEval_SaveThread", referenced from:
                pyo3::python::Python::allow_threads::h105c4d9ad13befb9 in erdos_python.47v7y8yi72iy0ab7.rcgu.o
                pyo3::python::Python::allow_threads::h69605b42ca1ef95b in erdos_python.47v7y8yi72iy0ab7.rcgu.o
                pyo3::python::Python::allow_threads::h7824bca5b6f5f49d in erdos_python.47v7y8yi72iy0ab7.rcgu.o
                pyo3::python::Python::allow_threads::hfd1a779d7cdd9773 in erdos_python.47v7y8yi72iy0ab7.rcgu.o
            "_PyObject_Free", referenced from:
                pyo3::pyclass::tp_init_additional::h07f065dc15e0fac8 in erdos_python.bisr1ua0raercj2.rcgu.o
                pyo3::pyclass::tp_init_additional::h0a1fb2a3f3c89217 in erdos_python.bisr1ua0raercj2.rcgu.o
                pyo3::pyclass::tp_init_additional::h1a0fe46809097c9c in erdos_python.bisr1ua0raercj2.rcgu.o
                pyo3::pyclass::tp_init_additional::h2ebe2ca40cef899e in erdos_python.bisr1ua0raercj2.rcgu.o
                pyo3::pyclass::tp_init_additional::h9383fcab93ae7dbd in erdos_python.bisr1ua0raercj2.rcgu.o
                pyo3::pyclass::tp_init_additional::h9bab9da1bcf38e80 in erdos_python.bisr1ua0raercj2.rcgu.o
                pyo3::pyclass::tp_init_additional::hbf26ca353ae9dd4a in erdos_python.bisr1ua0raercj2.rcgu.o
                ...
            "_PyObject_GetIter", referenced from:
                pyo3::types::iterator::PyIterator::from_object::h76aed69c8e87d3de in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.2.rcgu.o)
            "_PyDict_Next", referenced from:
                _$LT$pyo3..types..dict..PyDictIterator$u20$as$u20$core..iter..traits..iterator..Iterator$GT$::next::h4cd638da7f9214e0 in erdos_python.4w86bcgc2lc4wulx.rcgu.o
            "_PyObject_Malloc", referenced from:
                pyo3::pyclass::tp_init_additional::h07f065dc15e0fac8 in erdos_python.bisr1ua0raercj2.rcgu.o
                pyo3::pyclass::tp_init_additional::h0a1fb2a3f3c89217 in erdos_python.bisr1ua0raercj2.rcgu.o
                pyo3::pyclass::tp_init_additional::h1a0fe46809097c9c in erdos_python.bisr1ua0raercj2.rcgu.o
                pyo3::pyclass::tp_init_additional::h2ebe2ca40cef899e in erdos_python.bisr1ua0raercj2.rcgu.o
                pyo3::pyclass::tp_init_additional::h9383fcab93ae7dbd in erdos_python.bisr1ua0raercj2.rcgu.o
                pyo3::pyclass::tp_init_additional::h9bab9da1bcf38e80 in erdos_python.bisr1ua0raercj2.rcgu.o
                pyo3::pyclass::tp_init_additional::hbf26ca353ae9dd4a in erdos_python.bisr1ua0raercj2.rcgu.o
                ...
            "_PyObject_SetAttr", referenced from:
                pyo3::types::any::PyAny::setattr::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h5b3c6bc73b3bdf20 in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.7.rcgu.o)
                pyo3::types::any::PyAny::setattr::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::hc162aa0ccbde8787 in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.7.rcgu.o)
            "_PyEval_RestoreThread", referenced from:
                _$LT$pyo3..python..Python..allow_threads..RestoreGuard$u20$as$u20$core..ops..drop..Drop$GT$::drop::h79064b282515ab41 in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.3.rcgu.o)
            "_PyObject_GenericGetDict", referenced from:
                pyo3::pyclass::push_dict_getset::h1f5b8f42adcfeda9 in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.14.rcgu.o)
            "_PyErr_Fetch", referenced from:
                pyo3::err::PyErr::take::h5624bb2bfe614f3f in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.15.rcgu.o)
            "__Py_NotImplementedStruct", referenced from:
                pyo3::ffi::object::Py_NotImplemented::h9a3eb059dd172efc in erdos_python.flqabfgcxz43r2.rcgu.o
            "_PyUnicode_AsUTF8AndSize", referenced from:
                pyo3::types::string::PyString::to_str::hcb33794189cc9c57 in erdos_python.6zhjd1eskm0r3j3.rcgu.o
                pyo3::types::string::PyString::to_str::h99a9ffeded43257e in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.13.rcgu.o)
            "_PyExc_TypeError", referenced from:
                _$LT$pyo3..exceptions..PyTypeError$u20$as$u20$pyo3..type_object..PyTypeInfo$GT$::type_object_raw::hebc536565292a07c in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.0.rcgu.o)
            "_PyErr_NewException", referenced from:
                pyo3::err::PyErr::new_type::hb9190e49e1c74123 in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.15.rcgu.o)
            "__Py_TrueStruct", referenced from:
                pyo3::ffi::boolobject::Py_True::h685a8e11c1f9a93c in erdos_python.18yekmqo97jwh5vu.rcgu.o
                pyo3::ffi::boolobject::Py_True::hc99fcef93ce2dd43 in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.7.rcgu.o)
            "_PyObject_GetAttr", referenced from:
                pyo3::instance::Py$LT$T$GT$::call_method::_$u7b$$u7b$closure$u7d$$u7d$::h2515ddd9bd89d8a4 in erdos_python.4o69xy5l91ev29mc.rcgu.o
                pyo3::instance::Py$LT$T$GT$::call_method::_$u7b$$u7b$closure$u7d$$u7d$::h577f4c78d15d3649 in erdos_python.4o69xy5l91ev29mc.rcgu.o
                pyo3::instance::Py$LT$T$GT$::call_method::_$u7b$$u7b$closure$u7d$$u7d$::ha861ea35d8e3fe9f in erdos_python.4o69xy5l91ev29mc.rcgu.o
                pyo3::instance::Py$LT$T$GT$::call_method::_$u7b$$u7b$closure$u7d$$u7d$::hfdc7f67ae598d8dd in erdos_python.4o69xy5l91ev29mc.rcgu.o
                pyo3::instance::Py$LT$T$GT$::getattr::_$u7b$$u7b$closure$u7d$$u7d$::h3f71a55a17e1edb5 in erdos_python.4o69xy5l91ev29mc.rcgu.o
                pyo3::types::any::PyAny::getattr::_$u7b$$u7b$closure$u7d$$u7d$::hd9c9066e5cbf8344 in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.7.rcgu.o)
            "_PyEval_EvalCode", referenced from:
                pyo3::python::Python::run_code::hcdebb70890cc601b in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.3.rcgu.o)
            "_PyModule_GetName", referenced from:
                pyo3::types::module::PyModule::name::h62b90ec1e5c09e4c in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.2.rcgu.o)
            "_PyTuple_SetItem", referenced from:
                pyo3::types::tuple::_$LT$impl$u20$pyo3..conversion..IntoPy$LT$pyo3..instance..Py$LT$pyo3..types..tuple..PyTuple$GT$$GT$$u20$for$u20$$LP$T0$C$$RP$$GT$::into_py::h641f4545f1f0d914 in erdos_python.4o69xy5l91ev29mc.rcgu.o
                pyo3::types::tuple::_$LT$impl$u20$pyo3..conversion..IntoPy$LT$pyo3..instance..Py$LT$pyo3..types..tuple..PyTuple$GT$$GT$$u20$for$u20$$LP$T0$C$T1$RP$$GT$::into_py::h148c64488e997267 in erdos_python.4o69xy5l91ev29mc.rcgu.o
                pyo3::types::tuple::_$LT$impl$u20$pyo3..conversion..IntoPy$LT$pyo3..instance..Py$LT$pyo3..types..tuple..PyTuple$GT$$GT$$u20$for$u20$$LP$T0$C$T1$C$T2$RP$$GT$::into_py::hdc86ffabd05cec8d in erdos_python.4o69xy5l91ev29mc.rcgu.o
                pyo3::types::tuple::_$LT$impl$u20$pyo3..conversion..IntoPy$LT$pyo3..instance..Py$LT$pyo3..types..any..PyAny$GT$$GT$$u20$for$u20$$LP$T0$C$T1$RP$$GT$::into_py::h2008c562eea6cf75 in erdos_python.8sejcmq4e5nz023.rcgu.o
                pyo3::types::tuple::_$LT$impl$u20$pyo3..conversion..IntoPy$LT$pyo3..instance..Py$LT$pyo3..types..tuple..PyTuple$GT$$GT$$u20$for$u20$$LP$T0$C$$RP$$GT$::into_py::h7a4bfbe9345af1ec in erdos_python.cegfz3138195vk1.rcgu.o
                pyo3::types::tuple::_$LT$impl$u20$pyo3..conversion..IntoPy$LT$pyo3..instance..Py$LT$pyo3..types..tuple..PyTuple$GT$$GT$$u20$for$u20$$LP$T0$C$T1$RP$$GT$::into_py::h036143c23271057d in erdos_python.drmlcrq1aamodz1.rcgu.o
                pyo3::types::tuple::_$LT$impl$u20$pyo3..conversion..IntoPy$LT$pyo3..instance..Py$LT$pyo3..types..tuple..PyTuple$GT$$GT$$u20$for$u20$$LP$T0$C$T1$C$T2$RP$$GT$::into_py::h507b191e1a71e8d4 in erdos_python.drmlcrq1aamodz1.rcgu.o
                ...
            "_PyErr_Restore", referenced from:
                pyo3::err::PyErr::restore::h9927ce6bbda65927 in erdos_python.4qjcay4yuyg1tqjv.rcgu.o
                pyo3::err::PyErr::take::h5624bb2bfe614f3f in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.15.rcgu.o)
                pyo3::err::PyErr::restore::h3b8e8880de9cd0f6 in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.15.rcgu.o)
            "_PyBytes_AsString", referenced from:
                pyo3::types::bytes::PyBytes::as_bytes::h5ea66dda28d9eac9 in erdos_python.47b46rh3qn2mwosw.rcgu.o
                pyo3::types::bytes::PyBytes::as_bytes::h099652a35b3bdc12 in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.10.rcgu.o)
            "_PyType_FromSpec", referenced from:
                pyo3::pyclass::create_type_object::h181fe10035563334 in erdos_python.bisr1ua0raercj2.rcgu.o
                pyo3::pyclass::create_type_object::h34f4e0969eb9a036 in erdos_python.bisr1ua0raercj2.rcgu.o
                pyo3::pyclass::create_type_object::h60ccaad2d05d21e5 in erdos_python.bisr1ua0raercj2.rcgu.o
                pyo3::pyclass::create_type_object::h7e75473202f0e72c in erdos_python.bisr1ua0raercj2.rcgu.o
                pyo3::pyclass::create_type_object::h800d89c5a27df072 in erdos_python.bisr1ua0raercj2.rcgu.o
                pyo3::pyclass::create_type_object::h95336a2d8ddd0a5e in erdos_python.bisr1ua0raercj2.rcgu.o
                pyo3::pyclass::create_type_object::haa61076de53ecc77 in erdos_python.bisr1ua0raercj2.rcgu.o
                ...
            "__Py_FalseStruct", referenced from:
                pyo3::ffi::boolobject::Py_False::ha783acfd459976ba in erdos_python.18yekmqo97jwh5vu.rcgu.o
            "_PyType_GenericAlloc", referenced from:
                _$LT$pyo3..pyclass_init..PyNativeTypeInitializer$LT$T$GT$$u20$as$u20$pyo3..pyclass_init..PyObjectInit$LT$T$GT$$GT$::into_new_object::h8e220cd96ee46e30 in erdos_python.2ze6cujhuouzjoit.rcgu.o
            "_PyBytes_Size", referenced from:
                pyo3::types::bytes::PyBytes::as_bytes::h5ea66dda28d9eac9 in erdos_python.47b46rh3qn2mwosw.rcgu.o
                pyo3::types::bytes::PyBytes::as_bytes::h099652a35b3bdc12 in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.10.rcgu.o)
            "_PyLong_FromUnsignedLongLong", referenced from:
                pyo3::types::num::_$LT$impl$u20$pyo3..conversion..IntoPy$LT$pyo3..instance..Py$LT$pyo3..types..any..PyAny$GT$$GT$$u20$for$u20$u64$GT$::into_py::hc88d51cb7e9e88ee in erdos_python.3creat87iwuzxe8.rcgu.o
            "_PyModule_Create2", referenced from:
                pyo3::ffi::modsupport::PyModule_Create::hff6a74f055b5e865 in erdos_python.3cap3utahqi1jyp0.rcgu.o
            "_PyObject_Call", referenced from:
                pyo3::types::any::PyAny::call::h130601c19f73817f in erdos_python.2v9ruangams3xknl.rcgu.o
                pyo3::types::any::PyAny::call::h5986e7ee6bd0ea9d in erdos_python.2v9ruangams3xknl.rcgu.o
                pyo3::types::any::PyAny::call::hd4b2f85fd2143aec in erdos_python.2v9ruangams3xknl.rcgu.o
                pyo3::types::any::PyAny::call::hdf92234573a52e47 in erdos_python.2v9ruangams3xknl.rcgu.o
                pyo3::instance::Py$LT$T$GT$::call_method::_$u7b$$u7b$closure$u7d$$u7d$::h2515ddd9bd89d8a4 in erdos_python.4o69xy5l91ev29mc.rcgu.o
                pyo3::instance::Py$LT$T$GT$::call_method::_$u7b$$u7b$closure$u7d$$u7d$::h577f4c78d15d3649 in erdos_python.4o69xy5l91ev29mc.rcgu.o
                pyo3::instance::Py$LT$T$GT$::call_method::_$u7b$$u7b$closure$u7d$$u7d$::ha861ea35d8e3fe9f in erdos_python.4o69xy5l91ev29mc.rcgu.o
                ...
            "_PyUnicode_AsEncodedString", referenced from:
                pyo3::types::string::PyString::to_string_lossy::h99ed9dd46021602b in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.13.rcgu.o)
            "_PyDict_SetItem", referenced from:
                pyo3::types::dict::PyDict::set_item::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h09184eb549aeac8d in erdos_python.1gxqznnivutptqz0.rcgu.o
                pyo3::types::dict::PyDict::set_item::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h105b85e719a64457 in erdos_python.1gxqznnivutptqz0.rcgu.o
                pyo3::types::dict::PyDict::set_item::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h10dbc39666f1ce96 in erdos_python.1gxqznnivutptqz0.rcgu.o
                pyo3::types::dict::PyDict::set_item::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h82681fffbe929671 in erdos_python.1gxqznnivutptqz0.rcgu.o
                pyo3::types::dict::PyDict::set_item::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::hb0c73846b22b590f in erdos_python.1gxqznnivutptqz0.rcgu.o
            "_Py_CompileStringExFlags", referenced from:
                pyo3::ffi::cpython::pythonrun::Py_CompileString::h5bfa27fc58620660 in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.9.rcgu.o)
            "_PyTuple_New", referenced from:
                pyo3::types::tuple::_$LT$impl$u20$pyo3..conversion..IntoPy$LT$pyo3..instance..Py$LT$pyo3..types..tuple..PyTuple$GT$$GT$$u20$for$u20$$LP$T0$C$$RP$$GT$::into_py::h641f4545f1f0d914 in erdos_python.4o69xy5l91ev29mc.rcgu.o
                pyo3::types::tuple::_$LT$impl$u20$pyo3..conversion..IntoPy$LT$pyo3..instance..Py$LT$pyo3..types..tuple..PyTuple$GT$$GT$$u20$for$u20$$LP$T0$C$T1$RP$$GT$::into_py::h148c64488e997267 in erdos_python.4o69xy5l91ev29mc.rcgu.o
                pyo3::types::tuple::_$LT$impl$u20$pyo3..conversion..IntoPy$LT$pyo3..instance..Py$LT$pyo3..types..tuple..PyTuple$GT$$GT$$u20$for$u20$$LP$T0$C$T1$C$T2$RP$$GT$::into_py::hdc86ffabd05cec8d in erdos_python.4o69xy5l91ev29mc.rcgu.o
                pyo3::types::tuple::PyTuple::new::h2e150a5ca7556a94 in erdos_python.555cx17egk1h4r95.rcgu.o
                pyo3::types::tuple::PyTuple::new::h57a2cd2bab15c7e9 in erdos_python.555cx17egk1h4r95.rcgu.o
                pyo3::types::tuple::_$LT$impl$u20$pyo3..conversion..IntoPy$LT$pyo3..instance..Py$LT$pyo3..types..any..PyAny$GT$$GT$$u20$for$u20$$LP$T0$C$T1$RP$$GT$::into_py::h2008c562eea6cf75 in erdos_python.8sejcmq4e5nz023.rcgu.o
                pyo3::types::tuple::_$LT$impl$u20$pyo3..conversion..IntoPy$LT$pyo3..instance..Py$LT$pyo3..types..tuple..PyTuple$GT$$GT$$u20$for$u20$$LP$T0$C$$RP$$GT$::into_py::h7a4bfbe9345af1ec in erdos_python.cegfz3138195vk1.rcgu.o
                ...
            "_PySequence_Size", referenced from:
                pyo3::types::sequence::PySequence::len::hfc6ab9ea310fc9ec in erdos_python.2091xl2n4rgokxox.rcgu.o
            "_PyExc_Exception", referenced from:
                _$LT$pyo3..exceptions..PyException$u20$as$u20$pyo3..type_object..PyTypeInfo$GT$::type_object_raw::h7c4f62a11ebdcf87 in erdos_python.2hgwowqwktp2ewkg.rcgu.o
            "_PyModule_GetDict", referenced from:
                pyo3::python::Python::run_code::_$u7b$$u7b$closure$u7d$$u7d$::habb39b51ba033263 in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.3.rcgu.o)
            "_PyList_New", referenced from:
                pyo3::types::list::new_from_iter::h9b38ca1132715523 in erdos_python.4yq4p61w6u6q9tze.rcgu.o
                pyo3::types::list::PyList::empty::he2dc7cfee39014a8 in libpyo3-5c021ee12bba24f4.rlib(pyo3-5c021ee12bba24f4.pyo3.fc75e33f-cgu.11.rcgu.o)
          ld: symbol(s) not found for architecture x86_64
          clang: error: linker command failed with exit code 1 (use -v to see invocation)

System details as follows:

Python version: 3.7.12
MacOS version: 12.1
Command Line Tools version: 13.2.0.0.1.1638488800

opened by sukritkalra 6

[Python]can't find Rust compiler
I'm new for erdos, and I'm using README for installation and demo, I've already done this rustup default nightly # use nightly Rust toolchain git clone https://github.com/erdos-project/erdos.git && cd erdos and simple works fine, python3 python/examples/simple_pipeline.py But I'm stuck at python3 python/setup.py develop the error info is

running develop running egg_info writing python/erdos.egg-info/PKG-INFO writing dependency_links to python/erdos.egg-info/dependency_links.txt writing requirements to python/erdos.egg-info/requires.txt writing top-level names to python/erdos.egg-info/top_level.txt writing manifest file 'python/erdos.egg-info/SOURCES.txt' running build_ext running build_rust error: can't find Rust compiler

If you are using an outdated pip version, it is possible a prebuilt wheel is available for this package but pip is not able to install from it. Installing from the wheel would avoid the need for a Rust compiler.

To update pip, run:

pip install --upgrade pip

and then retry package installation.

If you did intend to build this package from source, try installing a Rust compiler from your system package manager and ensure it is on the PATH during installation. Alternatively, rustup (available at https://rustup.rs) is the recommended way to download and update the Rust compiler toolchain.

My host machine info:

Linux cluster01 4.15.0-112-generic #113~16.04.1-Ubuntu SMP Fri Jul 10 04:37:08 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

I've double check the requirements packges, reinstall pip but still the same error Any suggestion? many thanks advance.
opened by yuanhaorannnnnn 3
Replace YAPF with Black + isort in CI
Adds a setup_hooks.sh script to install pre-commit and initialize it.

Added isort to the pre-commit hooks for automatic sorting of Python imports.

Ran isort on all Python files.
opened by sukritkalra 3
[Python] Add name setters and getters
Add name setters and getters to PyStream, PyIngestStream, PyReadStream, PyWriteStream

Remove _name from Python constructors

Add set_name to streams in Python
opened by objorkman 3
Nested graph execution
Support nested graphs by overriding Graph.construct.

TODO:

[x] Support execution of operators in nested graphs.

[x] Add simple nested graph test.

[x] Verify implementation of nested graph execution algorithm with @ICGog.

[x] Add nested graph in loop test.

[x] Add test for multiple layers of nested graphs.

[x] Throw exceptions for illegal nested graph behavior and add tests.

Current implementation flattens child graphs and inserts all operators into the parent graph. Child graph operators are connected to the parent graph via noop input operator and noop output operator that forward data streams.

Due to this behavior, the final graph after flattening child graphs looks like this for simple_graph_test.py:

A potential weakness of the current implementation is that I cannot create a SquareOp for each of the inputs default_int1 and defaut_int2. However, this should be solved once we add the operator scale-up API.
opened by pschafhalter 3
[ROS] Automatically set publisher buffer size based on message types

The ROS publisher and subscriber API expects users to specify queue sizes and buffers sizes. Users are expected to provide a buffer size that is greater than the size of all messages in the queue. Otherwise, ROS drops messages.

Update ros_output_data_stream.py to automatically compute the size of the buffer based on the size of the queues.
enhancement good first issue

opened by ICGog 3
[Cross Compile with Python]
Hello dear community, now I'm trying to implement ERDOS on a NVIDIA device which the arch is aarch64. I've finished these steps:

apt install gcc-aarch64-linux-gnu

rustup target add aarch64-unknown-linux-gnu

modify the ~/.cargo/config.toml as

[target.aarch64-unknown-linux-gnu]

linker = "aarch64-linux-gnu-gcc"

ar = "aarch64-unknown-linux-gnu-gcc"

Then follow the README on Github, I can build ERDOS(X86-64) successfully both on Rust and Python

I add the following code on ~/.cargo/config.toml [build] target = "aarch64-unknown-linux-gnu" As for step 5, I want build ERDOS on aarch64 with Python api, but the bulld processs failed at the last minute, here are the error info: The most critical info is: Finished dev [unoptimized + debuginfo] target(s) in 1m 33s error: Rust build failed; unable to find any *erdos.so in /mnt/erdos/target/debug

Any help or insight for this? Thanks for all!
opened by yuanhaorannnnnn 2
[LINQ] Add timestamp join
Introduces the TimestampJoin, which joins messages with matching timestamps from two different streams by performing a cartesian product. I've tested the operator locally, and it seems to work.

A few things I wasn't sure about that can hopefully be addressed in review:

Consistent naming: TimestampJoin seems more ergonomic, but this should be named TimestampJoinOperator to stay consistent with the other operators.

Example: I wasn't sure how to best add the timestamp join to the existing LINQ and full pipeline examples. These examples may also need to be revamped.
opened by pschafhalter 2
Unable to pip install erdos==0.3.1
I'm unable to install using pip3 install erdos==0.3.1. I've tried 3.9 and 3.8, and my friend tried 3.7, but none of these work. My full error message is below. Thanks!

ERROR: Could not find a version that satisfies the requirement erdos==0.3.1 (from versions: 0.1, 0.1.1, 0.1.2, 0.1.3, 0.2.0) ERROR: No matching distribution found for erdos==0.3.1
opened by armaangoel78 2
Fault Tolerant basic pipeline

Got basic flow of the pipeline working for the first part of the paper. I set up some of the state for the take over and catch up part, but have not begun to implement it completely.

opened by sguduguntla 2
How to add_deadline to operator? example/full_pipeline does not work

Hi, thanks for your great work. I may need some help.

I tried to use the deadline control in one operator. I followed the full_pipeline example: implement setup func for SqureOperator, and in the function, invoke ctx.add_deadline to set a deadline for the operator. However, it is useless to do so. The deadline handler doesn't work.

I read the source code and find the reason is in arm_deadlines function. Take src/node/operator_executors/one_in_one_out_executor.rs as an example. The arm_deadlines func only creates a DeadlineEvent when the deadline.get_constrained_read_stream_ids is not empty. However, the deadline's constrained_read_stream(TimestampDeadline.read_stream_ids) is always empty. The only way to insert a read_strem_id is by invoking the on_read_stream func of TimestampDeadline, but on_read_stream is not invoked during the running of full_pipeline.

I solved the problem(deadline control does not work) by making the following changes to ctx.add_deadline in full_pipeline.rs: ctx.add_deadline(TimestampDeadline::new( move |_s: &(), _t: &Timestamp| -> Duration { Duration::new(2, 0) }, |_s: &(), _t: &Timestamp| { tracing::info!("SquareOperator @ {:?}: Missed deadline.", _t); }, ).on_read_stream(ctx.get_read_stream_ids()[0])); To achieve this, I have to make get_read_stream_ids pub.

I dont know if I am right. Please give me an example of how to add_deadline correctly. I appreciate your help. Thanks!

opened by wutianze 0
Events do not process when an operator is under high load

When an operator is under high load (i.e. messages are coming in at a very high rate), events are only inserted into the lattice and not removed/processed until the rate of incoming messages decreases.

This may be due to a prioritization issue related to how the operator executor interacts with Rust futures or the underlying runtime since the threads responsible for running events should access the lattice at the same time as the thread inserting events.

opened by pschafhalter 0

[Rust] Add a prelude

To use ERDOS for a simple dataflow, users currently need to do the following:

use erdos::{
    self,
    dataflow::{
        message::*,
        operators::{JoinOperator, MapOperator},
        stream::{ExtractStream, IngestStream, WriteStreamT},
        Operator, OperatorConfig, ReadStream, WriteStream,
    },
};

This should be simplified to use erdos::prelude::* to cover the essential imports, like in other libraries.

opened by pschafhalter 0

[Rust] Clean up imports and namespace

Some imports are duplicates, e.g. erdos::dataflow::operators::Operator and erdos::dataflow::Operator.

Only 1 path per object is simpler (this likely will impact #110).

opened by pschafhalter 0
Unsafe access to state in multi-threaded operators
Currently, access to state isn't locked. When using the TimeVersionedState, we could introduce unsafe accesses due to concurrency if a watermark callback for time t and a data callback for time t + 1 run simultaneously.

Currently, this shouldn't cause problems because only 1 callback can access state at a time. In the future, we may want to increase parallelism in multi-threaded operators and then this will cause problems.

Lines causing the problem (also in scripts/make_callback_builder.py): https://github.com/erdos-project/erdos/blob/fd590d0922aceb30a690a3e4192214154d8ae0ee/src/dataflow/stream/internal_stateful_read_stream.rs#L91-L95

Potential solutions:

Locking.

Cloning.

Providing a view on a subsection of the state.
opened by pschafhalter 0

Releases(v0.4.0)

v0.4.0(Apr 6, 2022)
Major API Changes

New operator API in which operators implement either Source, Sink OneInOneOut, OneInTwoOut, or TwoInOneOut.

New API for connecting streams which uses functions (e.g. connect_one_in_one_out) instead of macros.

New Features

[Rust] ROS bridge operators (via @objorkman).

[Rust] LINQ-style API for streams (via @sguduguntla).

[Rust] Support for time-versioned state.

[Rust] Experimental deadline support.

[Rust] use clippy for linting.

[Python] use black and isort for code formatting.

Implementation Changes

New worker architecture for processing events generated by operators.

Switch logging from slog to tracing.

New Contributors

@objorkman made their first contribution in https://github.com/erdos-project/erdos/pull/173

@sguduguntla made their first contribution in https://github.com/erdos-project/erdos/pull/170

Source code(tar.gz)
Source code(zip)
v0.3.2(Mar 10, 2022)
Bug Fixes

Fixed the use of global dataflow graph in order to be compatible with other executors (#145).

Added missing try_read for PyExtractStream (#146).

Ensure that destroy is invoked when source operators finish running (#152).

Source code(tar.gz)
Source code(zip)
v0.3.1(Sep 25, 2020)
New Features

Improved logging and documentation (#136, #138).

Major API Changes

Simplify OperatorConfig builder pattern (#141).

Implementation Changes

Refactor events and lattice governing operator execution (#135).

Several critical bugfixes (#133, #135, #139).

Updated dependencies (#142).

Source code(tar.gz)
Source code(zip)
v0.3.0(Jul 21, 2020)
New Features

Visualize dataflow graph using DOT files (#71).

Add general dataflow operators (#85, #93).

Time-versioned state (#95).

Shut down and reset nodes and dataflows (#74, #114)

Profiling in python (#78).

Major API Changes

Operators must implement the Operator trait (#77).

Expose name and id properties in Python operators (#79).

Must provide operator configs (#81).

Implementation Changes

Zero-copy communication for sending messages within a node, and faster message sending across nodes (#119).

Order events created upon message receipt with a lattice to control parallelism and safe execution (#103).

Introduce a control plane connecting nodes for internal communication (#59, #62).

Use control plane to ensure that all operators are set up before executing dataflow (#59).

Enable processing callbacks in parallel (#108).

Verify that corresponding types in new() and connect() match (#67).

Avoid busy-looping (#103).

Source code(tar.gz)
Source code(zip)
v0.2.0(Jan 20, 2020)

First release of the Rust-based ERDOS execution engine.
Source code(tar.gz)
Source code(zip)