Shred - Shared resource dispatcher

Overview

shred - Shared resource dispatcher

Build Status Crates.io MIT/Apache Docs.rs LoC

This library allows to dispatch systems, which can have interdependencies, shared and exclusive resource access, in parallel.

Usage

extern crate shred;

use shred::{DispatcherBuilder, Read, Resource, ResourceId, System, SystemData, World, Write};

#[derive(Debug, Default)]
struct ResA;

#[derive(Debug, Default)]
struct ResB;

#[derive(SystemData)] // Provided with `shred-derive` feature
struct Data<'a> {
    a: Read<'a, ResA>,
    b: Write<'a, ResB>,
}

struct EmptySystem;

impl<'a> System<'a> for EmptySystem {
    type SystemData = Data<'a>;

    fn run(&mut self, bundle: Data<'a>) {
        println!("{:?}", &*bundle.a);
        println!("{:?}", &*bundle.b);
    }
}

fn main() {
    let mut world = World::empty();
    let mut dispatcher = DispatcherBuilder::new()
        .with(EmptySystem, "empty", &[])
        .build();
    world.insert(ResA);
    world.insert(ResB);

    dispatcher.dispatch(&mut world);
}

Please see the benchmark for a bigger (and useful) example.

Required Rust version

1.40 stable

Features

  • lock-free
  • no channels or similar functionality used (-> less overhead)
  • allows both automated parallelization and fine-grained control

Contribution

Contribution is highly welcome! If you'd like another feature, just create an issue. You can also help out if you want to; just pick a "help wanted" issue. If you need any help, feel free to ask!

All contributions are assumed to be dual-licensed under MIT/Apache-2.

License

shred is distributed under the terms of both the MIT license and the Apache License (Version 2.0).

See LICENSE-APACHE and LICENSE-MIT.

Comments
  • Solve the

    Solve the "resource not found" problem

    In the past, many users had difficulties with panics just saying "No resource with the given id". This is neither beginner friendly, nor a good way to handle the error.

    The default behavior has been changed to adding the missing resource automatically. For this, a Default implementation needs to be in place.

    In case a resource does not have a Default implementation, it's recommended that one uses Option<Fetch> and checks whether the resource exists or not. If the system cannot work without the resource, it should simply return.

    In case the resource is absolutely required and the user is sure it exists, they can use FetchExpect. It is pretty much like Fetch was before, but now the panic message has been improved and there's a nightly feature that also prints the type name of the missing resource.

    The overall goal is to make shred more robust, easier to debug and beginner-friendly.


    This change is Reviewable

    enhancement 
    opened by torkleyy 43
  • WIP/PoC: Batch api ergonomics

    WIP/PoC: Batch api ergonomics

    Hello

    As mentioned in #197, I've sketched some API how the batch dispatching could be improved so it's less awkward and doesn't require unsafe.

    This is very much a PoC/WiP, I don't want to merge it as it is. I know there are missing docs, copy-pasted comments that are out of context, extra includes that are no longer needed, some things are public when in fact shouldn't be and in general a lot of garbage. I plan to clean it up before asking for it to get merged.

    What I would like now is to have some feedback on the API. I have some specific questions, but any other suggestions are fine too:

    • Naming. In particular, the simpler case where it is decided up-front how many times the sub-dispatcher should run is currently called multi-batch. I think this is a bad name, because it suggests it's somehow „more“ than the usual but more complex case.
    • The multi-batch is a separate method of the builder. Alternatively, it could be exposed as the wrapper type and one could create it directly. I'm not sure on what's better.

    In general, however, I feel like this makes the API more approachable (obviously, after it gets some docs and examples).

    @azriel91 Do you think I should continue in this? (and, btw, there's a tiny doc update PR from me sitting there for some time as well).

    opened by vorner 29
  • Implemented batch dispatching

    Implemented batch dispatching

    This is the second iteration to implement the Batch dispatching following the feedback received in this PR: https://github.com/slide-rs/shred/pull/144 (Was easier for me redo everything rather change it)

    This is the project to test this feature: shred_test.zip

    feature 
    opened by AndreaCatania 16
  • Update HashMaps to use hashbrown HashMap

    Update HashMaps to use hashbrown HashMap

    A small update of the FxHashMap HashMap to use the more perfomante hashbrown HashMap.

    hashbrown is a Rust port of Google's high-performance SwissTable hash map and is around 2 times faster than FxHashMap.

    opened by bdelmas 16
  • Replace System with FnMut

    Replace System with FnMut

    The current code for defining a new task looks like this (from the README)

    struct PrintSystem;
    
    impl<'a> System<'a> for PrintSystem {
        type SystemData = PrintData<'a>;
    
        fn work(&mut self, bundle: PrintData<'a>) {
            println!("{:?}", &*bundle.a);
            println!("{:?}", &*bundle.b);
            
            *bundle.b = ResB; // We can mutate ResB here
                              // because it's `FetchMut`.
        }
    }
    

    Every system needs an additional struct (often unit type from my own experience) which becomes quite verbose. Alternative might be replace the Task trait with a function:

    fn system_print(_: &mut (), bundle: PrintData<'a>) {
        println!("{:?}", &*bundle.a);
        println!("{:?}", &*bundle.b);
        *bundle.b = ResB;
    }
    

    The dispatch builder would then take a F: FnMut(&mut T, U), U: SystemData (or something similar).

    let mut dispatcher = DispatcherBuilder::new()
        .add((), system_print, "print", &[]) // Adds a system "print" without dependencies
        .finish();
    

    I'm not 100% sure if this will be possible, tests some time ago looked positive..

    enhancement help wanted difficult postponed 
    opened by msiglreith 16
  • Systems without lifetimes

    Systems without lifetimes

    Hey!

    So to start out: this is an experiment. I don't expect it to be merged, but I also want to make sure to document that I took this approach for a spin.

    I was reading the shred code to try to understand what the lifetime associated to systems was actually used for.

    The lifetime appears to be required to have a named lifetime in the fetch function that is also associated with the return value (e.g. Self).

    This guarantees that Resources cannot be modified, while the fetch function is running.

    I then tried to figure out what the scope of this lifetime would be. And it appears like this doesn't actually come into effect that much. Primarily it's used in RunNow: https://github.com/slide-rs/shred/blob/master/src/system.rs#L124

    But here is the kicker: resources is already borrowed here, so do we really need to track the lifetime of the values borrowed from it? These should already be valid for the duration of this function call, which calls into the System itself.

    So this is a patch that removes that lifetime, in favor of making Read and Write keep track of a raw pointer into TrustCell.

    Some major caveats

    Resources wrapped inRead and Write must never, ever, ever be able to leave or be stored in the System. This is normally accomplished with the lifetime associated with the system itself. This could probably be patched by treating them as weak pointers to a specific generation of the collection of resources, and panic in case someone tried to use them if this generation has changed.

    Some APIs have been completely butchered for their existing safety guarantees, and some safety holes are available as APIs. E.g. it's possible to discard and SystemFetch because that's needed to perform composition of fetches.

    To wrap up: this is probably a terrible idea. It was fun to try out, but please don't merge :).

    opened by udoprog 11
  • Add barriers and dispatch tests

    Add barriers and dispatch tests

    cc @nchashch

    This is how I imagined to have those systems divided into stages:

    let dispatcher = DispatcherBuilder::new()
        .add(InputHandler, "handle_input", &[])
        .add(NetworkSynchronizer, "netsync", &[])
        .new_stage()
        .add(ForceGenerator, "force", &[])
        .add(Integrator, "integrator", &["force"])
        .add(EnemySpawner)
        .new_stage()
        .add(Renderer, "render", &[])
        .add(Audio, "audio", &[])
        .build();
    

    cc https://github.com/amethyst/amethyst/issues/238

    discussion 
    opened by torkleyy 11
  • Update cgmath requirement from 0.16 to 0.17

    Update cgmath requirement from 0.16 to 0.17

    Updates the requirements on cgmath to permit the latest version.

    Changelog

    Sourced from cgmath's changelog.

    [v0.17.0] - 2019-01-17

    Added

    • Add signed Angle normalization

    Changed

    • Move lerp() from InnerSpace to VectorSpace
    • const constructors

    [v0.16.1] - 2018-03-21

    Added

    • Implement ElementWise trait for point types
    • Add map function to points and vectors

    Changed

    • Remove BaseNum trait requirement for PointN::new functions

    [v0.16.0] - 2018-01-03

    Added

    • Add InnerSpace::project_on
    • Add Array::len
    • Re-export Bounded and implement for vectors, points, and angles
    • Add vector subtraction to EuclideanSpace
    • Add swizzle functions behinde that "swizzle" feature
    • Add Matrix4::look_at_dir

    Changed

    • Return Option from cast functions

    [v0.15.0] - 2017-07-30

    Added

    • Implement mint conversions behind a feature
    • Add Quaternion::cast

    Changed

    • Rename use_simd feature to simd
    • Rename eders feature to serde

    Fixed

    ... (truncated)
    Commits

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Note: This repo was added to Dependabot recently, so you'll receive a maximum of 5 PRs for your first few update runs. Once an update run creates fewer than 5 PRs we'll remove that limit.

    You can always request more updates by clicking Bump now in your Dependabot dashboard.

    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot ignore this [patch|minor|major] version will close this PR and stop Dependabot creating any more for this minor/major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language
    • @dependabot badge me will comment on this PR with code to add a "Dependabot enabled" badge to your readme

    Additionally, you can set the following in your Dependabot dashboard:

    • Update frequency (including time of day and day of week)
    • Automerge options (never/patch/minor, and dev/runtime dependencies)
    • Pull request limits (per update run and/or open at any time)
    • Out-of-range updates (receive only lockfile updates, if desired)
    • Security updates (receive only security updates, if desired)

    Finally, you can contact us by mentioning @dependabot.

    dependencies 
    opened by dependabot-preview[bot] 9
  • Added DispatcherBuilder::contains

    Added DispatcherBuilder::contains

    Adds a contains method to DispatcherBuilder that allows the caller to determine if a system with the given name has already been added.

    This supports https://github.com/amethyst/specs/issues/736.

    opened by schell 8
  • Ideas for improving the Batch interface

    Ideas for improving the Batch interface

    Hello

    I've been mulling a bit over the batch execution API. To be honest, I don't find it very comfortable to use. I see several problems:

    • It's verbose. To use it, you have to implement two traits (System and BatchController), and for the System you have to implement even the methods that you don't really care about most of the time (eg. accessor and setup).
    • Unsafe is sticking out of it even though there's nothing the user needs to uphold towards the library or anything unsafe the user wants to do.
    • As the controller is passed as a type parameter, not as a value, there's no (reasonable) way to pass additional info into it.

    If my guess is correct, the System is used mostly to make the implementation easier, because the Dispatcher just works with Systems all the time.

    I have a proposal how to make the interface better while not introducing much more complexity.

    Let's have a trait, something like this (modulo naming, lifetimes...):

    trait BatchController {
        type SystemData: SystemData;
        type Tmp;
        fn run_pre(&mut self, data: Self::SystemData) -> Self::Tmp;
        fn run(&mut self, tmp: Self::Tmp, dispatcher: DispatcherProxy);
    }
    

    Then, internally, there would be some wrapper that would implement the System, hold the dispatcher, and the instance of this new BatchController.

    The relevant part of the example would look something like this:

    // No data needed here, the accessor and dispatcher are part of the wrapper below the surface.
    struct CustomBatchControllerSystem;
    
    impl BatchController for CustomBatchControllerSystem }
        type SystemData = Read<TomatoStore>;
        type Tmp = TomatoStore;
        fn run_pre(&mut self, data: Self::SystemData) -> Self::Tmp {
            *data
        }
        fn run(&mut self, _ts: TomatoStore, dispatcher: DispatcherProxy) {
          for _i in 0..3 {
             // The BatchUncheckedWorld is part of the proxy
             dispatcher.dispatch();
          }
    }
    
    ...
    DispatcherBuilder::new()
        .with(SayHelloSystem, "say_hello", &[])
        .with_batch(
             CustomBatchControllerSystem, // Passing value, not a type
             DispatcherBuilder::new().with(..)
             "BatchSystemTest",
             &[],
         )
         .build();
    

    If you like the idea, I can try making a PR with the changes.

    Alternatively, it would also be possible to create just the wrapper that would implement the current traits. That would preserve the current API, but the cost is that it would still not be possible to pass values in there, only types.

    opened by vorner 8
  • Bump `tynm` to `0.1.3` to be compatible with Rust 1.38.0.

    Bump `tynm` to `0.1.3` to be compatible with Rust 1.38.0.

    tynm 0.1.2 used the todo!() macro which is only stable in Rust 1.40. tynm 0.1.3 switches that to unimplemented!() to be compatible with Rust 1.38.

    This change ensures the fix is used (though consumers may already receive it simply with cargo update).

    opened by azriel91 8
  • Potential bug with DispatcherBuilder::add_batch()

    Potential bug with DispatcherBuilder::add_batch()

    I don't know enough about the inner workings of shred to know if this is actually a bug, but it looks like one.

    DispatcherBuilder::add_batch() has what looks like a copy/paste error:

    let mut reads = dispatcher_builder.stages_builder.fetch_all_reads();
    reads.extend(<T::BatchSystemData as SystemData>::reads());
    reads.sort();
    reads.dedup();
    
    let mut writes = dispatcher_builder.stages_builder.fetch_all_writes();
    writes.extend(<T::BatchSystemData as SystemData>::reads());                 <<<<< here
    writes.sort();
    writes.dedup();
    
    let accessor = BatchAccessor::new(reads, writes);
    

    When determining the write accesses for the batch being added, it calls reads() instead of writes(). It seems like this could pretty negatively affect the ability to parallelize things around the batch.

    I would make a PR, but I wasn't sure if there was actually a reason for doing things this way or not.

    opened by shimaowo 0
  • Separate RunNow and Setup

    Separate RunNow and Setup

    Hi,

    After experimenting a bith with Amethyst, I started wondering why the System trait is responsible for both setup and run (+dispose). I programmed my setup to fetch a resource, then store it as a struct member, but I am forced to wrap it inside an Option and unwrap it upon each call of run. Ideally I would have the System trait not having the run or dispose method, but move those to another trait. The setup would then consume the system and return an instance of this new trait. In this way there will be no more need for me to wrap the resource during setup.

    I tried to tinker with shred's code a bit myself, and it becomes clear that this is not a minor code change. Obviously it would break the API as well. So my question is : will this concept be in a future release ?

    enhancement question discussion 
    opened by Lotrotk 0
  • Rewrite `ThreadPoolWrapper` to use atomic pointer

    Rewrite `ThreadPoolWrapper` to use atomic pointer

    There seems to be no need to have a lock around an option wrapping an Arc - since we have an (allocated) Arc, we can just use its raw pointer for AtomicPtr, and swap it out whenever the pool should be changed.

    cc @AndreaCatania

    enhancement 
    opened by torkleyy 2
  • Ability to suspend Systems

    Ability to suspend Systems

    From what I've seen, there doesn't seem to be the ability to suspend Systems? Basically, after exploring through amethyst and specs, I ended up concluding that shred would presumably be where this functionality would go. Feel free to direct me in the right place if that's not the case.

    Right now, I know that amethyst supports a Pausable wrapper which simply checks a condition every time the system is run; this is okay, but it would be nice to be able to suspend a system (making it no longer dispatched) and be able to later signal it to be added back to the queue. Presumably, this would help make it easier to interface with Generators and the like later on.

    Alternatively, you could argue that the best case for this is to have multiple Dispatchers and use the right one depending on the state of the system. This might be better for cases where the set of currently running systems is swapped, e.g. a pause menu in a game, rather than the generic case which might be better for I/O. If this is the route that seems best, then I can move this issue to specs and discuss that there.

    Anyway, I didn't see a discussion on this, so, I figured I'd start one.

    opened by clarfonthey 1
  • Tools for debugging + profiling dispatch plans

    Tools for debugging + profiling dispatch plans

    It would be useful to be able to inspect the plans shred comes up with, e.g. what systems execute in parallel when, and so on. It would also be useful to be able to get information about the execution times of various systems after a dispatch. I could imagine writing a tool that hooks into shred and generates a graphviz .dot file that lets you debug systems causing bottlenecks, or an amethyst editor plugin that lets you watch the relative execution times of your various systems in real time.

    feature 
    opened by kazimuth 3
Releases(0.14.1)
  • 0.14.1(Jul 14, 2022)

    Changelog

    • Undo performance regression from removing hashbrown by using ahash hasher

    PRs

    • Use ahash instead of SipHash by @torkleyy in https://github.com/amethyst/shred/pull/220

    Code diff

    https://github.com/amethyst/shred/compare/0.14.0...0.14.1

    Source code(tar.gz)
    Source code(zip)
  • 0.14.0(Jul 12, 2022)

    Changelog

    • Removed dependency hashbrown since it is part of std since Rust 1.36
    • Removed dependency mopa since it is unmaintained and has a potential vulnerability

    PRs

    • Replace Ref{,Mut}::map with safer code by @torkleyy in https://github.com/amethyst/shred/pull/219
    • Eliminate safety issue in Resource::downcast* by @torkleyy in https://github.com/amethyst/shred/pull/218

    Code diff

    https://github.com/amethyst/shred/compare/0.13.0...0.14.0

    Source code(tar.gz)
    Source code(zip)
  • 0.11.0(Dec 21, 2020)

  • 0.4.4(Sep 2, 2017)

    • Allow optional fetching of resources using Option<Fetch> and Option<FetchMut>
    • Allow checking if AsyncDispatcher is running (thanks @Binero for that!)
    Source code(tar.gz)
    Source code(zip)
  • v0.4(Jun 6, 2017)

    This version includes all the important features I planned for shred, has a good level of performance and I consider it pretty stable.

    Documentation | crates.io link

    Additionally, shred-derive has been improved to handle more cases (like where clauses).

    Important PRs since 0.1:

    • #16 Thread-local systems
    • #17 Asynchronous Dispatcher
    • #19 Remove context
    • #20 Remove Debug bound (by @Object905)
    • #23 Improvements to SystemData
    • #24 Rename System::work to System::run
    • #30 More control over dispatching (by @Object905)
    • #31 Major optimization
    • #34 Remove unsafe marker from SystemData functions

    Thanks everybody for helping out, especially @Object905 for your PR and @WaDelma and @kvark for reviewing.

    Source code(tar.gz)
    Source code(zip)
  • 0.1(May 6, 2017)

Owner
Amethyst Foundation
Non-Profit with focus on Rust and the Game Dev ecosystem.
Amethyst Foundation
A Rust library for reading asset files and resource packs for any version of Minecraft

minecraft-assets A Rust library for reading asset files and resource packs for any version of Minecraft. Example use minecraft_assets::api::AssetPack;

Ben Reeves 7 Aug 14, 2022
Simple rust asset handling derive macro for enums, and a proc-macro learning resource!

asset-derive Summary • Todos • Docs Summary Simple Rust asset loading derive macro for Enums, and a resource for learning proc-macros! Please feel fre

Shadorain 5 Feb 9, 2023
Shared cockpit for Microsoft Flight Simulator.

This project is currently on hiatus until further notice. I'd love to continue improving this project to fulfill my vision of a no hassle, fully featu

Connor T 458 Dec 28, 2022
Shared memory - A Rust wrapper around native shared memory for Linux and Windows

shared_memory A crate that allows you to share memory between processes. This crate provides lightweight wrappers around shared memory APIs in an OS a

elast0ny 274 Dec 29, 2022
A fast, low-resource Natural Language Processing and Text Correction library written in Rust.

nlprule A fast, low-resource Natural Language Processing and Error Correction library written in Rust. nlprule implements a rule- and lookup-based app

Benjamin Minixhofer 496 Jan 8, 2023
A Minecraft Java Edition to Bedrock Edition resource pack convertor in Rust

j2be A Minecraft Java Edition to Bedrock Edition resource pack convertor in Rust How to use Note: This project is still in development Run cargo build

Cqdet 11 Sep 15, 2021
📊 Fetch & monitor your server's resource usage through Lua

?? gmsv_serverstat Simple serverside binary module which can expose information about system resource usage to Lua. Installation Download the relevant

William 21 Jul 30, 2022
Quick Pool: High Performance Rust Async Resource Pool

Quick Pool High Performance Rust Async Resource Pool Usage DBCP Database Backend Adapter Version PostgreSQL tokio-postgres qp-postgres Example use asy

Seungjae Park 13 Aug 23, 2022
An in-depth resource to learn Rust 🦀

Learning Rust ?? Hello friend! ?? Welcome to my "Learning Rust" repo, a home for my notes as I'm learning Rust. I'm structuring everything into lesson

Lazar Nikolov 7 Jan 28, 2022
An ability / resource / cooldown management library for Bevy.

About leafwing-abilities is an opinionated, ready-to-use Bevy library and plugin for handling abilities. It features: cooldown tracking resource manag

null 2 Mar 30, 2022
Learn-rust - An in-depth resource to learn Rust 🦀

Learning Rust ?? Hello friend! ?? Welcome to my "Learning Rust" repo, a home for my notes as I'm learning Rust. I'm structuring everything into lesson

Lazar Nikolov 7 Jan 28, 2022
defmt is a highly efficient logging framework that targets resource-constrained devices, like microcontrollers

defmt defmt ("de format", short for "deferred formatting") is a highly efficient logging framework that targets resource-constrained devices, like mic

Knurling 476 Jan 2, 2023
A Rust library for reading asset files and resource packs for any version of Minecraft

minecraft-assets A Rust library for reading asset files and resource packs for any version of Minecraft. Example use minecraft_assets::api::AssetPack;

Ben Reeves 7 Aug 14, 2022
rusty-riscy is a performance testing and system resource monitoring tool written in Rust to benchmark RISC-V processors.

rusty-riscy rusty-riscy is a performance testing and system resource monitoring tool written in Rust to benchmark RISC-V processors. Objectives To cre

Suhas KV 4 May 3, 2022
Android resource file parsing & writing

arsc arsc is a Rust library that provides the ability to parse and write Android resource file (arsc) [dependencies] arsc = "0.1" Compiler support: ru

Yaxin Cheng 7 Dec 25, 2022
Allows deploying modules to Aptos under resource accounts.

Aptos Deployer Module containing helpers for deploying resource accounts. Resource accounts allow the module to sign as itself on-chain, which is usef

Deploy DAO 6 Oct 31, 2022
Simple rust asset handling derive macro for enums, and a proc-macro learning resource!

asset-derive Summary • Todos • Docs Summary Simple Rust asset loading derive macro for Enums, and a resource for learning proc-macros! Please feel fre

Shadorain 5 Feb 9, 2023
A resource monitor in your browser, in Rust

axact A resource monitor in your browser, so you can view the state of a VM or some other remote host. Built with Rust & Preact, see the video: https:

amos 87 Apr 13, 2023
Cross-platform GameMaker extension for getting system information and resource usage

GM Sysinfo Cross-platform GameMaker extension for getting system information and resource usage Table of Contents Table of Contents Examples Display m

SpikeHD 3 Dec 1, 2023