Strictly speaking the issue as titled is not possible. To poll a future it requires &mut self which means synchronized. In other words, only one thread can call Future::poll at a time.

What you may want to do is to create your own custom implementation of Future for this. You'd have to keep track of all tasks waiting for a value to get produced, and then wake them all up once the value is ready. You'd then also have the logic for the future being immediately ready if the value is already there.

Does that make sense?

Yes, as I explained I've done something like this in my code already and it is working.

The question, I guess, is if you would be interested in this being submitted as a pull request for inclusion in the futures library itself, or if I should just go ahead and implement it separately.

Perhaps! Mind if I take a peek at what you've got going on?

Well it's kind of mixed up in the other code, but it's in this file:

https://github.com/wellbehavedsoftware/wbs-backup/blob/6428167c0bfe42144b8d0d42f6e94b597daa33d7/rzbackup/src/zbackup/repo.rs

Here's where I keep a map of "additional" clients waiting for a future to complete, presumably the map key would stay the same but the value would be a struct mapping the desired behaviour:

https://github.com/wellbehavedsoftware/wbs-backup/blob/6428167c0bfe42144b8d0d42f6e94b597daa33d7/rzbackup/src/zbackup/repo.rs#L74

Then in this function, I either create a direct future to produce a result, or I add a new oneshot to the list above. The first one will fire off all the others on completion:

https://github.com/wellbehavedsoftware/wbs-backup/blob/6428167c0bfe42144b8d0d42f6e94b597daa33d7/rzbackup/src/zbackup/repo.rs#L953

The idea is to refactor out this logic into some kind of reusable future. Presumably it could be constructed by consuming an existing future into some kind of other object, which could then hand out regular futures to be used and consumed by waiting as normal.

Here's a simple example of how you might use it for lazy loading:

let lazy_loader_future =
    create_lazy_loader ();

let lazy_loader_multiplexer =
    futures::multiplexer (lazy_loader_future);

// ...

for key in items {

    init_some_module (
        key,
        lazy_loader_multiplexer.spawn_future ());

}

That actually sounds pretty reasonable to me, this looks to be like some form of memoization going on I think? Perhaps something along the lines of:

pub fn cell<T, E>() -> (Cell<T, E>, Complete<T, E>) {
    // ...
}

impl Future for Cell<T, E> {
    type Item = CellComplete<T, E>;
    type Error = Canceled;

    // ...
}

impl<T, E> Clone for Cell<T, E> {
    // ...
}

impl<T, E> Deref for CellComplete<T, E> {
    type Target = Result<T, E>;
    // ...
}

That is, you'd have a oneshot-like interface. One Complete could be used to complete the computation, and otherwise Cell handles (modulo naming) can be used to resolve to a completed value which then implements Deref to the target (abstracting away the backing storage)

Ok, so in your example the Future itself implements Clone, and does the same kind of logic internally. That would work for me as well, although I would still like to be able to produce one of these by consuming a Future. I don't see any reason why both couldn't be supported.

To be honest, I feel like there should be more API support for directly plumbing together Futures and Completes, which would perhaps make this moot, although perhaps it's there and I've missed it. Probably a separate discussion...

I think a more general solution should be written, that will make it easier to wait for a task completion on multiple threads.
What do you think? @alexcrichton

@jamespharaoh I'm not sure I quite understand what you mean by plumbing together futures and completes? In theory that's exactly what oneshot is, but do you perhaps have something else in mind?

@liran-ringel perhaps yeah! Just trying to figure out what that more general solution is :)

(oops didn't mean to close)

@alexcrichton A solution might look like that:
let shared_future = SharedFuture::new(my_future);
let mut f1 = shared_future.future();
let mut f2 = shared_future.future();

@liran-ringel yeah that's actually basically what I was proposing above as well!

I'll put something together for everyone to check out. I have a feeling I can actually go one better than we've discussed but let's see what the borrow checker has to say about that before I promise anything ;-)

@alexcrichton I mean that I feel like Future and Complete are a bit like a producer and a consumer and that they should be able to be plugged together in a simple way and "forgotten" about by the developer. I probably need to come up with some examples, but I'll look at that later. I also may be missing something obvious, and from the work I've done with this library already I know that there are certain things that need to be done to keep the borrow checker happy and ensure that things actually get run.

@jamespharaoh ok! Looking forward to seeing what you come up with

There's more work to be done, but I'd like you @alexcrichton to check if it's the right direction:
master...liran-ringel:feature/multishot

@liran-ringel

Sorry for the delay, but I've now taken a look! I think that's along the right lines semantically but we'll likely want enable us to add a more efficient implementation eventually as well. I wonder if we could perhaps have an interface like:

trait Future {
    // ...

    fn shared(self) -> Shared<Self> {
        // ...
    }
}

struct Shared<F> {
    // ...
}

impl<F> Clone for Shared<F> {
    // ...
}

impl<F> Future for F {
    type Item = SharedItem<F::Item>;
    type Error = SharedError<F::Error>;
    // ...
}

struct SharedItem<T> { /* ... */ }
struct SharedError<T> { /* ... */ }

impl<T> Deref for SharedItem<T> {
    type Target = T;
    // ...
}

impl<T> Deref for SharedError<T> {
    type Target = T;
    // ...
}

That is, we could hide the abstraction of an Arc and also leverage Clone for "create a new reference"

@alexcrichton Before I came with the previous solution I tried to do something similar to the Shared future, but I had issues with synchronization, which would complicate it a lot.

To simplify that, I created a future Multishot that can create oneshots, and will trigger them when the original future is ready. The job of the Multishot future is actually to trigger the other oneshots, so no extra locks are needed.
The downside is that it's less flexible since all of the oneshots can only be created from the same Multishot.

I will try again to implement the Shared-like interface, but first, what did you mean by "a more efficient implementation"?
If you talk about performance, what should I avoid for the next time?

Yes depending on the interface you might have problems with synchronization or not. Right now though the memory requirement is O(number of clones) which seems unfortunate when it should in theory be O(1)?

I've also had a play with this but got a little confused trying to implement my own future. I'm still quite new to Rust and so am still getting used to some of the more complex things.

The "clever" thing I am still hoping to do, however, is to provide some way to simply clone a future, and have it work as expected. I now believe it should be possible to implement Clone for &mut BoxFuture, but would like some feedback before I play around with this.

Firstly, does anyone with more knowledge than me think this is possible? Secondly, do people think this is desirable?

Underneath, the implementation would replace the boxed future with a CloneableFuture, which either uses multishot underneath, or some manual bookkeeping. I think we'd need to add a method to the Future trait, which would consume the Future and returns CloneableFuture. The default implementation would instantiate the CloneableFuture, and the overriden implementation for CloneableFuture would simply return itself.

Of course, the CloneableFuture has other uses of its own, but it's nice of course to try and implement things which hide the details if possible, and having &mut BoxFuture be cloneable sounds very useful and intuitive to me.

I don't think we'll be able to clone arbitrary futures, but cloning a particular future seems reasonable to me. That is, creating a custom future wrapper that's cloneable sounds like it can work.

Other than that sounds reasonable!

Yeah well that was what the &mut was about, but I'm not sure at all it's even possible, for various reasons. To be honest, anyone who needs this feature can simply call a trait method such as cloneable(), probably followed by a call to boxed(), based on my experience with this library so far. I think the box will automatically be Clone based on built in generic trait implementations for a boxed, cloneable future, although again I'm still working out how these things work in rust.

Possibly we could provide transparent cloneability for boxed futures inside some kinds of wrappers, but again, it's just plumbing and not really necessary.

I will have another crack at this when I get a chance. Got a bit confused with the library when I tried it before. I am guessing the complexity is related to the "zero-cost" design goal you discuss in your blog post about the library.

Ok, let's see how far implementing the cloneable function takes us! If you need any help just let me know, and looking at some other combinators may provide insights of implementation methodologies.

@alexcrichton What do you say about that?
master...liran-ringel:feature/shared

I didn't dig too much into the implementation details but at a conceptual level at least looks reasonable to me!

I'm happy this has reached public release, and it has simplified my code a lot. I'm a bit embarrassed I didn't contribute anything concrete, but having used it, I have a suggestion for a further refinement.

You can see my code here:

https://github.com/wellbehavedsoftware/rzbackup/blob/master/src/misc/cloning_shared_future.rs

Basically this transforms a Shared into a CloningSharedFuture by called .cloning (). I realise this would need some modification to conform to the naming and formatting styles of this project, but I think this is going to be a /very/ common use case, and that the general idea is sound.

Probably you could add a .shared_cloning () function to the Future trait to return something like this, then, so long as your items and errors are cloneable (and you're going to have a rough time if they're not in a multi-threaded environment), then you can use them like normal futures, without any need for references.

Let me know what you think and I can do a pull request into futures-rs...

In fact, I wonder if Clone couldn't be implemented for Future in general if both the Item and Error types are Clone....

@jamespharaoh I think a .cloning() method on Shared<F> where F::Item: Clone, F::Error: Clone would make a lot of sense.

Makes sense to me too!

Coming back to this issue I'm actually going to close this now that we have Shared as this issue itself is solved.