does this pr need those .js files or are they more for referencing while this PR is in progress?

@ebebbington WIP, should be TS for when we end this

All committers have signed the CLA.

@Soremwar and i are working on this but are happy for other people to lend a hand!

@ebebbington Looks like we have readable streams!!

.pause and .resume seem to be working fine!

Writable streams are gonna be a b*tch. Duplex needs to inherit from both, but since JavaScript only has single prototypal inheritance it couldn't be done in an easy way. All instance checks for Duplex are for some reason inside Writable, so I'm having a hard time guessing what to yeet and what not.

Nice one @Soremwar !!!!!!! I think i know of an example that uses duplex if that might help?

@ebebbington Duplex will wait for now

I think I got working Writable streams, but we need tests to make sure everything is OK so Duplex can inherit from both safely

import Writable from '../deno/std/node/_stream/writable.js';

const myStream = new Writable();

myStream._write = (chunk, encoding, done) => {
  console.log({
    data: chunk.toString(),
    encoding,
  })
  done();
}

myStream.write("conquer");
myStream.write("and");
myStream.write("devour");
myStream.end();

just to check which modules will the pr implement?

• http
• streams
• any others?

ill try work on getting the http tests setup, or at least placeholders as nothing really works atm lol

Https too if we manage to make http extensible

Anyways if this grows too big we might just land streams for now

Yeah definitely will end up making it extensible, just a matter of finding what methods both would share

Green on readable streams !!

@ebebbington I think I got Duplex working :) But I'm needing some of the examples you said you had earlier to actually test them

@Soremwar It might not be specifically related so it might not actually help, but i just remember seeing something around duplex's: https://github.com/linux-china/rsocket-deno/blob/master/rsocket/DuplexConnection.ts

Good job tho!:)

After many hours of work I've decided to split this PR into pieces so it can be easier to review(over 17000 lines as of now) and convert to TypeScript.

This PR will only address Readable, and Writable streams.
Http progress, duplex, transform, pipeline, promises and others will be moved to other branch were progress will be continued.

For now none of the interfaces will be exposed directly, but through the _streams folder.

This is finally ready to go!!

There are a couple improvements that could be done, but this should be ready for review

Comments resolved

Well pretty sure merging an incompatible implementation of Readable/Writeable with subtle timing differences was a mistake 😅

#7569 (comment)

It'll be good for a "10 things I regret about" talk in a few years though.

@benjamingr there's nothing binding about any of the code here - there is a stable support target which is Node.js support, and breaks to achieve that are fine under the experimental status. Perhaps we unify on a separate package outside of Deno at a later point as a collaborative effort, and I'd be all for that. There are some hard problems there around code linking though that may be clearer in due course. Matteo mentions readable-stream needs work - therefore the more people gaining first-principles experience with this code and problem space, the better, to eventually work towards the best maintainable polyfill approach.

Look Guy - here is my perspective:

• I don't have context here.
• I was pinged to helped because I contributed 10ish PRs to Deno and am happy to help and like Deno.
• I said this is impossible to review.
• I mentioned this PR has merit but has lots of subtle breaking timing bugs here.
• Steven agreed and said he noticed said bugs.
• We talked about a path forward where Steven pulls Node/readable-stream and adds a Deno build.
• Node maintainers all embraced the idea and agreed to help with a stable path forward without the timing bugs.
• Steven opened an issue about this in Node.js/readable-stream
• Steven was doing active work over at https://github.com/nodejs/readable-stream to get a version that's actually working ready - this is ongoing work.
• ....
• Ryan merged the PR

I feel like if Steven was just given another week or two to work on the portable readable-stream bit we wouldn't get a version with lots of subtle timing bugs.

Why would Deno prefer to have a non-working version of Readable/Writeable and not a working one given ongoing work on is happening by Steven in nodejs/readable-stream.

@benjamingr great to see there is work at nodejs/readable-stream#452, I wasn't aware of this. I'm also only tangentially involved - I approved this PR before I was aware of those bugs and your comments. The path above you outlined sounds like a good plan to me, my comment was to say that there's nothing stopping that from still happening. I'm all for ripping this stuff out and better structuring it.

This PR has been open for two months, and merging it allows continuing to progress on stdlib work that relies on streams such as http / https. Otherwise everyone is blocked on Steven's hypothetical two week work, as am I with my own experiments here.

Ideally we'd certainly have a sort of build that does a code deploy from a well-maintained shared library into this deno stdlib though... and steps to work towards that would be beneficial. I was under the impression this PR was close to a manual first attempt at that and that the plan was to have follow-ups to better structure the code.