A Rust serverless function to retrieve and relay a playlist for Twitch livestreams/VODs.

Malloc Voidstar

Last update: Dec 15, 2021

Related tags

Overview

City17

A Rust serverless function to retrieve and relay a playlist for Twitch livestreams/VODs.

By running this in specific countries and using a browser extension to redirect certain network requests to this function, Twitch will not display any ads.

I don't provide any pre-built version of this, you have to set it up yourself.

My rough estimate is that a few hundred to a few thousand users can be supported while staying inside the free tier's limits, but I wouldn't recommend publicizing your function unless you actually understand how the pricing works. If you're the only user you won't need to pay anything. (Supposedly there's no free tier on outbound bandwidth, but my bill in May was 0.000000 USD.)

You can probably run this as an actual server, but I haven't tested that because that costs money.

^{_{(I don't remember why I named this City17.)}}

Requirements

The function needs to be run in a country where Twitch doesn't serve ads. Countries currently known: Russia, China.

I don't know of any serverless provider that operates in Russia, and supports Rust, and allows use by people outside Russia. China has at least two providers that should work fine, one of which I've personally tested. (Some amount of GFW evasion takes place, since Twitch is blocked in China, but at least we can be pretty sure that Twitch won't start running ads there...?)

I've only successfully run this on Aliyun (Alibaba Cloud) from Chinese regions, and the code will require modifications to run on a different provider. (Except Azure if I managed to avoid breaking it after my initial tests there; there's a feature flag.)

Tencent Cloud should also work with some minor modifications, but I've been unable to complete signup to test it.

Azure might work if any of its locations are non-ad countries. Twitch considers the UAE to be located inside the USA, so their Dubai location (UAE North) doesn't work. Azure doesn't allow use of their Chinese regions unless you're a business with a presence in China.

Building

Run build.sh. Cannot be truly built in Windows due to ring, but cargo check and cargo build work for checking the code. Ubuntu 20.04 via WSL works fine and is what I use.

Once city17.zip is built, see setup instructions below.

Requires:

Rust installed via rustup
The x86_64-unknown-linux-musl target for Rust (rustup target add x86_64-unknown-linux-musl)
7za (p7zip-full on Debian and Ubuntu, can modify script to use normal zip command)
musl-gcc (musl-tools)
Probably build-essential

Aliyun setup instructions

Sign up, enable function compute, etc. (I'm not creating a new account just to write down all the steps)
Set the region to China (Shanghai). As far as I know this is the region closest to Tokyo, which is where one of Twitch's servers is.
In the Function Compute menu, enter Services and Functions and create a service named a. (Or whatever you want, but you'll need to modify the code.)
Create a function:
- HTTP
- Named prx
- Custom Runtime
- Select the ZIP file city17.zip
- Give it 128MB of RAM and a 15-second timeout.

For future updates of the function, use the Code tab's "Upload Zip File". (Or get fcli, set it up, and run update.sh after build.sh)

Under the Code tab, use the URL listed there to set up the browser extension.

It looks like https://################.cn-shanghai.fc.aliyuncs.com/2016-08-15/proxy/a/prx/; you will need to add invoke to the end.

Issues

If the shell scripts fail due to having Windows line endings, run

dos2unix build.sh
dos2unix update.sh

in your Linux shell. (This shouldn't happen.)

Extra reading

The streamlink Twitch plugin has all the info needed in order to learn how to connect to Twitch and get the M3U8 playlist. Some of it doesn't quite match what Twitch is doing now, so watch a Twitch stream normally and look at the network requests your browser makes.

License

GNU GPLv3.

Comments

Bump openssl-src from 111.15.0+1.1.1k to 111.20.0+1.1.1o
Bumps openssl-src from 111.15.0+1.1.1k to 111.20.0+1.1.1o.

Commits

See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies
opened by dependabot[bot] 1
Switch to reqwest and accept any hostname for certificates

Fixes #1.

Makes it trivially easy to MITM the function, but it was specifically designed not to need any of your Twitch account's data so the worst thing that can happen is watching a different stream.

opened by AlyoshaVasilieva 0
Bump openssl-src from 111.15.0+1.1.1k to 111.22.0+1.1.1q
Bumps openssl-src from 111.15.0+1.1.1k to 111.22.0+1.1.1q.

Commits

See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies
opened by dependabot[bot] 0
Bump hyper from 0.14.11 to 0.14.19
Bumps hyper from 0.14.11 to 0.14.19.

Release notes

Sourced from hyper's releases.

v0.14.19

Bug Fixes

http1: fix preserving header case without enabling ffi (#2820) (6a35c175)

server: don't add implicit content-length to HEAD responses (#2836) (67b73138)

Features

server:

add Connection::http2_max_header_list_size option (#2828) (a32658c1, closes #2826)

add AddrStream::local_addr() (#2816) (ffbf610b, closes #2773)

Breaking Changes

ffi (unstable):

hyper_clientconn_options_new no longer sets the http1_preserve_header_case connection option by default. Users should now call hyper_clientconn_options_set_preserve_header_case if they desire that functionality. (78de8914)

New Contributors ❤️

@SabrinaJewson made their first contribution in hyperium/hyper#2775

@Some-Dood made their first contribution in hyperium/hyper#2812

@liamwarfield made their first contribution in hyperium/hyper#2798

@cuishuang made their first contribution in hyperium/hyper#2818

@jannes made their first contribution in hyperium/hyper#2836

@silence-coding made their first contribution in hyperium/hyper#2828

@nylonicious made their first contribution in hyperium/hyper#2762

@ryanrussell made their first contribution in hyperium/hyper#2875

v0.14.18

Bug Fixes

ffi: don't build C libraries by default (1c663706)

Features

client: add HttpInfo::local_addr() method (055b4e7e, closes #2767)

New Contributors

@ilyatrefilov made their first contribution in hyperium/hyper#2760

@stephank made their first contribution in hyperium/hyper#2765

@kubuzetto made their first contribution in hyperium/hyper#2768

@lqd made their first contribution in hyperium/hyper#2770

v0.14.17

Bug Fixes

... (truncated)

Changelog

Sourced from hyper's changelog.

v0.14.19 (2022-05-27)

Bug Fixes

http1: fix preserving header case without enabling ffi (#2820) (6a35c175)

server: don't add implicit content-length to HEAD responses (#2836) (67b73138)

Features

server:

add Connection::http2_max_header_list_size option (#2828) (a32658c1, closes #2826)

add AddrStream::local_addr() (#2816) (ffbf610b, closes #2773)

Breaking Changes

ffi (unstable):

hyper_clientconn_options_new no longer sets the http1_preserve_header_case connection option by default. Users should now call hyper_clientconn_options_set_preserve_header_case if they desire that functionality. (78de8914)

v0.14.18 (2022-03-22)

Bug Fixes

ffi: don't build C libraries by default (1c663706)

Features

client: add HttpInfo::local_addr() method (055b4e7e, closes #2767)

v0.14.17 (2022-02-10)

Bug Fixes

client: avoid panics in uses of Instant (#2746) (dcdd6d10)

Features

client: implement the HTTP/2 extended CONNECT protocol from RFC 8441 (#2682) (5ec094ca)

error: add Error::message (#2737) (6932896a, closes #2732)

http1: implement obsolete line folding (#2734) (1f0c177b)

... (truncated)

Commits

f8e2a83 v0.14.19

a929df8 docs(various): fix typos in VISION and ROADMAP (#2875)

3a755a6 chore(lib): update tokio-util to 0.7 (#2762)

4678be9 docs(contrib): add guide for Triaging Issues

775fac1 docs(lib): propose 1.0 roadmap (#2806)

a32658c feat(server): add Connection::http2_max_header_list_size option (#2828)

67b7313 fix(server): don't add implicit content-length to HEAD responses (#2836)

faf24c6 refactor(http1): assorted code readability improvements in h1/conn.rs (#2817)

6a35c17 fix(http1): fix preserving header case without enabling ffi (#2820)

89598df docs(lib): fix some typos (#2818)

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies
opened by dependabot[bot] 0
Bump tokio from 1.9.0 to 1.16.1
Bumps tokio from 1.9.0 to 1.16.1.

Release notes

Sourced from tokio's releases.

Tokio v1.16.1

1.16.1 (January 28, 2022)

This release fixes a bug in #4428 with the change #4437.

#4428: tokio-rs/tokio#4428 #4437: tokio-rs/tokio#4437

Tokio v1.16.0

Fixes a soundness bug in io::Take (#4428). The unsoundness is exposed when leaking memory in the given AsyncRead implementation and then overwriting the supplied buffer:

impl AsyncRead for Buggy { fn poll_read( self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut ReadBuf<'_> ) -> Poll<Result<()>> { let new_buf = vec![0; 5].leak(); *buf = ReadBuf::new(new_buf); buf.put_slice(b"hello"); Poll::Ready(Ok(())) } }

Also, this release includes improvements to the multi-threaded scheduler that can increase throughput by up to 20% in some cases (#4383).

Fixed

io: soundness don't expose uninitialized memory when using io::Take in edge case (#4428)

fs: ensure File::write results in a write syscall when the runtime shuts down (#4316)

process: drop pipe after child exits in wait_with_output (#4315)

rt: improve error message when spawning a thread fails (#4398)

rt: reduce false-positive thread wakups in the multi-threaded scheduler (#4383)

sync: don't inherit Send from parking_lot::*Guard (#4359)

Added

net: TcpSocket::linger() and set_linger() (#4324)

net: impl UnwindSafe for socket types (#4384)

rt: impl UnwindSafe for JoinHandle (#4418)

sync: watch::Receiver::has_changed() (#4342)

sync: oneshot::Receiver::blocking_recv() (#4334)

sync: RwLock blocking operations (#4425)

Unstable

... (truncated)

Commits

91b9850 chore: prepare Tokio v1.16.1 release (#4438)

3c46705 io: fix take pointer check (#4437)

afd2189 chore: prepare Tokio v1.16 release (#4431)

986b88b chore: update year in LICENSE files (#4429)

257053e util: add spawn_pinned (#3370)

5af9e0d sync: add blocking lock methods to RwLock (#4425)

8f77ee8 net: add generic trait to combine UnixListener and TcpListener (#4385)

2747043 tests: enable running wasm32-unknown-unknown tests (#4421)

2a5071f feat: implement Framed::map_codec (#4427)

621790e io: fix take when using evil reader (#4428)

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies
opened by dependabot[bot] 0
Unreliable GFW evasion

The current version of the code uses a now-broken workaround to access the GQL API from inside China. Fixing that shows that Twitch appears to be blocking access to the GQL API, at least the way I use it from China. (Or the Great Firewall is blocking it.)

Working on a resolution.

opened by AlyoshaVasilieva 2