An implementation of a mirror SponsorBlock server in Rust.

Related tags

Command-line rust sponsorblock
Overview

sponsorblock-mirror

This is a Rust-based mirror of the SponsorBlock API.

It also uses sb-mirror for mirroring the CSV dumps via rsync.

Instances

Feel free to add your instance to this list by making a pull request.

You can also configure Piped-Backend to use your mirror by changing the SPONSORBLOCK_SERVERS configuration value.

Compatibility

This implementation does not implement the full SponsorBlock server API. It supports hash-based queries to /api/skipSegments/<hash>, with optional categories parameter, and queries to /api/skipSegments with required videoID and optional categories parameters.

The browser extension works with only the hash-based query endpoint, but other clients, such as the one in ReVanced, require the video ID endpoint, and additionally query /api/userInfo and /api/isUserVip. Right now there are stub implementations for these. ReVanced had not yet been verified as compatible.

Using with Docker Compose

To run the server under Docker Compose, run:

docker compose up

This starts the API server, a database, and a mirroring service to download the SponsorBlock data from the sponsorblock.kavin.rocks mirror and keep it up to date.

The API will be available on http://localhost:8000. For example, you can try http://localhost:8000/api/skipSegments/aabf or http://localhost:8000/api/skipSegments?videoID=eQ_8F4nzyiw. It will take a few minutes at least for the database to download and import, so these will not return data on the first run.

Building

To make a local release build, use cargo build --release. This will produce a binary in target/release/sponsorblock-mirror.

To make a Docker container, you need to do a BuildKit Docker build, not a normal Docker build. Make sure you have buildx available in your Docker, and run:

docker buildx build --load -t 1337kavin/sponsorblock-mirror .

Troubleshooting

  • If the linker complains about a missing -lpq, make sure you have the PostgreSQL development libraries, which may be in a libpq-dev package or your distribution's equivalent.

  • If Docker complains that the --mount option requires BuildKit, make sure you are building with docker buildx build and not docker build.

  • To access the PostgreSQL database directly, you can docker exec -ti postgres-sb-mirror bash -c 'psql $POSTGRES_DB $POSTGRES_USER'.

  • Requests for videos not in the database are forwarded to https://sponsor.ajay.app/, which may be down or malfunctioning. A response of the string Internal Server Error is likely to be from there, rather than from this application.

Comments
  • pg fails to import data on initial docker compose

    pg fails to import data on initial docker compose

    running clean clone using docker compose on debian 11 fails to import data

    [+] Running 3/0
 β Ώ Container postgres-sb-mirror   Created                                                                                                    0.0s
 β Ώ Container sb-mirror            Created                                                                                                    0.0s
 β Ώ Container sponsorblock-mirror  Created                                                                                                    0.0s
Attaching to postgres-sb-mirror, sb-mirror, sponsorblock-mirror
sb-mirror            | Uses SponsorBlock data from https://sponsor.ajay.app/
postgres-sb-mirror   |
postgres-sb-mirror   | PostgreSQL Database directory appears to contain a database; Skipping initialization
postgres-sb-mirror   |
sb-mirror            | Downloading from mirror: sponsorblock.kavin.rocks
postgres-sb-mirror   | 2022-10-27 16:31:33.147 UTC [1] LOG:  starting PostgreSQL 15.0 on x86_64-pc-linux-musl, compiled by gcc (Alpine 11.2.1_git20220219) 11.2.1 20220219, 64-bit
postgres-sb-mirror   | 2022-10-27 16:31:33.147 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
postgres-sb-mirror   | 2022-10-27 16:31:33.147 UTC [1] LOG:  listening on IPv6 address "::", port 5432
postgres-sb-mirror   | 2022-10-27 16:31:33.162 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres-sb-mirror   | 2022-10-27 16:31:33.173 UTC [24] LOG:  database system was shut down at 2022-10-27 16:30:07 UTC
postgres-sb-mirror   | 2022-10-27 16:31:33.180 UTC [1] LOG:  database system is ready to accept connections
sponsorblock-mirror  | Importing database...
postgres-sb-mirror   | 2022-10-27 16:31:34.040 UTC [35] ERROR:  missing data for column "startTime"
postgres-sb-mirror   | 2022-10-27 16:31:34.040 UTC [35] CONTEXT:  COPY sponsorTimesTemp, line 2: "/*!"
postgres-sb-mirror   | 2022-10-27 16:31:34.040 UTC [35] STATEMENT:  BEGIN; DROP TABLE IF EXISTS "sponsorTimesTemp"; CREATE UNLOGGED TABLE "sponsorTimesTemp"(LIKE "sponsorTimes" INCLUDING defaults INCLUDING constraints INCLUDING indexes); COPY "sponsorTimesTemp" FROM '/mirror/sponsorTimes.csv' DELIMITER ',' CSV HEADER; DROP TABLE "sponsorTimes"; ALTER TABLE "sponsorTimesTemp" RENAME TO "sponsorTimes"; COMMIT;
sponsorblock-mirror  | Imported database in 55ms
sponsorblock-mirror  | Failed to import database: missing data for column "startTime"
postgres-sb-mirror   | 2022-10-27 16:31:34.043 UTC [35] ERROR:  current transaction is aborted, commands ignored until end of transaction block
postgres-sb-mirror   | 2022-10-27 16:31:34.043 UTC [35] STATEMENT:  VACUUM "sponsorTimes";
sponsorblock-mirror  | Failed to vacuum database: current transaction is aborted, commands ignored until end of transaction block
sb-mirror            | receiving incremental file list
sb-mirror            | ./
sb-mirror            | mirror.sb.mchang.xyz/
sb-mirror            | mirror.sb.mchang.xyz/categoryVotes.csv
     24,827,717 100%  124.62MB/s    0:00:00 (xfr#1, to-chk=17/29)
sb-mirror            | mirror.sb.mchang.xyz/index.html
         17,694 100%   90.94kB/s    0:00:00 (xfr#2, to-chk=16/29)
sb-mirror            | mirror.sb.mchang.xyz/index.html?sort=name&order=asc
         17,967 100%   92.35kB/s    0:00:00 (xfr#3, to-chk=15/29)
sb-mirror            | mirror.sb.mchang.xyz/index.html?sort=name&order=desc
         17,968 100%   92.35kB/s    0:00:00 (xfr#4, to-chk=14/29)
sb-mirror            | mirror.sb.mchang.xyz/index.html?sort=namedirfirst&order=asc
         17,694 100%   90.47kB/s    0:00:00 (xfr#5, to-chk=13/29)
sb-mirror            | mirror.sb.mchang.xyz/index.html?sort=namedirfirst&order=desc
         17,695 100%   90.47kB/s    0:00:00 (xfr#6, to-chk=12/29)
sb-mirror            | mirror.sb.mchang.xyz/index.html?sort=size&order=asc
         17,967 100%   91.38kB/s    0:00:00 (xfr#7, to-chk=11/29)
sb-mirror            | mirror.sb.mchang.xyz/index.html?sort=size&order=desc
         17,968 100%   91.39kB/s    0:00:00 (xfr#8, to-chk=10/29)
sb-mirror            | mirror.sb.mchang.xyz/index.html?sort=time&order=asc
         17,967 100%   91.38kB/s    0:00:00 (xfr#9, to-chk=9/29)
sb-mirror            | mirror.sb.mchang.xyz/index.html?sort=time&order=desc
         17,968 100%   91.39kB/s    0:00:00 (xfr#10, to-chk=8/29)
sb-mirror            | mirror.sb.mchang.xyz/lockCategories.csv
     15,347,300 100%   54.82MB/s    0:00:00 (xfr#11, to-chk=7/29)
sb-mirror            | mirror.sb.mchang.xyz/ratings.csv
        924,650 100%    3.19MB/s    0:00:00 (xfr#12, to-chk=6/29)
sb-mirror            | mirror.sb.mchang.xyz/sponsorTimes.csv
  3,797,080,277 100%  135.07MB/s    0:00:26 (xfr#13, to-chk=5/29)
sb-mirror            | mirror.sb.mchang.xyz/unlistedVideos.csv
      5,729,807 100%    6.23MB/s    0:00:00 (xfr#14, to-chk=4/29)
sb-mirror            | mirror.sb.mchang.xyz/userNames.csv
     13,941,428 100%   13.32MB/s    0:00:00 (xfr#15, to-chk=3/29)
sb-mirror            | mirror.sb.mchang.xyz/videoInfo.csv
    401,983,079 100%   89.01MB/s    0:00:04 (xfr#16, to-chk=2/29)
sb-mirror            | mirror.sb.mchang.xyz/vipUsers.csv
          6,117 100%   19.21kB/s    0:00:00 (xfr#17, to-chk=1/29)
sb-mirror            | mirror.sb.mchang.xyz/warnings.csv
        454,524 100%    1.38MB/s    0:00:00 (xfr#18, to-chk=0/29)
sb-mirror            |
sb-mirror            | sent 388 bytes  received 2,554,704,353 bytes  47,751,490.49 bytes/sec
sb-mirror            | total size is 4,260,906,327  speedup is 1.67
sb-mirror            | Starting rsync daemon
sb-mirror            | Downloading from mirror: sponsorblock.kavin.rocks
sb-mirror            | Downloading from mirror: sponsorblock.kavin.rocks
sb-mirror            | receiving incremental file list
sb-mirror            | receiving incremental file list
sb-mirror            | categoryVotes.csv
         50,060 100%   47.74MB/s    0:00:00 (xfr#1, to-chk=27/29)
sb-mirror            | ./
sb-mirror            | categoryVotes.csv
         50,060 100%   47.74MB/s    0:00:00 (xfr#1, to-chk=27/29)
sb-mirror            | lockCategories.csv
sb-mirror            | lockCategories.csv
         50,060 100%   47.74MB/s    0:00:00 (xfr#2, to-chk=26/29)
sb-mirror            | ratings.csv
         50,060 100%   47.74MB/s    0:00:00 (xfr#3, to-chk=25/29)
sb-mirror            | sponsorTimes.csv
         50,060 100%   47.74MB/s    0:00:00 (xfr#4, to-chk=24/29)
sb-mirror            | unlistedVideos.csv
         50,060 100%   23.87MB/s    0:00:00 (xfr#2, to-chk=26/29)
         50,060 100%   47.74MB/s    0:00:00 (xfr#5, to-chk=23/29)
sb-mirror            | userNames.csv
         50,060 100%   47.74MB/s    0:00:00 (xfr#6, to-chk=22/29)
sb-mirror            | ratings.csv
sb-mirror            | videoInfo.csv
         50,060 100%   47.74MB/s    0:00:00 (xfr#7, to-chk=21/29)
sb-mirror            | vipUsers.csv
         50,060 100%   23.87MB/s    0:00:00 (xfr#8, to-chk=20/29)
sb-mirror            | warnings.csv
         50,060 100%   15.91MB/s    0:00:00 (xfr#3, to-chk=25/29)
         50,060 100%   23.87MB/s    0:00:00 (xfr#9, to-chk=19/29)
sb-mirror            | sponsorTimes.csv
         50,060 100%   15.91MB/s    0:00:00 (xfr#4, to-chk=24/29)
sb-mirror            | unlistedVideos.csv
         50,060 100%   11.94MB/s    0:00:00 (xfr#5, to-chk=23/29)
sb-mirror            | userNames.csv
         50,060 100%   11.94MB/s    0:00:00 (xfr#6, to-chk=22/29)
sb-mirror            | videoInfo.csv
         50,060 100%   11.94MB/s    0:00:00 (xfr#7, to-chk=21/29)
sb-mirror            | vipUsers.csv
         50,060 100%    9.55MB/s    0:00:00 (xfr#8, to-chk=20/29)
sb-mirror            | warnings.csv
         50,060 100%    9.55MB/s    0:00:00 (xfr#9, to-chk=19/29)
sponsorblock-mirror  | Importing database...
postgres-sb-mirror   | 2022-10-27 16:35:34.046 UTC [35] ERROR:  current transaction is aborted, commands ignored until end of transaction block
postgres-sb-mirror   | 2022-10-27 16:35:34.046 UTC [35] STATEMENT:  BEGIN; DROP TABLE IF EXISTS "sponsorTimesTemp"; CREATE UNLOGGED TABLE "sponsorTimesTemp"(LIKE "sponsorTimes" INCLUDING defaults INCLUDING constraints INCLUDING indexes); COPY "sponsorTimesTemp" FROM '/mirror/sponsorTimes.csv' DELIMITER ',' CSV HEADER; DROP TABLE "sponsorTimes"; ALTER TABLE "sponsorTimesTemp" RENAME TO "sponsorTimes"; COMMIT;
postgres-sb-mirror   | 2022-10-27 16:35:34.047 UTC [35] ERROR:  current transaction is aborted, commands ignored until end of transaction block
postgres-sb-mirror   | 2022-10-27 16:35:34.047 UTC [35] STATEMENT:  VACUUM "sponsorTimes";
sponsorblock-mirror  | Imported database in 0ms
sponsorblock-mirror  | Failed to import database: current transaction is aborted, commands ignored until end of transaction block
sponsorblock-mirror  | Failed to vacuum database: current transaction is aborted, commands ignored until end of transaction block
sb-mirror            |
sb-mirror            |
sb-mirror            | sent 4,095 bytes  received 12,867 bytes  389.93 bytes/sec
sb-mirror            | total size is 4,260,906,327  speedup is 251,203.06
sb-mirror            | sent 4,098 bytes  received 12,870 bytes  390.07 bytes/sec
sb-mirror            | total size is 4,260,906,327  speedup is 251,114.23
postgres-sb-mirror   | 2022-10-27 16:36:33.272 UTC [22] LOG:  checkpoint starting: time
postgres-sb-mirror   | 2022-10-27 16:36:36.726 UTC [22] LOG:  checkpoint complete: wrote 37 buffers (0.2%); 0 WAL file(s) added, 0 removed, 0 recycled; write=3.423 s, sync=0.009 s, total=3.455 s; sync files=28, longest=0.005 s, average=0.001 s; distance=164 kB, estimate=164 kB
    opened by pulseone-cp 4
  • Update Rust crate serde to 1.0.148

    Update Rust crate serde to 1.0.148

    Mend Renovate

    This PR contains the following updates:

    | Package | Type | Update | Change | |---|---|---|---| | serde (source) | dependencies | patch | 1.0.147 -> 1.0.148 |

    Release Notes

    serde-rs/serde

    v1.0.148

    Compare Source

    • Support remote derive for generic types that have private fields (#​2327)

    Configuration

    πŸ“… Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

    🚦 Automerge: Enabled.

    β™» Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

    πŸ”• Ignore: Close this PR and you won't be reminded about this update again.

    • [ ] If you want to rebase/retry this PR, check this box

    This PR has been generated by Mend Renovate. View repository job log here.

    opened by renovate[bot] 1
  • [Request] ARM64 image

    [Request] ARM64 image

    Any possibility to get arm64 image built in 1337kavin/sponsorblock-mirror? Seems like it's already built with buildx so adding arm64 shouldnt be too difficult

    opened by Roki100 1
  • fix(deps): update rust crate serde_json to 1.0.93

    fix(deps): update rust crate serde_json to 1.0.93

    Mend Renovate

    This PR contains the following updates:

    | Package | Type | Update | Change | |---|---|---|---| | serde_json | dependencies | patch | 1.0.92 -> 1.0.93 |

    Release Notes

    serde-rs/json

    v1.0.93

    Compare Source

    • Support 128-bit integers in serde_json::to_value (#​982)

    Configuration

    πŸ“… Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

    🚦 Automerge: Enabled.

    β™» Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

    πŸ”• Ignore: Close this PR and you won't be reminded about this update again.

    • [ ] If you want to rebase/retry this PR, check this box

    This PR has been generated by Mend Renovate. View repository job log here.

    opened by renovate[bot] 0
  • fix(deps): update rust crate serde_json to 1.0.92

    fix(deps): update rust crate serde_json to 1.0.92

    Mend Renovate

    This PR contains the following updates:

    | Package | Type | Update | Change | |---|---|---|---| | serde_json | dependencies | patch | 1.0.91 -> 1.0.92 |

    Release Notes

    serde-rs/json

    v1.0.92

    Compare Source

    • Documentation improvements

    Configuration

    πŸ“… Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

    🚦 Automerge: Enabled.

    β™» Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

    πŸ”• Ignore: Close this PR and you won't be reminded about this update again.

    • [ ] If you want to rebase/retry this PR, check this box

    This PR has been generated by Mend Renovate. View repository job log here.

    opened by renovate[bot] 0
  • fix(deps): update rust crate diesel to 2.0.3

    fix(deps): update rust crate diesel to 2.0.3

    Mend Renovate

    This PR contains the following updates:

    | Package | Type | Update | Change | |---|---|---|---| | diesel (source) | dependencies | patch | 2.0.2 -> 2.0.3 |

    Release Notes

    diesel-rs/diesel

    v2.0.3: Diesel 2.0.3

    Compare Source

    Fixed

    • Fixed a bug with our transaction manager implementation that caused by marking transactions as broken which could be recovered.
    • Fixed an issue with the combination of BoxableExpression and order clauses

    Configuration

    πŸ“… Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

    🚦 Automerge: Enabled.

    β™» Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

    πŸ”• Ignore: Close this PR and you won't be reminded about this update again.

    • [ ] If you want to rebase/retry this PR, check this box

    This PR has been generated by Mend Renovate. View repository job log here.

    opened by renovate[bot] 0
  • fix(deps): update rust crate reqwest to 0.11.14

    fix(deps): update rust crate reqwest to 0.11.14

    Mend Renovate

    This PR contains the following updates:

    | Package | Type | Update | Change | |---|---|---|---| | reqwest | dependencies | patch | 0.11.13 -> 0.11.14 |

    Release Notes

    seanmonstar/reqwest

    v0.11.14

    Compare Source

    • Adds Proxy::no_proxy(url) that works like the NO_PROXY environment variable.
    • Adds multipart::Part::headers(headers) method to add custom headers.
    • (wasm) Add Response::bytes_stream().
    • Perf: several internal optimizations reducing copies and memory allocations.

    Configuration

    πŸ“… Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

    🚦 Automerge: Enabled.

    β™» Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

    πŸ”• Ignore: Close this PR and you won't be reminded about this update again.

    • [ ] If you want to rebase/retry this PR, check this box

    This PR has been generated by Mend Renovate. View repository job log here.

    opened by renovate[bot] 0
  • fix(deps): update rust crate regex to 1.7.1

    fix(deps): update rust crate regex to 1.7.1

    Mend Renovate

    This PR contains the following updates:

    | Package | Type | Update | Change | |---|---|---|---| | regex | dependencies | patch | 1.7.0 -> 1.7.1 |

    Release Notes

    rust-lang/regex

    v1.7.1

    Compare Source

    ================== This release was done principally to try and fix the doc.rs rendering for the regex crate.

    Performance improvements:

    • PERF #​930: Optimize replacen. This also applies to replace, but not replace_all.

    Bug fixes:

    • BUG #​945: Maybe fix rustdoc rendering by just bumping a new release?

    Configuration

    πŸ“… Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

    🚦 Automerge: Enabled.

    β™» Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

    πŸ”• Ignore: Close this PR and you won't be reminded about this update again.

    • [ ] If you want to rebase/retry this PR, check this box

    This PR has been generated by Mend Renovate. View repository job log here.

    opened by renovate[bot] 0
  • fix(deps): update rust crate serde to 1.0.152

    fix(deps): update rust crate serde to 1.0.152

    Mend Renovate

    This PR contains the following updates:

    | Package | Type | Update | Change | |---|---|---|---| | serde (source) | dependencies | patch | 1.0.151 -> 1.0.152 |

    Release Notes

    serde-rs/serde

    v1.0.152

    Compare Source

    • Documentation improvements

    Configuration

    πŸ“… Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

    🚦 Automerge: Enabled.

    β™» Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

    πŸ”• Ignore: Close this PR and you won't be reminded about this update again.

    • [ ] If you want to rebase/retry this PR, check this box

    This PR has been generated by Mend Renovate. View repository job log here.

    opened by renovate[bot] 0
  • fix(deps): update rust crate num_cpus to 1.15.0

    fix(deps): update rust crate num_cpus to 1.15.0

    Mend Renovate

    This PR contains the following updates:

    | Package | Type | Update | Change | |---|---|---|---| | num_cpus | dependencies | minor | 1.14.0 -> 1.15.0 |

    Release Notes

    seanmonstar/num_cpus

    v1.15.0

    Compare Source

    Fixes
    • update hermit-abi

    Configuration

    πŸ“… Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

    🚦 Automerge: Enabled.

    β™» Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

    πŸ”• Ignore: Close this PR and you won't be reminded about this update again.

    • [ ] If you want to rebase/retry this PR, check this box

    This PR has been generated by Mend Renovate. View repository job log here.

    opened by renovate[bot] 0
  • fix(deps): update rust crate serde_json to 1.0.91

    fix(deps): update rust crate serde_json to 1.0.91

    Mend Renovate

    This PR contains the following updates:

    | Package | Type | Update | Change | |---|---|---|---| | serde_json | dependencies | patch | 1.0.90 -> 1.0.91 |

    Release Notes

    serde-rs/json

    v1.0.91

    Compare Source

    Configuration

    πŸ“… Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

    🚦 Automerge: Enabled.

    β™» Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

    πŸ”• Ignore: Close this PR and you won't be reminded about this update again.

    • [ ] If you want to rebase/retry this PR, check this box

    This PR has been generated by Mend Renovate. View repository job log here.

    opened by renovate[bot] 0
  • Failed to import database: missing data for column

    Failed to import database: missing data for column "views"

    From a clean clone:

    $ sudo docker compose up
[+] Running 4/0
 β Ώ Network sponsorblock-mirror_default  Created                                                                                                                               0.0s
 β Ώ Container postgres-sb-mirror         Created                                                                                                                               0.0s
 β Ώ Container sb-mirror                  Created                                                                                                                               0.0s
 β Ώ Container sponsorblock-mirror        Created                                                                                                                               0.0s
Attaching to postgres-sb-mirror, sb-mirror, sponsorblock-mirror
postgres-sb-mirror   |
postgres-sb-mirror   | PostgreSQL Database directory appears to contain a database; Skipping initialization
postgres-sb-mirror   |
postgres-sb-mirror   | 2023-02-08 23:48:23.986 UTC [1] LOG:  starting PostgreSQL 15.1 on x86_64-pc-linux-musl, compiled by gcc (Alpine 11.2.1_git20220219) 11.2.1 20220219, 64-bit
postgres-sb-mirror   | 2023-02-08 23:48:23.986 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
postgres-sb-mirror   | 2023-02-08 23:48:23.986 UTC [1] LOG:  listening on IPv6 address "::", port 5432
postgres-sb-mirror   | 2023-02-08 23:48:23.987 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres-sb-mirror   | 2023-02-08 23:48:23.990 UTC [23] LOG:  database system was shut down at 2022-11-29 13:32:08 UTC
postgres-sb-mirror   | 2023-02-08 23:48:23.994 UTC [1] LOG:  database system is ready to accept connections
sb-mirror            | Uses SponsorBlock data from https://sponsor.ajay.app/
sb-mirror            | Downloading from mirror: sponsorblock.kavin.rocks
sb-mirror            | receiving incremental file list
sb-mirror            | ./
sb-mirror            | categoryVotes.csv
     30,177,893 100%   18.12MB/s    0:00:01 (xfr#1, to-chk=8/10)
sb-mirror            | lockCategories.csv
     18,444,297 100%   24.46MB/s    0:00:00 (xfr#2, to-chk=7/10)
sb-mirror            | ratings.csv
        924,650 100%    1.21MB/s    0:00:00 (xfr#3, to-chk=6/10)
sb-mirror            | sponsorTimes.csv
    435,421,184 100%   28.53MB/s    0:00:14 (xfr#4, to-chk=5/10)
sb-mirror            | unlistedVideos.csv
      6,468,526 100%   11.82MB/s    0:00:00 (xfr#5, to-chk=4/10)
sb-mirror            | userNames.csv
     17,156,024 100%   14.33MB/s    0:00:01 (xfr#6, to-chk=3/10)
sb-mirror            | videoInfo.csv
sponsorblock-mirror  | Importing database...
    476,180,585 100%   25.84MB/s    0:00:17 (xfr#7, to-chk=2/10)
sb-mirror            | vipUsers.csv
          6,637 100%   14.00kB/s    0:00:00 (xfr#8, to-chk=1/10)
sb-mirror            | warnings.csv
        577,930 100%    1.01MB/s    0:00:00 (xfr#9, to-chk=0/10)
sb-mirror            |
sb-mirror            | sent 209 bytes  received 746,827,143 bytes  18,907,021.57 bytes/sec
sb-mirror            | total size is 985,357,726  speedup is 1.32
sb-mirror            | Starting rsync daemon
postgres-sb-mirror   | 2023-02-08 23:49:26.992 UTC [46] ERROR:  missing data for column "views"
postgres-sb-mirror   | 2023-02-08 23:49:26.992 UTC [46] CONTEXT:  COPY sponsorTimesTemp, line 1548497: "anr6QUZ0mSY,532.5612,537.316,0,0,1,e8d0e34783b3c0adec456bda5eda5cd76ee06b1db4b5b9b4570c4c85aad3cb4f,..."
postgres-sb-mirror   | 2023-02-08 23:49:26.992 UTC [46] STATEMENT:  BEGIN; DROP TABLE IF EXISTS "sponsorTimesTemp"; CREATE UNLOGGED TABLE "sponsorTimesTemp"(LIKE "sponsorTimes" INCLUDING defaults INCLUDING constraints INCLUDING indexes); COPY "sponsorTimesTemp" FROM '/mirror/sponsorTimes.csv' DELIMITER ',' CSV HEADER; DROP TABLE "sponsorTimes"; ALTER TABLE "sponsorTimesTemp" RENAME TO "sponsorTimes"; COMMIT;
sponsorblock-mirror  | Failed to import database: missing data for column "views"
sb-mirror            | Downloading from mirror: sponsorblock.kavin.rocks
sb-mirror            | @ERROR: max connections (1) reached -- try again later
sb-mirror            | rsync error: error starting client-server protocol (code 5) at main.c(1837) [Receiver=3.2.5]
sponsorblock-mirror  | Importing database...
postgres-sb-mirror   | 2023-02-08 23:50:59.514 UTC [46] ERROR:  missing data for column "views"
postgres-sb-mirror   | 2023-02-08 23:50:59.514 UTC [46] CONTEXT:  COPY sponsorTimesTemp, line 1548497: "anr6QUZ0mSY,532.5612,537.316,0,0,1,e8d0e34783b3c0adec456bda5eda5cd76ee06b1db4b5b9b4570c4c85aad3cb4f,..."
postgres-sb-mirror   | 2023-02-08 23:50:59.514 UTC [46] STATEMENT:  BEGIN; DROP TABLE IF EXISTS "sponsorTimesTemp"; CREATE UNLOGGED TABLE "sponsorTimesTemp"(LIKE "sponsorTimes" INCLUDING defaults INCLUDING constraints INCLUDING indexes); COPY "sponsorTimesTemp" FROM '/mirror/sponsorTimes.csv' DELIMITER ',' CSV HEADER; DROP TABLE "sponsorTimes"; ALTER TABLE "sponsorTimesTemp" RENAME TO "sponsorTimes"; COMMIT;
sponsorblock-mirror  | Failed to import database: missing data for column "views"
sponsorblock-mirror  | Importing database...
postgres-sb-mirror   | 2023-02-08 23:52:31.515 UTC [46] ERROR:  missing data for column "views"
postgres-sb-mirror   | 2023-02-08 23:52:31.515 UTC [46] CONTEXT:  COPY sponsorTimesTemp, line 1548497: "anr6QUZ0mSY,532.5612,537.316,0,0,1,e8d0e34783b3c0adec456bda5eda5cd76ee06b1db4b5b9b4570c4c85aad3cb4f,..."
postgres-sb-mirror   | 2023-02-08 23:52:31.515 UTC [46] STATEMENT:  BEGIN; DROP TABLE IF EXISTS "sponsorTimesTemp"; CREATE UNLOGGED TABLE "sponsorTimesTemp"(LIKE "sponsorTimes" INCLUDING defaults INCLUDING constraints INCLUDING indexes); COPY "sponsorTimesTemp" FROM '/mirror/sponsorTimes.csv' DELIMITER ',' CSV HEADER; DROP TABLE "sponsorTimes"; ALTER TABLE "sponsorTimesTemp" RENAME TO "sponsorTimes"; COMMIT;
sponsorblock-mirror  | Failed to import database: missing data for column "views"
postgres-sb-mirror   | 2023-02-08 23:53:23.483 UTC [21] LOG:  checkpoint starting: time
postgres-sb-mirror   | 2023-02-08 23:53:23.618 UTC [21] LOG:  checkpoint complete: wrote 4 buffers (0.0%); 1 WAL file(s) added, 0 removed, 0 recycled; write=0.102 s, sync=0.003 s, total=0.136 s; sync files=53, longest=0.001 s, average=0.001 s; distance=358 kB, estimate=358 kB
sponsorblock-mirror  | Importing database...
postgres-sb-mirror   | 2023-02-08 23:54:04.374 UTC [46] ERROR:  missing data for column "views"
postgres-sb-mirror   | 2023-02-08 23:54:04.374 UTC [46] CONTEXT:  COPY sponsorTimesTemp, line 1548497: "anr6QUZ0mSY,532.5612,537.316,0,0,1,e8d0e34783b3c0adec456bda5eda5cd76ee06b1db4b5b9b4570c4c85aad3cb4f,..."
postgres-sb-mirror   | 2023-02-08 23:54:04.374 UTC [46] STATEMENT:  BEGIN; DROP TABLE IF EXISTS "sponsorTimesTemp"; CREATE UNLOGGED TABLE "sponsorTimesTemp"(LIKE "sponsorTimes" INCLUDING defaults INCLUDING constraints INCLUDING indexes); COPY "sponsorTimesTemp" FROM '/mirror/sponsorTimes.csv' DELIMITER ',' CSV HEADER; DROP TABLE "sponsorTimes"; ALTER TABLE "sponsorTimesTemp" RENAME TO "sponsorTimes"; COMMIT;
sponsorblock-mirror  | Failed to import database: missing data for column "views"

    Not clear to me how much this error matters (it seems to be responding to API requests fine?), but it repeats in the logs seemingly forever.

    opened by AlyoshaVasilieva 0
  • chore(deps): update docker/build-push-action action to v4

    chore(deps): update docker/build-push-action action to v4

    Mend Renovate

    This PR contains the following updates:

    | Package | Type | Update | Change | |---|---|---|---| | docker/build-push-action | action | major | v3 -> v4 |

    Release Notes

    docker/build-push-action

    v4

    Compare Source

    Configuration

    πŸ“… Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

    🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

    β™» Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

    πŸ”• Ignore: Close this PR and you won't be reminded about this update again.

    • [ ] If you want to rebase/retry this PR, check this box

    This PR has been generated by Mend Renovate. View repository job log here.

    opened by renovate[bot] 0
  • could not open file

    could not open file "/mirror/sponsorTimes.csv" for reading: Permission denied

    So for a while now my mirror logs are filled with

    Importing database...
Failed to import database: could not open file "/mirror/sponsorTimes.csv" for reading: Permission denied
Importing database...
Failed to import database: could not open file "/mirror/sponsorTimes.csv" for reading: Permission denied
Importing database...
Failed to import database: could not open file "/mirror/sponsorTimes.csv" for reading: Permission denied
Importing database...
Failed to import database: could not open file "/mirror/sponsorTimes.csv" for reading: Permission denied
Importing database...
Failed to import database: could not open file "/mirror/sponsorTimes.csv" for reading: Permission denied
Importing database...
Failed to import database: could not open file "/mirror/sponsorTimes.csv" for reading: Permission denied
Importing database...
Failed to import database: could not open file "/mirror/sponsorTimes.csv" for reading: Permission denied
Importing database...
Failed to import database: could not open file "/mirror/sponsorTimes.csv" for reading: Permission denied
Importing database...
Failed to import database: could not open file "/mirror/sponsorTimes.csv" for reading: Permission denied
Importing database...
...

    and i am assuming its about the files being owned by root:root, is that an issue on my side or a possible bug in the mirror itself? i only compose up the stack, nothing else, i've tried removing the mirror folder and recreating the stack but that didnt do the trick πŸ€”

    opened by Roki100 0
  • Dependency Dashboard

    Dependency Dashboard

    This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

    Open

    These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

    Detected dependencies

    cargo
    Cargo.toml
    • diesel 2.0.3
    • diesel_logger 0.2.0
    • diesel_migrations 2.0.0
    • lazy_static 1.4.0
    • num_cpus 1.15.0
    • regex 1.7.1
    • reqwest 0.11.14
    • serde 1.0.152
    • serde_json 1.0.93
    • tokio 1
    docker-compose
    docker-compose.yml
    • postgres 15-alpine
    dockerfile
    Dockerfile
    Dockerfile.ci
    github-actions
    .github/workflows/build-docker.yml
    • actions/checkout v3
    • docker/setup-qemu-action v2
    • docker/setup-buildx-action v2
    • docker/login-action v2
    • docker/build-push-action v3
    .github/workflows/build.yml
    • actions/checkout v3
    • Swatinem/rust-cache v2
    • [ ] Check this box to trigger a request for Renovate to run again on this repository
    opened by renovate[bot] 0
Owner
Team Piped
The developers of the Piped application.
Team Piped
GitHub https://sponsorblock.kavin.rocks
A fast, efficient osu! beatmap mirror written in asynchronous Rust

A fast, efficient osu! beatmap mirror written in asynchronous Rust. Supports cheesegull, aswell as osu!api v2 formats.

James 4 Oct 28, 2022
Mirror of https://gitlab.redox-os.org/redox-os/termion

Documentation Examples Changelog Tutorial Termion is a pure Rust, bindless library for low-level handling, manipulating and reading information about

Redox OS 1.9k Dec 31, 2022
Check the reproducibility status of your Arch Linux packages (read-only mirror)

arch-repro-status A CLI tool for querying the reproducibility status of the Arch Linux packages using data from a rebuilderd instance such as reproduc

Arch Linux 12 Nov 16, 2022
A library that creates a terminal-like window with feature-packed drawing of text and easy input handling. MIRROR.

BearLibTerminal provides a pseudoterminal window with a grid of character cells and a simple yet powerful API for flexible textual output and uncompli

Tommy Ettinger 43 Oct 31, 2022
By mirroring traffic to and from your machine, mirrord surrounds your local service with a mirror image of its cloud environment.

mirrord lets you easily mirror traffic from your Kubernetes cluster to your development environment. It comes as both Visual Studio Code extension and

MetalBear 2.1k Jan 3, 2023
A mirror of lib.rs/main

Lib.rs (Crates.rs) Lib.rs is a fast, lightweight index of Rust libraries and applications. Crates published to crates.io will automatically show up on

Max Bruce 3 Aug 4, 2023
Video Management System. Mirror of Codeberg.

Screenshots | Changelog | Codeberg | Matrix Overview Full resolution live view with sub 2 second delay 24/7 recording to custom database TFlite object

null 44 Nov 14, 2023
Bruteforce connecting to a specific Sea of Thieves server. Useful if you want to be in the same server as your friends.

SoT Server Finder Find which Sea of Thieves server you're connected to. Useful if you want to be in the same server as your friends. Setup Download so

Martin 4 Mar 19, 2023
Leptos server signals synced through Server-Sent-Events (SSE)

Leptos Server Sent Events Server signals are leptos signals kept in sync with the server through server-sent-events (SSE). The signals are read-only o

messense 13 Oct 3, 2023
A fast and lightweight HTTP server implementation in Rust.

server_nano A tiny, fast, and friendly web server written in rust and inspired by express. It uses may to coroutines Usage First, add this to your Car

Jonny Borges 5 May 2, 2023
An implementation of a Windows Event Collector server running on GNU/Linux.

OpenWEC OpenWEC is a free and open source (GPLv3) implementation of a Windows Event Collector server running on GNU/Linux and written in Rust. OpenWEC

CEA IT Security 15 Jun 15, 2023
A SIMD implementation of Keccak256 for aarch64, forked from Remco Bloeman's Goldilocks K12 implementation.

keccak256-aarch64 Warning This crate was forked from real cryptographers (Goldilocks by Remco Bloeman), by not a real cryptographer. Do not use this k

null 6 Oct 24, 2023
Rust API Server: A versatile template for building RESTful interfaces, designed for simplicity in setup and configuration using the Rust programming language.

RUST API SERVER Introduction Welcome to the Rust API Server! This server provides a simple REST interface for your applications. This README will guid

Harry Nguyen 3 Feb 25, 2024
A Rust CLI tool that helps you enforce Git policies through Git hooks both server and client side

GitPolicyEnforcer This is a command line utility written in Rust, that helps you utilize Git hooks, to enforce various policies. It currently supports

Vagelis Prokopiou 4 Aug 14, 2022
A small CLI tool to query ArcGIS REST API services, implemented in Rust. The server response is returned as pretty JSON.

A small CLI tool to query ArcGIS REST API services, implemented in Rust. The server response is returned as pretty JSON.

Andrew Vitale 2 Apr 25, 2022
A blazing fast and easy to use TRPC-like server for Rust.

rspc ?? Work in progress ?? A blazing fast and easy to use TRPC-like server for Rust. Website Example You define a trpc router and attach resolvers to

Oscar Beaumont 344 Dec 31, 2022
Bracket QOS - a Quality of Service/Experience server system writtin in Rust. Derived from the excellent LibreQOS Project.

Bracket QOS - Open Source Edition This project is based on the excellent LibreQOS project. It is hoped that this can be useful for the upstream projec

Herbert 12 Dec 2, 2022
Email test server for development, written in Rust

MailCrab Email test server for development, written in Rust. Inspired by MailHog and MailCatcher. MailCrab was created as an exercise in Rust, trying

Tweede golf 30 Feb 7, 2023
Fast turbo remote cache server written in Rust

Fast turbo remote cache server written in Rust. if you are using turbo and you want to have a self hosted remote cache server this is for you.

Salama Ashoush 10 May 24, 2023