Building a graph of the Internet, one button at a time

Related tags

Graphics eightyeightthirtyone
Overview

eightyeightthirtyone

Building a graph of the Internet, one button at a time. Website available here.

This project crawls the links between 88x31s on the Internet, which are small badges on websites that link to other websites. It's split into three projects:

  • A host server (Rust) that manages work between scraper nodes and talks to a Redis database using axum and fred
  • A scraper (Rust) that talks to the server, fetches URLs, and returns information using scraper
  • A web app (TypeScript/React) to render the graph using Cosmograph

Websites are scanned for images, and images that match the 88x31 resolution and link to another site are logged. It respects robots.txt and is aware of redirects.

Scrapers can either run a WebDriver or just parse the HTML - note that the latter will break discovery for websites that use JavaScript to create the button elements (e.g. React apps).

Opting out

The scrapers respect robots.txt, so block this user agent (or allow if you want to opt in):

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 eightyeightthirtyone/1.0.0 (https://github.com/NotNite/eightyeightthirtyone)

Note that this only somewhat works:

  • You will still appear on the graph if anyone else links to you.
  • Previous entries in the database are not deleted - please email me your domain and affected URLs if known (see Credits & contact).

Credits & contact

This project wouldn't be possible without the following people:

  • notnite
    • for starting the project, hosting the website and server
  • adryd
    • for her help on the frontend, designing the project's 88x31, hosting a scraper node
  • breq
    • for her help designing the Redis database schema, contributions to the server
  • Inferno Comms
    • for lending us a server to host a scraper node on

If you have any questions or concerns, you can send an email to me (NotNite), which is available on my website (not written here for spam concerns).

Thanks!

You might also like...
Theorem relational dependencies automatic extraction and visualization as a graph for Lean4.
Theorem relational dependencies automatic extraction and visualization as a graph for Lean4.

Lean Graph Interactive visualization of dependencies for any theorem/definitions in your Lean project. How to use In your browser: lean-graph.com Or r

Patrik Číhal 8 Jan 3, 2024
A fast, lightweight and extensible implementation of a graph data structure.

fast-graph A fast, lightweight and extensible implementation of a graph data structure. Note ⚠️ There will be some breaking changes in the coming 1-2

Henrik 34 Jul 6, 2024
Pathfinding library for calculating all node pairs' shortest paths in an unweighted undirected graph.

bit_gossip bit_gossip, named after its implementation technique, is a simple pathfinding library for calculating all node pairs' shortest paths in an

Jack Lee 49 Sep 4, 2024
Real-time 3D orientation visualization of a BNO055 IMU using Bissel and Bevy
Real-time 3D orientation visualization of a BNO055 IMU using Bissel and Bevy

orientation This is a demonstration of real-time visualization of the attitude of a BNO055 IMU across a wireless network to a Bevy app using the Bisse

chris m 4 Dec 10, 2022
An example repository on how to start building graph applications on streaming data. Just clone and start building πŸ’» πŸ’ͺ
An example repository on how to start building graph applications on streaming data. Just clone and start building πŸ’» πŸ’ͺ

An example repository on how to start building graph applications on streaming data. Just clone and start building πŸ’» πŸ’ͺ

Memgraph 40 Dec 20, 2022
Dank - The Internet Computer Decentralized Bank - A collection of Open Internet Services - Including the Cycles Token (XTC)
Dank - The Internet Computer Decentralized Bank - A collection of Open Internet Services - Including the Cycles Token (XTC)

Dank - The Internet Computer Decentralized Bank Dank is a collection of Open Internet Services for users and developers on the Internet Computer. In t

Psychedelic 56 Nov 12, 2022
Open Internet Service to store transaction history for NFTs/Tokens on the Internet Computer
Open Internet Service to store transaction history for NFTs/Tokens on the Internet Computer

CAP - Certified Asset Provenance Transaction history & asset provenance for NFT’s & Tokens on the Internet Computer CAP is an open internet service pr

Psychedelic 42 Nov 10, 2022
Cover is an open internet service for canister code verification on the Internet Computer
Cover is an open internet service for canister code verification on the Internet Computer

Cover Cover (short for Code Verification) is an open internet service that helps verify the code of canisters on the Internet Computer. Visit our webs

Psychedelic 14 Oct 31, 2022
Egui node graph is a featureful, customizable library to create node graph applications using egui
Egui node graph is a featureful, customizable library to create node graph applications using egui

Egui node graph is a featureful, customizable library to create node graph applications using egui. The library takes care of presenting a node graph to your users, and allows customizing many aspects of the interaction, creating the semantics you want for your specific application.

null 367 Jan 8, 2023
A game for the game jam
A game for the game jam "1-Button Jam 2021"

One click ninja A game for the game jam "1-Button Jam 2021" written in Rust with the Bevy engine. A rhythm game, where you play a soldier that can def

Alex Helfet 7 Apr 12, 2022
Use the osu!direct button on the osu! website to download beatmaps directly to osu
Use the osu!direct button on the osu! website to download beatmaps directly to osu

osum!direct-web Use the osu!direct button on the osu! website to download beatmaps directly to osu! - without osu! supporter. Badges Usage Run the app

oSumAtrIX 1 Jan 21, 2022
Example of an dark-mode toggle button based on progressive enhancement

Leptos Starter Template This is a template for use with the Leptos web framework and the cargo-leptos tool. Creating your template repo If you don't h

Leptos 5 Jan 12, 2023
Building a better screen reader for the Linux desktop, one step at a time.

Building a better screen reader for the Linux desktop, one step at a time.

Odilia 44 Dec 31, 2022
GraphScope: A One-Stop Large-Scale Graph Computing System from Alibaba
GraphScope: A One-Stop Large-Scale Graph Computing System from Alibaba

A One-Stop Large-Scale Graph Computing System from Alibaba GraphScope is a unified distributed graph computing platform that provides a one-stop envir

Alibaba 2.2k Jan 1, 2023
The Graph is a protocol for building decentralized applications (dApps) quickly on Ethereum and IPFS using GraphQL.

Graph Node The Graph is a protocol for building decentralized applications (dApps) quickly on Ethereum and IPFS using GraphQL. Graph Node is an open s

Mindy.wang 2 Jun 18, 2022
A library that can be used as a building block for high-performant graph algorithms.

graph A library that can be used as a building block for high-performant graph algorithms. Graph provides implementations for directed and undirected

Martin Junghanns 222 Jan 1, 2023
Limit screen time to children's various mobile devices by blocking internet access on the family Wifi router.

Device Blocker Limit screen time to children's various mobile devices by blocking internet access on the family Wifi router. This is the server which

null 36 Oct 4, 2022
⌚ A command-line tool (and library) for the rusty Swatch Internet Time.

⌚ A command-line tool (and library) for the rusty Swatch Internet Time. Comes with XBar/Swiftbar support.

Gil Desmarais 4 Jul 18, 2022
A simple menu to keep all your most used one-liners and scripts in one place
A simple menu to keep all your most used one-liners and scripts in one place

Dama Desktop Agnostic Menu Aggregate This program aims to be a hackable, easy to use menu that can be paired to lightweight window managers in order t

null 47 Jul 23, 2022
Comments
  • Better styling + experimental light-mode support, node/link sharing via hashes, and more.

    Better styling + experimental light-mode support, node/link sharing via hashes, and more.

    • Better looks for the overlaying UI
      • Use variables for colors and other things
      • Better loading screen
        • Yes I duplicated the loading screen element because I think it'd make not much sense to throw it into its own component since it's so small.
      • Experimental light mode, probably doesn't look that neat. but hey it's something.
    • Sharing functionality via hashes. (e.g https://eightyeightthirty.one/#neocities.org), and a copy button.
      • Changing the hash in the url also acts the same as using the search box in the controls
    • Reset view button
    • "Go!" button next to the search bar

    chrome_GOfBIWvSA5

    opened by flleeppyy 1
  • Attempt to find canonical URLs

    Attempt to find canonical URLs

    This MR tries to handle situations where someone's hosting the same page on a set of slightly different URLs (for instance, http://example.com, https://example.com, http://www.example.com, https://www.example.com).

    The scraper will attempt to identify a canonical version of the current URL using either a rel=canonical tag (preferred) or an og:url tag (fallback). If either of these are found, the scraper will redirect to the referenced URL before continuing. This redirection gets reported to the server in the same way an HTTP 301 would be reported.

    opened by breqdev 0
  • Implement leaderboard for scrapers

    Implement leaderboard for scrapers

    Adds a leaderboard for the work done by each scraper, in case it's useful for:

    • Comparing different scraper implementations for speed
    • Comparing scraper host machines for performance
    • Validating the breakdown of work between scrapers
    • Bragging rights
    opened by breqdev 0
  • AVIF support

    AVIF support

    eightyeightthirtyone currently does not scrape/save AVIF 88x31 images. This could be fixed by bumping the version of image-rs to the latest version (0.25.2 at the time of writing) and adding the "avif-native" feature.1
    Alternatively it seems that enabling the "avif-decoder" feature on 0.24.8 might work as well.2

    1 https://docs.rs/image/latest/image/codecs/index.html 2 https://docs.rs/image/0.24.8/image/codecs/avif/struct.AvifDecoder.html

    opened by onzecki 0
Owner
Jules
jack of all trades, master of none. programmer, game modder, reverse engineer, shitposter. FOSS fanatic. let's build cuteware!
Jules
GitHub https://eightyeightthirty.one/
Graph data structure library for Rust.

petgraph Graph data structure library. Supports Rust 1.41 and later. Please read the API documentation here Crate feature flags: graphmap (default) en

null 2k Jan 9, 2023
A graph library for Rust.

Gamma A graph library for Rust. Gamma provides primitives and traversals for working with graphs. It is based on ideas presented in A Minimal Graph AP

Metamolecular, LLC 122 Dec 29, 2022
Simple but powerful graph library for Rust

Graphlib Graphlib is a simple and powerful Rust graph library. This library attempts to provide a generic api for building, mutating and iterating ove

Purple Protocol 177 Nov 22, 2022
πŸ¦€ Rust Graph Routing runtime for Apollo Federation πŸš€

Apollo Router The Apollo Router is a configurable, high-performance graph router for a federated graph. Getting started Follow the quickstart tutorial

Apollo GraphQL 502 Jan 8, 2023
A simple and elegant, pipewire graph editor

pw-viz A simple and elegant, pipewire graph editor This is still a WIP, node layouting is kinda jank at the moment. Installation A compiled binary is

null 180 Dec 27, 2022
A graph crate with simplicity in mind

A graph crate with simplicity in mind. Prepona aims to be simple to use (for users of the crate) and develop further (for contributors). Nearly every

Mohamad Amin Rayej 80 Dec 15, 2022
Graph API client writen in Rust

graph-rs Now available on stable Rust at crates.io graph-rs-sdk = "0.1.0" 0.1.0 and above use stable Rust. Anything before 0.1.0 uses nightly Rust. M

Sean Reeise 56 Jan 3, 2023
Rust library for of graph ensembles

Rust library for random graph ensembles Minimal Rust version: 1.55.0 Implements simple sampling and monte carlo (or rather markov-) steps, that can be

Yannick Feld 2 Dec 14, 2022
A Graph implemented using nothing but `Vec`s in rust

VecGraph A Graph implemented using nothing but Vecs in rust. Details The graph is implemented using two Vecs: nodes and edges. nodes stores "nodes". w

null 1 Jan 27, 2022
Simple, performant graph generator for Feynman diagrams* βš›οΈ

Feynman diagram generator βš›οΈ A simple generator of "Feynman diagram" permutations (as defined by problem 781). Incrementally builds isomorphically uni

eugene huang 3 Jan 1, 2023