A distributed, cryptographically-verifiable blog / social network

Related tags

Cryptography feoblog
Overview

FeoBlog

FeoBlog is a distributed blogging platform. It takes a lot of its inspiration from Mastodon and Scuttlebutt. It aims to solve a couple of problems with those services.

On Mastodon, your user identity is tied to a particular server. If that server goes down, your content, your user ID and your social network disappear with it. But with FeoBlog, your ID is yours. Your content can be hosted on multiple servers, so that if one of them goes down, your content isn't all lost. If you want to leave a server, you can just move your content to another server without getting a new user ID and having to manually tell all of your friends.

Scuttlebutt tries to be a distributed system, but it has one big problem: Only one device can post new content. If you copy your identity to two devices and post from both of them, you can get into a state where your identity is broken. This is because Scuttlebutt tries to maintain an append-only, linear chain of history (a bit like a blockchain) which could fork if you use it on devices. (And Scuttlebutt doesn't deal well with forks.) FeoBlog treats your identity as a collection of posts by you. (Not an append-only chain.) This lets you post from multiple devices, and lets servers eventually fetch and include all posts from all of those devices. This also lets servers and users delete some of your content, for example, if they only want to host the most recent posts by you, or want to exclude content that uses too much disk space.

OK, well, all of this is theory so far. These are the grand plans. Quotas aren't implemented yet.

Core Features

The main feature of FeoBlog is its data structure. Anyone can create a NaCL public key, and use it (base58-encoded) as a user ID. A "blog" for a user is just a collection if binary protobuf objects which are signed by that user. Each object must be an Item, which is defined in feoblog.proto.

Methods for storing and displaying these items can be developed separately by different clients. This repository includes two renderers for such content. One is plain HTML, rendered server-side, accessed at paths like /u/:userID/i/:signature/. The other is an in-browser JavaScript client which fetches the binary data directly from the server and renders it client-side, at a URL like /client/#/u/:userID/i/:signature.

Because items are cryptographically signed by the user who posts them, they can be safely copied around to different servers and caches by anyone. And anyone can read them knowing that they haven't been modified.

For servers/clients on the web, there are standard URLs for getting and sending Items.

Other features

  • Uses a safe subset of CommonMark markdown for posts.
  • Can easily run a server locally
    • Sync content from those you follow to have offline.
    • Compose posts offline, and send them all when you're back online.

Planned features

  • Comments
  • "Reply" posts which link to the Item they reply to
  • File attachments
  • Revoking user IDs (i.e.: "Delete my account.")

Unplanned features

There are certain features that I do not plan to implement, because I think they are detrimental in social networks.

  • Likes, or counts for likes, replies, or follows. These are easy to game and people assign too much meaning to them.
  • Edits or deletes. Content you post is cryptographically signed and visible forever, unless you revoke your userID.
  • Reblogging. I believe there should be a higher barrier to sharing others' content. You'll need to comment, post, or "Reply" post to share content to your followers.

The Name

I'm not a great UI designer, so my blog will be a bit feo. Fe2O3 is also the chemical formula for rust, and this implementation is written in Rust. :p

Implementation Details

Development

Dependencies:

You may need to run npm install inside the web-client subdirectory.

To develop the interactive web client, run npm run watch in the web-client subdirectory, then (in another window) run cargo run serve --open.

Building

To build a "release"/self-contained version of Feoblog:

  • In web-client/, run npm run build
  • In the root directory, run cargo build --release
    • or, alternatively: cargo install --path . --locked

Getting Started

If you don't want to set up your server right away, check out the Demo on YouTube!

Run the server

Once you've built or downloaded feoblog, you can run it locally by just running:

feoblog serve --open

This will:

  • Start a server on localhost:8080. (You can override w/ the --bind option)
  • Create a database called feoblog.sqlite3 in the current directory.
  • Open a web browser window pointing to your new empty database.

Create a User ID

By default, http://localhost:8080/ will show you the plain HTML version of the site. This is a version of the site that's indexable by search engines, and readable by software and browsers that don't have JavaScript available.

To do more than just read existing content, you'll need to use a client. FeoBlog comes with a built-in client that runs in your browser.

Click the "Client" link to open the in-browser client.

Next, click the "Log in" link.

At the bottom of the page, click the "Create new ID button". This will generate a new user ID for you. It'll look like a random string of characters. For example: A719rvsCkuN2SC5W2vz5hypDE2SpevNTUsEXrVFe9XQ7.

The page will also generate a password for you. It's important to save this key in a secure location like a password manager. You can't change or reset this password. (It's a cryptographic private key that corresponds to your public ID.)

Add Yourself to the Server

Now that you've generated a userID for yourself, you need to tell the server who you are.

A FeoBlog server doesn't contain any passwords, all it knows is a list of user IDs that are allowed to post content to it. Since all content is cryptographically signed by a user, the server can verify that a post came from you without your password.

So, using the userID above, to add myself to the server I'd just run:

feoblog user add A719rvsCkuN2SC5W2vz5hypDE2SpevNTUsEXrVFe9XQ7 --on-homepage --comment "Official FeoBlog Blob

You can do this by stopping the server with Ctrl-C first, or by running the command in a new window. (But make sure to re-start the server before the next steps!)

The optional --on-homepage argument says that posts you post to this ID should appear on the Home page of the feoblog, as well as in your individual user page.

And the optional --comment X argument is just a comment to help you, the server admin, keep track of who that ID is. It's only ever shown in the output of feoblog user list.

Log In

Return to the "Log In" page in the browser.

Paste your userID (not password!) into the "Log In As:" field, and click the "Log In" button. You'll get a warning that your profile doesn't exist. (We'll write one next!) Click the "Confirm" button.

Now you're "logged in". You may be surprised that no password was required. "Logging in" to the client just tells it to present data to you as if you are that user. Don't worry, a password will be required to write any data.

The client is built with the idea that you may manage multiple identities. If you generate another ID, you can "log in" as that identity as well, and the client will remember so that you can easily switch between them.

To help distinguish your identities, you can (and should!) give them different names and colors. Just edit the "Name" and "Color" fields as you wish. Changes are saved immediately. Colors should be 3- or 6-digit hexadecimal colors like #03c or #0033cc. Here's a handy color picker for you.

Create Your Profile

User IDs are not a great way to remember people. Thankfully FeoBlog supports user profiles. There, you can set a name for yourself and provide a short description of yourself and/or the purpose of your blog.

Click on the "My Profile" link.

FeoBlog again warns you that it can't find an existing profile for you. If a profile does exist, you have the option to "Sync from another server" so that you can re-use (or modify) the existing one. But since we're creating a new ID, click "Create New Profile".

For now, the important parts to set are:

  • Profile Display Name -- this is a friendly name or nickname to display instead of your long randomly-generated userID. You should set this, but it's not required, if you really prefer being an anonymous number.
  • The "Your profile here..." text box serves as a description for you and/or your blog. Write whatever you want. This box accepts "CommonMark" Markdown formatting.

Once you've got those set, now paste your password into the "Private Key" field, and click "Sign".

This will automatically generate a cryptographic signature for this content. If everything still looks good, click the "Send" button to send this profile to the server.

Write Your First Post

Click on the "New Post" link at the top/left of the page.

This interface is very much like the "My Profile" page. Fill out the title and body of your post, then sign and send the post.

You can view it in "My Feed" and (if you enabled --on-homepage above) on the "Home" feed.

Linking

Since FeoBlog content is distributed, and may be hosted on multiple servers, you should avoid hard-linking to a particular server. If you want to link to a FeoBlog userID or post within your post, use relative links, like this:

Did you see the [FeoBlog] [first post]?

[FeoBlog]: /u/A719rvsCkuN2SC5W2vz5hypDE2SpevNTUsEXrVFe9XQ7/
[first post]: /u/A719rvsCkuN2SC5W2vz5hypDE2SpevNTUsEXrVFe9XQ7/i/2F6NB6PYKDTPGTc9dfaQHpmPzd3LSjVgBuC6qa2hcLUJA74LbZpV8wL5HoXDmvzyfZWaX6sLyg3DoGtqh3t2rJt5/

Of course, if you're linking someone outside of FeoBlog to a partticular post, you can link them directly to a page on a particular server like this:

https://blog.nfnitloop.com/u/A719rvsCkuN2SC5W2vz5hypDE2SpevNTUsEXrVFe9XQ7/i/2F6NB6PYKDTPGTc9dfaQHpmPzd3LSjVgBuC6qa2hcLUJA74LbZpV8wL5HoXDmvzyfZWaX6sLyg3DoGtqh3t2rJt5/

The URLs are a bit long, but many services (like Twitter) will shorten them for you anyway. Plus, the URL contains a globally unique ID which can also be used to cryptographically verify the contents of the post. If any one server goes down, the /u/... relative path can be used on any other FeoBlog server that contains a copy of that item.

Advanced Topics

I should probably write more about these things? Tell me if you'd find them useful.

  • Using Sync to copy your content between servers. (Hopefully the in-client info is enough for now?)
  • Running a server behind Apache
  • Running a server in Docker
  • Writing your own client
  • Writing your own server
Comments
  • Enable link previews

    Enable link previews

    When someone shares a link to a FeoBlog post onto other social networks, it would be nice if they would get a preview.

    • title
    • text
    • any images if attached
    • Fallback image(?)

    Seems like stuff uses this Open Graph Protocol https://ogp.me/ to pull that info.

    • [x] Do I want to have a logo as a fallback? OGP seems to require an image. (#64)
    • [x] Should probably add a "Share" button to copy the correct link, since the /client/#/... links don't actually exist server-side.
    opened by NfNitLoop 3
  • Faster Sync

    Faster Sync

    Now that I've got some "blogs" with lots of "posts", the initial sync can be a bit slow.

    For example:

    image

    There's probably room to parallelize the sync more than it is already.

    I think I can also walk down the list of items instead of fetching the full list then doing set comparisons. Though I doubt that's the slow part, it's probably just the latency overhead of copying items from and to a server one-by-one.

    opened by NfNitLoop 2
  • Better browser caching for static files

    Better browser caching for static files

    I thought I'd need support for this in rust-embed, but as I wrote up https://github.com/pyros2097/rust-embed/issues/140 I realized a workaround would be simple:

    Create some wrapper that calculates an ETag (and also Last-Modified date?) on server startup or lazily on the first request. Then serve static files w/ that ETag so browsers will cache them longer.

    opened by NfNitLoop 2
  • Redirect JS-enabled browsers to the JS client

    Redirect JS-enabled browsers to the JS client

    The UI of the JS and plain HTML versions of the site has diverged a bit, and TBH the JS version is much nicer. For a better initial experience, redirect browsers that have JavaScript enabled to the corresponding URL in the client.

    • [x] Should include an option to disable the redirect.
    opened by NfNitLoop 2
  • Continuous Integration

    Continuous Integration

    Need to get some automated builds and releases on this.

    • [X] Automatic builds for develop branch. I don't care about warnings at this stage. OMG Rust has so many warnings.
    • [X] Automatic builds for pull requests.
    • [x] Automatic builds & release for release stage. OK I should probably clean up warnings for that.

    Oh, right, also, some tests. 😬

    opened by NfNitLoop 2
  • File Attachments

    File Attachments

    Users should be able to attach files to a post. This would make sharing images, PDFs, etc. possible.

    Rough plans are:

    • Implement a content-addressable store.
    • Re-use whatever hash-algorithm NaCL signatures currently use, since we're already using it.
    • File names, hashes & sizes are stored within the protobuf.
    • Standard URL sub-directory where files live.
    • File uploads to the file names can be performed by anyone.
    • File attachment quotas may disallow attaching large files.

    Additional work:

    • Update the client to know how to sync file attachments along with their Items.
    planned In Progress 
    opened by NfNitLoop 2
  • Comments

    Comments

    • Users should be able to comment on any Item (including other comments).

    • Clients should have a way to query for comments in response to an Item.

    • By default, the displayed comments should be limited to:

      • Those followed by the author of the original Item.
      • (optionally) Additionally those followed by the logged-in user.
    • Comments should not be rendered on the main HTML-only home page.

    • Comments may optionally be rendered in clients' feeds (chronologically, detached, but linking to the Item they respond to.)

    • Comments should be rendered when viewing an individual item.

    • Comments are not Posts. In particular, they lack:

      • Titles
      • The ability to attach files (once that's available)
      • The ability to inline images. (TBD)
    opened by NfNitLoop 2
  • Fix back behavior in iOS Safari

    Fix back behavior in iOS Safari

    See: https://gomakethings.com/fixing-safaris-back-button-browser-cache-issue-with-vanilla-js/

    which recommends:

    /**
     * If browser back button was used, flush cache
     * This ensures that user will always see an accurate, up-to-date view based on their state
     * https://stackoverflow.com/questions/8788802/prevent-safari-loading-from-cache-when-back-button-is-clicked
     */
    (function () {
    	window.onpageshow = function(event) {
    		if (event.persisted) {
    			window.location.reload();
    		}
    	};
    })();
    

    Need to investigate further and make sure that only affects Safari.

    next-release 
    opened by NfNitLoop 1
  • Add

    Add "Share" button(s)

    Since there's there are raw (/u/_/i/_/) links and client (/client/#/u/_/...) links, it ends up being easier to share the client link, though the other one would likely result in better SEO and discoverability. Add a share button. It could:

    • Share to Twitter
    • Copy Link
    • ???
    opened by NfNitLoop 1
  • Make the

    Make the "feed" page the default page in the client

    When I visit blog.nfnitloop.com, the thing I usually want to see is the feed of all my follows' (and my) posts. But it currently takes me to the "Home" view by default:

    image

    This is just because we redirect to the client from /, and I'm trying to mirror the URL structure from the web site in the client. However, I think we can make an exception for the home page, give it an explicit /client/#/home URL, and redirect /client/#/ to the user's feed.

    opened by NfNitLoop 1
  • Support attachments from iOS (& Android?)

    Support attachments from iOS (& Android?)

    Right now you have to drag/drop attachments into the web UI to add them. This doesn't work on iOS (and probably not Android?)

    I suppose I should add some UI element that's more discoverable than drag & drop to enable that.

    opened by NfNitLoop 1
  • Store `Item` bytes in content-addressable store

    Store `Item` bytes in content-addressable store

    When a user uploads an Item, its bytes are currently stored directly into the item table for easy access. But, that means that if multiple users sign and upload the same Item, it gets stored multiple times. And with #92 we'll already be storing some items into the store.

    Instead, we should always store the item's bytes into the content-addressable store.

    Implementation details to update:

    • Initial Item uploads store the bytes in store.
    • DB migration of old item bytes into store
    • Fix db usage and prune to account for items stored in this way. (i.e.: don't prune rows from store that have a corresponding item entry!)
    impl 
    opened by NfNitLoop 1
  • Allow Items to

    Allow Items to "embed" other Items.

    Unlike file attachments, these allow embedding other FeoBlog Items "into" an item that needs to reference them.

    • The "embedded"[^1] Item is not stored inside of the protobuf in whole, but is referenced by its (userID, signature) pair (similar to a ReplyRef).
    • Servers will allow you to upload a copy of that item in association with the embedding item, as one currently does for attachments.
    • Sync tools can/should copy embedded items over as they would attachments.
    • These items are available for clients to fetch to render Items that refer to (possibly multiple) other items. (ex: Comments. Reply Posts. Likes. Reblogs. Reblog chains like Tumblr.)
    • The items are ONLY available to fetch within the context of the Item that embeds them. i.e.: Copying an embedded item does not make it available at a server's /u/:userID/i/:signature/proto3 endpoint. Thus items can be replied to and discussed in context without necessarily making them broadly available on servers that do not wish to host them.

    Implementations should store embedded Items in a content-addressable store so that commenting/sharing doesn't end up creating duplicates of items.

    This unblocks #6

    [^1]: TBD: better term? We already have a ReplyRef type.

    opened by NfNitLoop 1
  • Support server-side search

    Support server-side search

    The current client-side search is a bit of a hack. It's a brute-force search back through history, so is O(n) and uses O(n) network requests. (And network traffic, if you haven't got a cached copy of the Items locally.)

    SQLite actually supports full text search indexes! Could add an option to enable it, and let servers advertise that they support server-side search, at which point clients can use that.

    Use cases:

    All

    • Sort by search relevance
    • Allow users to choose which users' posts to search

    Search My (a) Feed

    • A user is viewing their feed in the UI.

    • Enters a search term in the search filter.

    • That UI calls the server-side search to find items that match.

    • Order by search relevance

    • Only show things in the user's feed. (option to tell server which user's feed to search? Though maybe this is too much a specialization and it should just accept a list of userIDs to search?)

    Server Search

    • Just search "all" users on a server.
    • Should we have a way for users to opt into/out of being globally searchable on servers?
      • Possibly limit it to servers that their profile claims to be hosted on?
      • What are the defaults?
    • A way for server admins to further limit what's globally searchable/discoverable?

    See also: #11

    opened by NfNitLoop 0
  • "Search" continues after you navigate away from the page

    To reproduce:

    1. On the feed page, start a text-based search.
    2. when an item shows up, click the link to navigate to its page.
    3. View the network control panel in the browser.

    Expected:

    • No more search-related traffic

    Observed:

    • The browser is still loading posts from my feed trying to perform the brute-force search.

    This is probably because of the current use of Async Generators that don't have a mechanism to get canceled.

    https://seg.phault.net/blog/2018/03/async-iterators-cancellation/ has a good writeup.

    Maybe I can just reuse the AbortController and pass that into the async generators, and make sure to abort it when the page is unloaded.

    bug 
    opened by NfNitLoop 1
  • Item

    Item "Extension" type.

    Add a freeform "extensions" field in the Item type so that third-party apps can add additional (meta)data to Items in an easily discoverable way.


    There's nothing stopping someone from shoving more data into an Item as it is, since protobuf if designed to ignore fields it doesn't (yet) know about. But leaving it up to folks to choose how to do that will possibly lead to collisions, and also make it hard to discover how to parse the embedded data.

    I'm thinking something like this:

    message Item {
        // ...
    
        // A place for third-party clients to add any additional metadata they wish.
        // the string key is the name of your extension.
        // Ex: if you have a client called "Foo" and a feature called "Bar Baz", use the format:
        // "foo:bar-baz"
        // Or, you could just use an extension name for your client ("foo") and map 
        // its various values to message properties inside of that object:  
        map <string, AnyMessage> extensions = // ...
    }
    
    message AnyMessage {
        // This space left blank. You may specify any fields you wish in your extension.
    }
    
    opened by NfNitLoop 0
Releases(v0.7.0-rc6)
Owner
Cody Casterline
Senior Software Engineer. Rust, Python, Java, Kotlin, Go, Bash. 😱
Cody Casterline
Peer-to-Peer, incentive social feed application on Linera

ResPeer: Peer-to-Peer content publishing application on Linera ResPeer on Linera ResPeer is a Peer-to-Peer content publishing application on Linera. R

null 10 Aug 17, 2023
As part of the IOP Stack™ Morpheus is a toolset to have gatekeeper-free identity management and verifiable claims as a 2nd layer on top of a blockchain

Internet of People Internet of People (IoP) is a software project creating a decentralized software stack that provides the building blocks and tools

We are building a complete decentralized ecosystem with the IOP Stack™ 9 Nov 4, 2022
Experiments on blockchain technology (also known as Hashed & Zero-trust Verifiable Linked List)

AngeloChain Experiments on blockchain technology (also known as Hashed & Zero-trust Verifiable Linked List) ⚠️ Before We Get Started Before we get sta

Angelo 1 Jan 20, 2022
RISC Zero is a zero-knowledge verifiable general computing platform based on zk-STARKs and the RISC-V microarchitecture.

RISC Zero WARNING: This software is still experimental, we do not recommend it for production use (see Security section). RISC Zero is a zero-knowledg

RISC Zero 653 Jan 3, 2023
Arkworks circuits for verifiable time-lock encryption

zk-timelock This repo contains arithmetic circuits for verifiable time-lock encryption made using arkworks-rs toolkit. For more details on such an enc

Timofey 68 Apr 5, 2023
Cross-chain bridge message delivery network. We are hiring, [email protected]

Introduction Implementation of a https://darwinia.network node in Rust based on the Substrate framework. This repository contains runtimes for the Dar

Darwinia Network 225 Nov 8, 2022
ARYA Network is a polkadot/substrate based chain for Non-fungible Token platform on which we can own sell and buy the NFT's on polkadot network.

ARYA Network ARYA Network is a polkadot/substrate based chain for Non-fungible Token platform on which we can own sell and buy the NFT's on polkadot n

Pankaj Chaudhary 6 Dec 20, 2022
The Zenotta Network Protocol (ZNP), the network that supports the Zenotta blockchain

Zenotta Network Protocol A repo for the development of the Zenotta Network Protocol (ZNP). We will regularly be updating links and easter eggs inside

Zenotta AG 10 Apr 2, 2023
dWallet Network, a composable modular signature network is the home of dWallets

Welcome to dWallet Network dWallet Network, a composable modular signature network is the home of dWallets. A dWallet is a noncollusive and massively

dWallet Labs 8 Feb 26, 2024
FS-DKR: One Round Distributed Key Rotation

FS-DKR: One Round Distributed Key Rotation Intro In this note we aim to re-purpose the Fouque-Stern Distributed Key Generation (DKG) to support a secu

[ZenGo X] 28 Dec 18, 2022
End-to-end encryption and mutual authentication for distributed applications.

✨ Hands-on Introduction: Build end-to-end encrypted, mutually-authenticated, secure messaging in Rust ✨ Rust and Elixir libraries for end-to-end encry

Ockam | Trust for Data-in-Motion 2.8k Jan 2, 2023
This is an experiment in designing a distributed connected garden experience.

This is an experiment in designing a distributed connected garden experience. It started as an implementation of a blockchain, but has moved on to be a bit more than that. There is no proof of work like a cryptocurrency, but plays with the idea of a distributed consensus-building system.

Greg Tatum 4 Feb 28, 2022
Distributed Vault For Your Secrets

https://meta-secret.github.io Application Design Activity Diagram graph TD User --> |split password| MSS{MetaSecret} MSS --> |split| Hash1

Meta Secret 4 Nov 9, 2022
The fly.io distributed systems challenges solved in Rust

The fly.io distributed systems challenges solved in Rust. Live-streamed in https://youtu.be/gboGyccRVXI License Licensed under either of Apache Licens

Jon Gjengset 162 Apr 19, 2023
Diem’s mission is to build a trusted and innovative financial network that empowers people and businesses around the world.

Note to readers: On December 1, 2020, the Libra Association was renamed to Diem Association. The project repos are in the process of being migrated. A

Diem 16.7k Jan 8, 2023
The Nervos CKB is a public permissionless blockchain, and the layer 1 of Nervos network.

Nervos CKB - The Common Knowledge Base master develop About CKB CKB is the layer 1 of Nervos Network, a public/permissionless blockchain. CKB uses Pro

Nervos Network 1k Dec 30, 2022
The Phala Network Blockchain, pRuntime and the bridge.

Phala Blockchain Phala Network is a TEE-Blockchain hybrid architecture implementing Confidential Contract. This repo includes: node/: the main blockch

Phala Network 314 Jan 6, 2023
Employ your built-in wetware pattern recognition and signal processing facilities to understand your network traffic

Nethoscope Employ your built-in wetware pattern recognition and signal processing facilities to understand your network traffic. Check video on how it

Vesa Vilhonen 86 Dec 5, 2022
A value transfer bridge between the Monero blockchain and the Secret Network.

Secret-Monero-Bridge A value transfer bridge between the Monero blockchain and the Secret Network. Proof-of-Concept Video Demonstration: https://ipfs.

null 28 Dec 7, 2022