Y-Octo is a high-performance CRDT implementation compatible with yjs

Overview

Y-Octo

test docs crates codecov

Y-Octo is a high-performance CRDT implementation compatible with yjs.

Introduction

Y-Octo is a tiny, ultra-fast CRDT collaboration library built for all major platforms. Developers can use Y-Octo as the Single source of truth for their application state, naturally turning the application into a local-first collaborative app.

Y-Octo also has interoperability and binary compatibility with yjs. Developers can use yjs to develop local-first web applications and collaborate with Y-Octo in native apps alongside web apps.

Who are using

AFFiNE is using y-octo in production. There are Electron app and Node.js server using y-octo in production.

Mysc is using y-octo in the Rust server, and the iOS/Android client via the Swift/Kotlin bindings (Official bindings coming soon).

Features

  • โœ… Collaborative Text
    • โœ… Read and write styled Unicode compatible data.
    • ๐Ÿšง Add, modify and delete text styles.
    • ๐Ÿšง Embedded JS data types and collaborative types.
    • โœ… Collaborative types of thread-safe.
  • Collaborative Array
    • โœ… Add, modify, and delete basic JS data types.
    • โœ… Recursively add, modify, and delete collaborative types.
    • โœ… Collaborative types of thread-safe.
    • ๐Ÿšง Recursive event subscription
  • Collaborative Map
    • โœ… Add, modify, and delete basic JS data types.
    • โœ… Recursively add, modify, and delete collaborative types.
    • โœ… Collaborative types of thread-safe.
    • ๐Ÿšง Recursive event subscription
  • ๐Ÿšง Collaborative Xml (Fragment / Element)
  • โœ… Collaborative Doc Container
    • โœ… YATA CRDT state apply/diff compatible with yjs
    • โœ… State sync of thread-safe.
    • โœ… Store all collaborative types and JS data types
    • โœ… Update event subscription.
    • ๐Ÿšง Sub Document.
  • โœ… Yjs binary encoding
    • โœ… Awareness encoding.
    • โœ… Primitive type encoding.
    • โœ… Sync Protocol encoding.
    • โœ… Yjs update v1 encoding.
    • ๐Ÿšง Yjs update v2 encoding.

Testing & Linting

Put everything to the test! We've established various test suites, but we're continually striving to enhance our coverage๏ผš

  • Rust Tests
  • Node Tests
  • Smoke Tests
  • Eslint, Clippy
  • OctoBase: The open-source embedded database based on Y-Octo.
  • yjs: Shared data types for building collaborative software in web.

Maintainers

Why not yrs

See Why we're not using yrs

License

Y-Octo are MIT licensed.

Comments
  • docs: update markdown link

    docs: update markdown link

    https://github.com/toeverything/y-octo/blob/main/y-octo/yrs-is-unsafe/README.md#L16:~:text=release/yrs%2Dmem-,The%20source%20codes%3A,-mem_usage.rs

    --->>>

    https://github.com/toeverything/y-octo/blob/6d3c131f2df7295c54358f4bd9af23ea8e422615/y-octo/yrs-is-unsafe/README.md#L62:~:text=release/yrs%2Dmem-,The%20source%20codes%3A,-mem_usage.rs

    opened by thorseraq 1
  • fix: soundness issue in Somr

    fix: soundness issue in Somr

    Currently, the following code causes undefined behavior.

        #[test]
        fn test_inner_mut() {
            let five = Somr::new(5);
            fn add(a: &Somr<i32>, b: &Somr<i32>) {
                a.get_mut().map(|x| *x += *b.get().unwrap()).unwrap();
            }
    
            add(&five, &five);
            assert_eq!(five.get().copied().unwrap(), 10);
        }
    
    CleanShot 2023-08-25 at 23 04 22@2x

    The original fn get_mut(&self) -> Option<&mut T> is error-prone, as the compiler cannot prevent programmers from creating multiple mut ref to the same element. Making it unsafe could reduce the scope that needs to be audited.

    opened by zxch3n 0
  • Potential code optimization paths

    Potential code optimization paths

    • [x] map code clean #21
    • [x] narrow Item::{left/right} type from Node to Somr<Item> #24
    • [x] refactor map implementation #26
    • [ ] make Item::Content inner easier to share, especially for Value::Read (perf)
    • [ ] standardize Value::read(Content) for type content reading (perf/correctness)
    • [ ] try separate map/list into branches (mem reduce)
    • [ ] Wrapper Client/Clock in transparent struct for better custom logic and typecheck
    • [ ] Update consumes [u8] instead of Vec<u8>
    enhancement refactory 
    opened by forehalo 0
  • add sub doc callback api

    add sub doc callback api

    In yjs, sub doc is loaded in the following way:

    let doc = new Y.Doc()
    console.log('init main doc');
    
    doc.on('subdocs', ({ loaded }) => {
      loaded.forEach(subdoc => {
        console.log('init subdoc provider');
        new SomethingProvider(subdoc.guid, subdoc)
      })
    })
    
    let subDoc = doc.getMap("map").get("some_sub_doc")
    subDoc.on('synced', () => {
      console.log('synced');
      // when doc synced by provider, we can get sub doc's data
    })
    console.log('loading');
    subDoc.load()
    
    // exec workflow:
    // init main doc -> loading -> init subdoc provider -> synced
    

    In y-octo, we have implemented the parsing of sub doc identifiers in ydoc, this means that to add subdoc support to y-octo, what we need to do is:

    • provide an API to allow doc to know its own synchronization status
    • provide an API, pass in a doc instance, and use external implementation to load the doc binary from anywhere and apply update
    • maintain a thread-safe weak reference to a subdoc instance inside doc
    opened by darkskygit 0
  • add binary format that supports partial reading and self-verification as a storage format

    add binary format that supports partial reading and self-verification as a storage format

    ybinary v1 is a binary format optimized for one-time network transmission.

    It only supports overall reading and cannot know whether binary is damaged before the reading process goes wrong.

    For specific analysis, please refer to this review:

    https://github.com/toeverything/OctoBase/issues/383#issuecomment-1513577058

    We need to design a binary format that supports partial reading and self-verification to store crdt state permanently and robustly

    opened by darkskygit 2
  • add more compatibility test baseline using yjs

    add more compatibility test baseline using yjs

    Currently we still use yrs as a baseline for binary compatible behavior, which increases testing complexity

    I consider introducing a js runtime to provide the execution results of yjs as a reference benchmark during testing

    It may be a convenient choice to directly use node to execute and output to stdio. Another option is to introduce quickjs or a similar streamlined js runtime, but this will increase compilation time and dependency complexity.

    I will first investigate the feasibility of using nodejs to obtain execution results. This is the easiest way

    ### Tasks
    - [x] https://github.com/toeverything/y-octo/pull/4
    - [x] impl the function binding of node: #6
    - [ ] test case for verify the data between yjs and y-octo-node #6
    
    opened by darkskygit 1
  • Verify the correctness and consistency of jwst with yjs

    Verify the correctness and consistency of jwst with yjs

    In the previous iteration, we have implemented most of lib0 and crdt logics:

    • https://github.com/toeverything/OctoBase/issues/383
    • https://github.com/toeverything/OctoBase/issues/403

    We need to verify the correctness and consistency of it with yjs in our new Node.js sync server.

    opened by Brooooooklyn 2
An easy-to-use, high-performance Interledger implementation written in Rust

Interledger implementation in Rust ?? Requirements All crates require Rust 2018 edition and are tested on the following channels: stable Connecting to

Interledger.rs 184 Dec 13, 2022
A high performance blockchain kernel for enterprise users.

English | ็ฎ€ไฝ“ไธญๆ–‡ What is CITA CITA is a fast and scalable blockchain kernel for enterprises. CITA supports both native contract and EVM contract, by whi

CITAHub 1.3k Dec 22, 2022
An extremely high performance matching engine written in Rust.

Galois Introduction Galois is an extremely high performance matching engine written in Rust, typically used for the crypto currency exchange service.

UINB Tech 66 Jan 7, 2023
High Performance Blockchain Deserializer

bitcoin-explorer bitcoin_explorer is an efficient library for reading bitcoin-core binary blockchain file as a database (utilising multi-threading). D

Congyu 18 Dec 22, 2022
A high performance blockchain kernel for enterprise users.

English | ็ฎ€ไฝ“ไธญๆ–‡ What is CITA CITA is a fast and scalable blockchain kernel for enterprises. CITA supports both native contract and EVM contract, by whi

CITAHub 1.3k Dec 22, 2022
An open source Rust high performance cryptocurrency trading API with support for multiple exchanges and language wrappers. written in rust(๐Ÿฆ€) with โค๏ธ

Les.rs - Rust Cryptocurrency Exchange Library An open source Rust high performance cryptocurrency trading API with support for multiple exchanges and

Crabby AI 4 Jan 9, 2023
Glommio Messaging Framework (GMF) is a high-performance RPC system designed to work with the Glommio framework.

Glommio Messaging Framework (GMF) The GMF library is a powerful and innovative framework developed for facilitating Remote Procedure Calls (RPCs) in R

Mohsen Zainalpour 29 Jun 13, 2023
An open source, high performance limit order book for the Seaport smart contracts. Implemented in Rust using ethers-rs, this offers a turnkey option for digital asset marketplaces.

Quay Quay is an open source, high performance backend for the Seaport smart contracts. The project is implemented in Rust, using Postgres as a storage

Valorem Labs Inc. 169 Jun 23, 2023
A high performance Remote Procedure Call system.

A high performance Remote Procedure Call (RPC) system. Usage Add this to your Cargo.toml file. [dependencies] frpc = { git = "https://github.com/nurmo

Nur 5 Jul 28, 2023
Flexible Rust implementation of the MuSig2 multisignature protocol, compatible with Bitcoin.

MuSig2 This crate provides a flexible rust implementation of MuSig2, an optimized digital signature aggregation protocol, on the secp256k1 elliptic cu

null 4 Oct 31, 2023
EVM compatible chain with NPoS/PoC consensus

Reef Chain Reef chain is written in Rust. A basic familiarity with Rust tooling is required. To learn more about Reef chain, please refer to Documenta

Reef Finance 148 Dec 31, 2022
Trustworthy encrypted command line authenticator app compatible with multiple backups.

cotp - command line totp authenticator I believe that security is of paramount importance, especially in this digital world. I created cotp because I

Reply 71 Dec 30, 2022
Decrypt your LUKS partition using a FIDO2 compatible authenticator

fido2luks This will allow you to unlock your LUKS encrypted disk with an FIDO2 compatible key. Note: This has only been tested under Fedora 31, Ubuntu

null 118 Dec 24, 2022
An Ethereum compatible Substrate blockchain for bounties and governance for the Devcash community.

Substrate Node Template A fresh FRAME-based Substrate node, ready for hacking ?? Getting Started Follow the steps below to get started with the Node T

null 4 Mar 30, 2022
Eternally liquid. Forward compatible. Nested, conditional, & Multi-resourced NFTs.

RMRK Substrate Rust Setup First, complete the basic Rust setup instructions. Run Use Rust's native cargo command to build and launch the template node

RMRK Team 67 Dec 25, 2022
Fiddi is a command line tool that does the boring and complex process of checking and processing/watching transactions on EVM compatible Blockchain.

Fiddi is a command line tool that does the boring and complex process of checking and processing/watching transactions on EVM compatible Blockchain.

Ahmad Abdullahi Adamu 7 Jan 9, 2023
Selendra is a multichains interoperable nominated Proof-of-Stake network for developing and running Substrate-based and EVM compatible blockchain applications.

Selendra An interoperable nominated Proof-of-Stake network for developing and running Substrate-based and EVM compatible blockchain applications. Read

Selendra 16 Nov 29, 2022
Minimalistic EVM-compatible chain indexer.

EVM Indexer Minimalistic EVM-compatible blockchain indexer written in rust. This repository contains a program to index helpful information from any E

Kike B 14 Dec 24, 2022
Minimalistic EVM-compatible chain indexer.

EVM Indexer Minimalistic EVM-compatible blockchain indexer written in rust. This repository contains a program to index helpful information from any E

LlamaFolio 11 Dec 15, 2022