I have this error:

"thread '' panicked at 'called Option::unwrap() on a None value', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/yrs-0.7.1/src/transaction.rs:571:59"

when applying this update to a document:

b"\x01\x0e\xde\xd6\xe0\xbe\x0c\t\xc1\xaa\xb7\xc3\xa8\x04s\xaa\xb7\xc3\xa8\x04z\x01\x00\x06\xc1\xde\xd6\xe0\xbe\x0c\t\xaa\xb7\xc3\xa8\x04z\x01\x00\x06\xc7\xc0\xcb\xf4\xe6\r\x96\x01\xab\xf3\xab\xff\r\n\x01(\x00\xde\xd6\xe0\xbe\x0c\x17\x02id\x01w$a584a3d2-2490-4926-b156-0e9fa9b5d718(\x00\xde\xd6\xe0\xbe\x0c\x17\tcell_type\x01w\x04code'\x00\xde\xd6\xe0\xbe\x0c\x17\x06source\x02\x04\x00\xde\xd6\xe0\xbe\x0c\x1a\x052 + 2'\x00\xde\xd6\xe0\xbe\x0c\x17\x08metadata\x01(\x00\xde\xd6\xe0\xbe\x0c \x07trusted\x01x(\x00\xde\xd6\xe0\xbe\x0c\x17\x0fexecution_count\x01}\x0c'\x00\xde\xd6\xe0\xbe\x0c\x17\x07outputs\x00\x08\x00\xde\xd6\xe0\xbe\x0c#\x01v\x04\x0fexecution_count}\x0c\x0boutput_typew\x0eexecute_result\x04datav\x01\ntext/plainw\x014\x08metadatav\x00\x1c\xf6\xe7\x84\x95\x0f\x01\x00\x05\xd4\x80\x96\xbb\x02\x01\x00\r\xa0\x84\x82\xd9\t\x01\x00?\xf9\xa3\xae\xba\x06\x01\x00\x1f\xce\xeb\x90\xaf\x0b\x01\x00\x03\xab\xf3\xab\xff\r\x02\x00\xce\x01\xd5\x01B\xff\xc6\xe3\xe8\x0b\x01\x00\x03\x90\x8b\x8d\xbe\x07\x01\x00\x03\x82\xc0\xd5\xa1\n\x01\x00^\xd0\xa5\xa2\xa7\x02\x01\x00\x03\xfc\xbc\x9a\xa9\x03\x01\x00\x03\xed\xbf\xb4\xd5\x0b\x02\x00\x07\tm\xaa\xb7\xc3\xa8\x04\x02\x00elF\x8c\xe0\xcc\xcc\x03\x01\x00\x95\x01\xc1\x80\xa5\xa4\x07\x02\x00\x03\x17\x92\x02\x9f\xdc\xc6\xa8\x01\x01\x00\x03\xc0\xcb\xf4\xe6\r\x01\x00\xc5\x01\xdf\xdc\xe4\xe2\x0e\x01\x00\x10\xed\xec\x9d\xfd\x08\x01\x00\x03\xa6\xc5\xfd\xa0\x0e\x01\x009\xe0\x93\xc3\xf6\x04\x03\x009J\x01LH\xde\xd6\xe0\xbe\x0c\x02\x00\x04\x06\x11\xe7\xce\xda\xc1\x0f\x01\x00\x03\xb6\xab\xb3c\x01\x00\xd4\x01\x89\xb1\x84\xcd\x0e\x01\x00\x05\xb6\xdb\xec\x87\x05\x01\x00\x03\xff\xe6\xd5\xf2\x0b\x01\x00\x03\xbc\xc5\x89\xcf\x0b\x03\x00\xa4\x04\xbf\x05\x14\xf5\x05\x14"

I'm not sure if you can figure out something wrong with the update itself. Let me know if you need mode details.

I have a pyo3_runtime.PanicException: Defect: item has no parent error while applying the last update in this code:

import y_py as Y

d = Y.YDoc()

x = d.get_text("test")

updates = [
    [1, 2, 208, 216, 199, 216, 11, 0, 40, 1, 5, 115, 116, 97, 116, 101, 5, 100, 105, 114, 116, 121, 1, 121, 40, 1, 7, 99, 111, 110, 116, 101, 120, 116, 4, 112
, 97, 116, 104, 1, 119, 13, 117, 110, 116, 105, 116, 108, 101, 100, 50, 46, 116, 120, 116, 0],
    [1, 1, 208, 216, 199, 216, 11, 2, 4, 1, 6, 115, 111, 117, 114, 99, 101, 1, 32, 0],
    [1, 1, 208, 216, 199, 216, 11, 3, 168, 208, 216, 199, 216, 11, 0, 1, 120, 1, 208, 216, 199, 216, 11, 1, 0, 1],
    [1, 1, 208, 216, 199, 216, 11, 4, 40, 1, 7, 99, 111, 110, 116, 101, 120, 116, 13, 108, 97, 115, 116, 95, 109, 111, 100, 105, 102, 105, 101, 100, 1, 119, 2
7, 50, 48, 50, 50, 45, 48, 52, 45, 48, 54, 84, 48, 55, 58, 53, 49, 58, 52, 57, 46, 51, 50, 49, 53, 54, 52, 90, 0],
    [1, 1, 208, 216, 199, 216, 11, 5, 168, 208, 216, 199, 216, 11, 3, 1, 121, 1, 208, 216, 199, 216, 11, 1, 3, 1],
    [1, 1, 208, 216, 199, 216, 11, 6, 168, 208, 216, 199, 216, 11, 5, 1, 120, 1, 208, 216, 199, 216, 11, 1, 2, 1],
    [1, 1, 208, 216, 199, 216, 11, 6, 168, 208, 216, 199, 216, 11, 5, 1, 120, 1, 208, 216, 199, 216, 11, 1, 5, 1],
    [1, 2, 208, 216, 199, 216, 11, 7, 132, 208, 216, 199, 216, 11, 2, 1, 32, 168, 242, 241, 203, 145, 14, 0, 1, 120, 0]
]

for update in updates:
    Y.apply_update(d, bytes(update))

I'm not sure what's going on, maybe it is not related to y-py, but it's hard to debug. Any idea?

In this example, the following characters are supposed to be appended to ysource: a, b, c, d. But if you run it you can see that only a and b are appended.

import y_py as Y

ydoc = Y.YDoc()
ystate = ydoc.get_map("state")
ysource = ydoc.get_text("source")

updates = [
    [1, 2, 242, 196, 218, 129, 3, 0, 40, 1, 5, 115, 116, 97, 116, 101, 5, 100, 105, 114, 116, 121, 1, 121, 40, 1, 7, 99, 111, 110, 116, 101, 120, 116, 4, 112, 97, 116, 104, 1, 119, 13, 117, 110, 116, 105, 116, 108, 101, 100, 52, 46, 116, 120, 116, 0],
    [1, 1, 242, 196, 218, 129, 3, 2, 40, 1, 7, 99, 111, 110, 116, 101, 120, 116, 13, 108, 97, 115, 116, 95, 109, 111, 100, 105, 102, 105, 101, 100, 1, 119, 27, 50, 48, 50, 50, 45, 48, 52, 45, 49, 51, 84, 49, 48, 58, 49, 48, 58, 53, 55, 46, 48, 55, 51, 54, 50, 51, 90, 0],
    [1, 2, 242, 196, 218, 129, 3, 3, 4, 1, 6, 115, 111, 117, 114, 99, 101, 1, 97, 168, 242, 196, 218, 129, 3, 0, 1, 120, 0],
    [1, 1, 242, 196, 218, 129, 3, 4, 168, 242, 196, 218, 129, 3, 0, 1, 120, 1, 242, 196, 218, 129, 3, 1, 0, 1],
    [1, 1, 152, 182, 129, 244, 193, 193, 227, 4, 0, 168, 242, 196, 218, 129, 3, 4, 1, 121, 1, 242, 196, 218, 129, 3, 2, 0, 1, 4, 1],
    [1, 2, 242, 196, 218, 129, 3, 5, 132, 242, 196, 218, 129, 3, 3, 1, 98, 168, 152, 190, 167, 244, 1, 0, 1, 120, 0],
    [1, 1, 242, 196, 218, 129, 3, 6, 168, 152, 190, 167, 244, 1, 0, 1, 120, 1, 152, 190, 167, 244, 1, 1, 0, 1],
    [1, 1, 242, 196, 218, 129, 3, 7, 132, 242, 196, 218, 129, 3, 5, 1, 99, 0],
    [1, 1, 242, 196, 218, 129, 3, 8, 132, 242, 196, 218, 129, 3, 7, 1, 100, 0],
]

for update in updates:
    Y.apply_update(ydoc, update)
    print(ystate.to_json())
    print(ysource.to_json())

Nooo! https://github.com/y-crdt/ypy/actions/runs/3566940726/jobs/5994217644

I did not see this coming. Apologies, I should have tried to upload an emscripten wheel to test pypi on another project. Looks like https://github.com/pypi/warehouse/issues/10416 is relevant.

I'll revert the wasm build in ci/cd later today if someone else doesn't get to it first.

ERROR    HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/        
         Binary wheel 'y_py-0.5.5-cp310-cp310-emscripten_3_1_14_wasm32.whl' has 
         an unsupported platform tag 'emscripten_3_1_14_wasm32'.                
Error: Process completed with exit code 1.

API Updates

• YMap.delete -> YMap.pop: Reflects the dict api. Pop returns the removed value and throws a key error if it doesn't find a value.
• Added YMap.get: gets value from array with optional default value.
• YArray.delete -> YArray.delete_range
• YArray.insert -> YArray.insert_range: Now takes any iterable
• Added YArray.insert: Inserts a single element at index
• Added YArray.append: adds value to the end of the list
• Added YArray.delete: Deletes individual value

Internel updates

• py_into_any refactored with more explicit error handling. Instead of returning None on error conditions, it now returns either a MultipleIntegrationError when attempting to integrate a non preliminary value and a TypeError when the supplied type cannot be integrated into YCRDT as Any. This centralizes the definition of errors and resolves issues in #46

closes https://github.com/y-crdt/ypy/issues/90.

I believe Pyodide needs a wheel built with a specific version of Python and specific version of emscripten that match the Pyodide version. For latest Pyodide (0.21.3, released Sep 15 2022), and going back through Pyodide 0.20.0 (Aug 9) that is Python 3.10.2 and emscripten 3.14.1. If we wanted to support earlier versions, we would need to review Pyodide changelog to make a matrix of python/emscripten versions to build the wasm wheel with.

See https://github.com/kafonek/ypy/actions/runs/3452230687/jobs/5761992236 for an example of the wheel being built. I don't understand why that's building a wasm wheel for py310 and py311 there but it doesn't hurt anything 🤷‍♂️ .

I've noticed that since I've been using y-py in the back-end in Jupyter, I have threading issues while exiting with Ctrl-C:

Exception ignored in: <module 'threading' from '/home/david/mambaforge/envs/jupyterlab-dev/lib/python3.10/threading.py'>
Traceback (most recent call last):
  File "/home/david/mambaforge/envs/jupyterlab-dev/lib/python3.10/threading.py", line 1560, in _shutdown
    lock.acquire()
KeyboardInterrupt: 
FATAL: exception not rethrown
Aborted (core dumped)

I know that Yrs uses threads, so it might be related.

These updates are supposed to push "b" to the initial content "a" of a YText, but it doesn't work:

import y_py as Y

ydoc1 = Y.YDoc()
ysource1 = ydoc1.get_text("source")

with ydoc1.begin_transaction() as t:
    ysource1.push(t, "a")

updates = [
    [1, 2, 201, 210, 153, 56, 0, 40, 1, 5, 115, 116, 97, 116, 101, 5, 100, 105, 114, 116, 121, 1, 121, 40, 1, 7, 99, 111, 110, 116, 101, 120, 116, 4, 112, 97, 116, 104, 1, 119, 13, 117, 110, 116, 105, 116, 108, 101, 100, 52, 46, 116, 120, 116, 0],
    [1, 1, 201, 210, 153, 56, 2, 168, 201, 210, 153, 56, 0, 1, 120, 1, 201, 210, 153, 56, 1, 0, 1],
    [1, 1, 201, 210, 153, 56, 3, 40, 1, 7, 99, 111, 110, 116, 101, 120, 116, 13, 108, 97, 115, 116, 95, 109, 111, 100, 105, 102, 105, 101, 100, 1, 119, 27, 50, 48, 50, 50, 45, 48, 52, 45, 49, 54, 84, 49, 52, 58, 48, 51, 58, 53, 51, 46, 57, 51, 48, 52, 54, 56, 90, 0],
    [1, 1, 201, 210, 153, 56, 4, 168, 201, 210, 153, 56, 2, 1, 121, 1, 201, 210, 153, 56, 1, 2, 1],
    [1, 2, 201, 210, 153, 56, 5, 132, 228, 254, 237, 171, 7, 0, 1, 98, 168, 201, 210, 153, 56, 4, 1, 120, 0],
    [1, 1, 201, 210, 153, 56, 6, 168, 201, 210, 153, 56, 4, 1, 120, 1, 201, 210, 153, 56, 1, 4, 1],
]

for update in updates:
    Y.apply_update(ydoc1, update)

print(ysource1.to_json())

Alternative to, and closes https://github.com/y-crdt/ypy/pull/86 if you prefer this one.

Use hatch to manage Python virtualenv when developing instead of manually creating one or re-using a system Python.

This is what I'm seeing in the wheel metadata after building

Good morning maintainers. This PR for your consideration moves some metadata from Cargo.toml to pyproject.toml, and adds maturin / pytest as Python dev dependencies in pyproject.toml so that virtualenv management can be handled with poetry.

• poetry install && poetry run maturin develop && poetry run pytest for local development
• I think license will show up in https://pypi.org/project/y-py/ after this (currently does not)

Hi @Waidhoferj.

I noticed the move feature had not yet been implemented here in the python bindings. I trying to implement it since we need it for JupyterLab.

Can you take a look at this? It is the first time I have looked at y-rs and y-py, and the first time I used rust, so I'm unsure if I'm implementing it correctly.

I'm keeping the PR as a draft because I want to implement move_range_to in the same PR, but first, I would like you to take a look at the move_to function in case I'm doing something wrong.

Fixes #98

observe_after_transaction callbacks should only trigger if there are contentful updates to the CRDT. There are certain operations that need a YTransaction to complete, but don't mutate any underlying state. This caused empty updates in observe_after_transaction, which is a meaningless event.

To avoid propagating these noops, @davidbrochart and I added this boilerplate to our callbacks:

def send_updates(self, txn_event: Y.AfterTransactionEvent):
        update = txn_event.get_update()
        if update != b'\x00\x00':
            send_update(update) 

Since we expect this behavior across the board, this PR filters out empty updates before the callback is triggered, so we will no longer have to add this check on a per-case basis.

We are using ypy in a project and in some cases we are getting following error:

thread '<unnamed>' panicked at 'Defect: parent points to a block which is not a shared type', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/yrs-0.12.2/src/block.rs:1128:30

and once this error comes, the document gets corrupted and starts throwing following error:

pyo3_runtime.PanicException: Couldn't get item's parent

Here is how I am processing the message from clients (this is copied from ypy-websocket and modified for our use case):

async def process_message(self, message: bytes, ydoc: Y.YDoc):
        if message[0] != YMessageType.SYNC:
            return
        message_type = message[1]
        msg = message[2:]
        if message_type == YSyncMessageType.SYNC_STEP1:
            state = read_message(msg)
            update = Y.encode_state_as_update(ydoc, state)      **# this is where the error occurs.**
            reply = create_sync_step2_message(update)
            await self.send_message(reply)
        elif message_type in (YSyncMessageType.SYNC_STEP2, YSyncMessageType.SYNC_UPDATE):
            update = read_message(msg)
            # Ignore empty updates (see https://github.com/y-crdt/ypy/issues/98)
            if update == b"\x00\x00":
                return
            try:
                Y.apply_update(ydoc, update)
            except:  # noqa: E722
                # here i am ignoring errors from Rust.
                logger.exception("Bad message - not applying and storing in redis")
            else:
                return update

Before the ypy==0.5.5 the error used to happen only on Y.apply_update(ydoc, update) but now it has started happening inside Y.encode_state_as_update(ydoc, state) also.

I tried to debug it lot but couldn't identify the exact problem. If it helps I am using Lexicaljs as my frontend which generates these updates and I am storing them in redis (also merging updates every 5 minutes when there is no client connected).

This was the last update which might have caused the shared type error:

b'\x01\x01\xea\xe9\x89!:\x84\xea\xe9\x89!9\x01n\x00'

Following is the state and message which caused bad parent afterwards: state: b'\x00' message: b'\x00\x00\x01\x00'

PS: https://github.com/y-crdt/ypy-websocket/blob/main/ypy_websocket/yutils.py#L46 - read_message in above code.

@Waidhoferj There is this code in your drawing example: https://github.com/y-crdt/ypy/blob/792dc96c8a55052b9123f1620856b977234188cf/examples/drawing/client.py#L28-L31

Why are empty updates emitted? This creates issues here and there.

The following example shows that calling to_json() actually generates an update:

import y_py as Y

ydoc = Y.YDoc()
a = ydoc.get_array("a")

def cb(event):
    print("update:", event.get_update())

ydoc.observe_after_transaction(cb)
print("a =", a.to_json())

# update: b'\x00\x00'
# a = []

For a notebook document, it can generate quite a lot of them. I am surprised that it happens for an action that doesn't modify the document. Is it expected? What do these updates correspond to?

cc @ellisonbg

When using the new move feature from a Yjs client, If there is a Ypy client connected, the first client (the Yjs client that moved an element) receives an event undoing the move.

Everything works fine if there is not a Ypy client connected or if the client moving an element is a python client (Ypy) . I suspect the problem comes from my PR for the move feature #83, which I guess is missing handling a moving event coming from another client.

I have been looking into the code to see how Yrs or Ypy handles the events coming from other clients but I couldn't find anything. @Waidhoferj, @Horusiath or @dmonad do you know where should I look in the code to debug it?

Context

To reproduce the bug you can use a Yjs client like:

const doc = new Doc();
const test = doc.getArray('test');
const provider = new WebsocketProvider('ws://localhost:8888', 'rtc_yjs_test', doc);

const observe = (event) => console.log("OBSERVER:", event.changes);
test.observe(observe);

test.push([0,1,2,3,4]);
test.move(0, 2);

and a Ypy client like:

def callbackArray(event):
    print("OBSERVER:", event.delta)

doc = Y.YDoc()
test = doc.get_array('test')
idTest = test.observe(callbackArray)

ws = await connect("ws://localhost:8888/rtc_yjs_test")
WebsocketProvider(doc, ws)

and you can start a WebSocket server with: HOST=localhost PORT=8888 npx y-websocket

When the Yjs client moves an element it receives two events, the first event is the correct move of the element:

{
  "added": {},
  "deleted": {},
  "delta": [
    { "delete": 1 },
    { "retain": 1 },
    { "insert": [0] }
  ]
}

But the second event is reverting the move:

{
  "added": { Item([0]) },
  "deleted": {},
  "delta": [
    { "insert": [0]  },
    { "retain": 1 },
    { "delete": 1}
  ]
}