The following functions are currently missing:

• [x] acos
• [x] acosh
• [x] asin
• [x] asinh
• [x] atan
• [x] atanh
• [x] cos
• [x] cosh
• [x] exp
• [x] log
• [x] log10
• [x] sin
• [x] sinh
• [x] sqrt
• [x] tan
• [x] tanh

Inspiration for how their implementation might look can be found in Modules/cmathmodule.c (and Modules/clinic/cmathmodule.c.h for argument parsing) of the CPython codebase. Lib/src/stdlib/math.rs equivalent functions should also be roughly similar.

Note to contributors: due to the way tests are organized, you'll need to add your function to the list of tested functions in the test_functions class attribute of CMathTests (on line 58 in Lib/tests/test_cmath.py). I.e, if you want to add log, add it alphabetically at the list used in the comprehension:

# add it to the list
test_functions = [getattr(cmath, fname) for fname in ['cos', ..., 'log', 'sqrt']]

If anyone wants to pick a function or two up, leave a comment naming the functions so other people can see who's working on what. Hopefully that will minimize conflicts.

Summary

I think we have got to the stage we should support threading in RustPython.

Detailed Explanation

There was already some discussion on threading but I wanted to start a dedicated issue as it is a big decision. I will try to break the issue in order to make some order:

Definition of done

We need a user to be able to create a new thread in python and execute code on the same Python object from multiple threads:

import threading

def worker(num):
    """thread worker function"""
    print 'Worker: %s' % num
    return

threads = []
for i in range(5):
    t = threading.Thread(target=worker, args=(i,))
    threads.append(t)
    t.start()

GIL

I think that one of the biggest discussion is the GIL. Should we add a GIL to RustPython? Should we allow only one thread to use each object at a time?

Suggested approach

I suggest the following changes in order to reach the ability to spawn a new thread:

• I suggest we will create a new VirtualMachine for each thread. Thus making VirtualMachine !sync and !Send.
• Use Arc instead of Rc in PyObjectRef. pub type PyObjectRef = Arc<PyObject<dyn PyObjectPayload>>
• PyValue and PyObjectPayload traits will implement Sync and Send. Thus forcing us to make all Py* structs sync as well.
• We will need to convert most of the code Rc, Cell and RefCell use to thread safe options. For example Arc, Mutex and Atomic*.
• When accessing data of an Object wrapped in Mutex we will lock the mutex for our use. This will require careful handling when using internal objects to avoid deadlocks when used between threads.

A simple example of the start_new_thread method in _thread will look something like:

fn start_new_thread(func: PyFunctionRef, args: PyFuncArgs, vm: &VirtualMachine) -> u64 {
    let handle = thread::spawn(move || {
        let thread_vm = VirtualMachine::new(PySettings::default()); // Should get some params from original VM
        thread_vm.invoke(func.as_object(), args).unwrap();
    });
    get_id(handle.thread())
}

Drawbacks, Rationale, and Alternatives

• I suggest that we will ignore the GIL and allow multiple threads to execute python code simultaneously. This can be a big advantage of RustPython on CPython. The user will be expected to make his own code thread safe. This might prove problematic to code that rely on the GIL.

• What about third party library's? We do not yet have an API but would we force them to implement Sync and Send as well?

• There was a lot of talk about alternatives and I cannot find all so please fill free to add in the comments. One alternatives that was suggested is Crossbream.

Unresolved Questions

• How can we do this change in small steps?
• How to test for deadlocks?
• Is this the right time to do this?

cc @satyajeetkolhapure

To add your own modules, you should make your own crate that depends on

# for wasm-bindgen
[lib]
crate-type = ["cdylib", "rlib"]

# can remove `branch="..."` once  this PR is merged
rustpython_wasm = { git = "https://github.com/RustPython/RustPython", branch = "wasm-hooks" }
rustpython_vm = { git = "https://github.com/RustPython/RustPython", branch = "wasm-hooks" }
wasm-bindgen = "0.2"

# might also want this so that compilation doesn't take forever
[package.metadata.wasm-pack.profile.release]
wasm-opt = false

Then your lib.rs should look like:

pub use rustpython_wasm::exports::*;

use wasm_bindgen::prelude::*;
use rustpython_vm::VirtualMachine;

#[wasm_bindgen(start)]
fn init() {
    rustpython_wasm::add_init_func(init_vm);
}

fn init_vm(vm: &mut VirtualMachine) {
    vm.add_frozen(rustpython_vm::py_freeze!(dir = "deps"));
}

deps/ should be a directory next to your Cargo.toml, with a subdirectory deps/pytezos with stuff like deps/pytezos/__init__.py.

I tried to make the code sharing between set and frozenset more nice. I created a shared trait SetProtocol that implements all the shared function. I also moved the set only functions to the new format. I was not sure how to move the new implementation to SetProtocol if anyone has suggestions?

f_trace is initialized with PyNone. I do not know if it would be better on memory/cpu to handle Option::None instead.

No tests for now, just wanting a feedback. Specially on the RwLock used.

   Compiling rustpython-parser v0.1.1 (https://github.com/RustPython/RustPython#a04c19cc)
   Compiling rustpython-compiler v0.1.1 (https://github.com/RustPython/RustPython#a04c19cc)
   Compiling rustpython-derive v0.1.1 (https://github.com/RustPython/RustPython#a04c19cc)
error: Compile error: Got unexpected token '.' at line 1 column 1
##[error]  --> C:\Users\runneradmin\.cargo\git\checkouts\rustpython-f8ef4d934ac33cd8\a04c19c\vm\src\frozen.rs:21:18
   |
21 |     ext_modules!(dir = "Lib/python_builtins/");
   |     -------------^^^^^^^^^^^^^^^^^^^^^^^^^^^^-- in this macro invocation

error: Compile error: Got unexpected token '.' at line 1 column 1
##[error]  --> C:\Users\runneradmin\.cargo\git\checkouts\rustpython-f8ef4d934ac33cd8\a04c19c\vm\src\frozen.rs:27:22
   |
27 |         ext_modules!(dir = "Lib/core_modules/");
   |         -------------^^^^^^^^^^^^^^^^^^^^^^^^^-- in this macro invocation

error: aborting due to 2 previous errors
##[error]aborting due to 2 previous errors
error: could not compile `rustpython-vm`.
##[error]could not compile `rustpython-vm`.
To learn more, run the command again with --verbose.
##[error]The process 'C:\Rust\.cargo\bin\cargo.exe' failed with exit code 101
##[error]Node run failed with exit code 1

All the naming is up for debate, and also the solution for using proc macros inside rustpython_vm. I think it's a pretty good system, and I don't think there are really any other crates trying to use a proc macro in their own crate and in consumer crates.

My work on #3458. Marshaling still needs support for tuples, dicts, sets, etc. But I wanted to post a PR for some feedback (and possibly an early merge).

How It Works

When marshaled, each buffer is prefixed with a byte indicating the type of data. Then while decoding we can check the first byte to see which datatype the buffer holds.

Lists are more complicated because they have different types, and can be recursive (lists inside of lists). Each list has a prefix of its size, and then each element's data is stored directly after another 8 bytes indicating its size. Resulting in [LIST_BYTE, <bytes for size>, <size><element>,<size><element>...]

Already there are some improvement areas. Most notably, each list/element size is stored as usize (u64) and that can probably be changed to a u32.

Testing

I added a python file for testing. It passes for RustPython, marshaling/unmarshaling primitives and lists.

However, CPython marshals their data differently. I am not sure if they are supposed to match, however, I thought it was best to optimize the code to fit the RustPython Datastructures and not worry about how CPython handles marshaling.

Closing

Please let me know if there are any problems or more idiomatic ways of writing this. I've never written rust code outside of personal projects so I expect some issues.

Thanks!

Bug behavior

Added a is_drop field in PyInner. and found out both BufferedReader&FileIO gets double drop? Did some search on the code, no clue where misses a clone() to PyRef or PyObjectRef

Possible root of bug

a obscure memcpy on a struct containing a PyRef/PyObjectRef

Necessity of fixing the bug

Useful for writing garbage collecting, also prevent UB

Part of the log

the remaining of log file just basically repeats those lines. The command is just cargo run, and alright it did run, but with all those double drop: log_buf.log

[ERROR rustpython_vm::object::core] Double drop on PyRef, type="rustpython_vm::stdlib::io::_io::BufferedReader"
[ERROR rustpython_vm::object::core] Double drop on PyRef, type="rustpython_vm::stdlib::io::fileio::FileIO"
[ERROR rustpython_vm::object::core] Double drop on PyObjectRef, typeid=TypeId { t: 14576741135102236696 }, type=Some("rustpython_vm::stdlib::io::fileio::FileIO")
[ERROR rustpython_vm::object::core] Double drop on PyRef, type="rustpython_vm::stdlib::io::_io::BufferedReader"
                    backtrace=   0: <rustpython_vm::object::core::PyRef<T> as core::ops::drop::Drop>::drop
                 at vm/src/object/core.rs:979:17
       1: core::ptr::drop_in_place<rustpython_vm::object::core::PyRef<rustpython_vm::stdlib::io::_io::BufferedReader>>
                 at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/core/src/ptr/mod.rs:486:1
       2: rustpython_vm::stdlib::io::_io::BufferedMixin::close
                 at vm/src/stdlib/io.rs:1575:9
       3: core::ops::function::Fn::call
                 at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/core/src/ops/function.rs:77:5
       4: rustpython_vm::function::builtin::<impl rustpython_vm::function::builtin::sealed::PyNativeFuncInternal<(rustpython_vm::function::builtin::OwnedParam<T1>,),R,rustpython_vm::vm::VirtualMachine> for F>::call_
                 at vm/src/function/builtin.rs:79:17
       5: <F as rustpython_vm::function::builtin::IntoPyNativeFunc<(T,R,VM)>>::call
                 at vm/src/function/builtin.rs:47:9
          rustpython_vm::function::builtin::IntoPyNativeFunc::into_func::{{closure}}
                 at vm/src/function/builtin.rs:35:51
       6: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
                 at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/alloc/src/boxed.rs:1886:9
       7: <rustpython_vm::builtins::builtinfunc::PyBuiltinMethod as rustpython_vm::types::slot::Callable>::call
                 at vm/src/builtins/builtinfunc.rs:210:9
       8: rustpython_vm::types::slot::Callable::slot_call
                 at vm/src/types/slot.rs:562:13
       9: rustpython_vm::vm::vm_object::<impl rustpython_vm::vm::VirtualMachine>::_invoke
                 at vm/src/vm/vm_object.rs:176:30
      10: rustpython_vm::vm::vm_object::<impl rustpython_vm::vm::VirtualMachine>::invoke
                 at vm/src/vm/vm_object.rs:189:9
          rustpython_vm::vm::method::PyMethod::invoke
                 at vm/src/vm/method.rs:121:9
      11: rustpython_vm::vm::vm_object::<impl rustpython_vm::vm::VirtualMachine>::call_method
                 at vm/src/vm/vm_object.rs:127:9
      12: <rustpython_vm::stdlib::io::_io::_IOBase as rustpython_vm::types::slot::Destructor>::slot_del
                 at vm/src/stdlib/io.rs:541:21
      13: rustpython_vm::object::core::PyObject::drop_slow_inner::call_slot_del::{{closure}}
                 at vm/src/object/core.rs:768:33
      14: rustpython_vm::vm::thread::with_vm::{{closure}}
                 at vm/src/vm/thread.rs:38:14
      15: std::thread::local::LocalKey<T>::try_with
                 at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/std/src/thread/local.rs:445:16
      16: std::thread::local::LocalKey<T>::with
                 at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/std/src/thread/local.rs:421:9
      17: rustpython_vm::vm::thread::with_vm
                 at vm/src/vm/thread.rs:27:5
      18: rustpython_vm::object::core::PyObject::drop_slow_inner::call_slot_del
                 at vm/src/object/core.rs:766:23
      19: rustpython_vm::object::core::PyObject::drop_slow_inner
                 at vm/src/object/core.rs:789:13
          rustpython_vm::object::core::PyObject::drop_slow
                 at vm/src/object/core.rs:801:26
      20: <rustpython_vm::object::core::PyObjectRef as core::ops::drop::Drop>::drop
                 at vm/src/object/core.rs:871:22
      21: core::ptr::drop_in_place<rustpython_vm::object::core::PyObjectRef>
                 at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/core/src/ptr/mod.rs:486:1
      22: core::ptr::drop_in_place<core::option::Option<rustpython_vm::object::core::PyObjectRef>>
                 at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/core/src/ptr/mod.rs:486:1
      23: core::ptr::drop_in_place<[core::option::Option<rustpython_vm::object::core::PyObjectRef>]>
                 at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/core/src/ptr/mod.rs:486:1
      24: core::ptr::drop_in_place<alloc::boxed::Box<[core::option::Option<rustpython_vm::object::core::PyObjectRef>]>>
                 at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/core/src/ptr/mod.rs:486:1
      25: core::ptr::drop_in_place<core::cell::UnsafeCell<alloc::boxed::Box<[core::option::Option<rustpython_vm::object::core::PyObjectRef>]>>>
                 at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/core/src/ptr/mod.rs:486:1
      26: core::ptr::drop_in_place<lock_api::mutex::Mutex<parking_lot::raw_mutex::RawMutex,alloc::boxed::Box<[core::option::Option<rustpython_vm::object::core::PyObjectRef>]>>>
                 at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/core/src/ptr/mod.rs:486:1
      27: core::ptr::drop_in_place<rustpython_vm::frame::Frame>
                 at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/core/src/ptr/mod.rs:486:1
      28: core::ptr::drop_in_place<rustpython_vm::object::core::PyInner<rustpython_vm::frame::Frame>>
                 at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/core/src/ptr/mod.rs:486:1
      29: core::ptr::drop_in_place<alloc::boxed::Box<rustpython_vm::object::core::PyInner<rustpython_vm::frame::Frame>>>
                 at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/core/src/ptr/mod.rs:486:1
      30: core::mem::drop
                 at /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/core/src/mem/mod.rs:974:24
      31: rustpython_vm::object::core::drop_dealloc_obj
                 at vm/src/object/core.rs:89:5
      32: rustpython_vm::object::core::PyObject::drop_slow
                 at vm/src/object/core.rs:807:9

Test Code

added is_drop field to PyInner<T> in core.rs This is the git diff patch file(Also added a backtrace crate for debugging in this patch)

drop_guard.patch.txt

And here is core code to check double dropping:

diff --git a/vm/src/object/core.rs b/vm/src/object/core.rs
index 19055df25..0eed91f4a 100644
--- a/vm/src/object/core.rs
+++ b/vm/src/object/core.rs
@@ -31,6 +31,7 @@ use std::{
     any::TypeId,
     borrow::Borrow,
     cell::UnsafeCell,
+    collections::HashMap,
     fmt,
     marker::PhantomData,
     mem::ManuallyDrop,
@@ -38,6 +39,11 @@ use std::{
     ptr::{self, NonNull},
 };
 
+use once_cell::sync::Lazy;
+
+pub static ID2TYPE: Lazy<PyMutex<HashMap<TypeId, String>>> =
+    Lazy::new(|| PyMutex::new(HashMap::new()));
+
 // so, PyObjectRef is basically equivalent to `PyRc<PyInner<dyn PyObjectPayload>>`, except it's
 // only one pointer in width rather than 2. We do that by manually creating a vtable, and putting
 // a &'static reference to it inside the `PyRc` rather than adjacent to it, like trait objects do.
@@ -109,6 +115,7 @@ impl PyObjVTable {
 #[repr(C)]
 struct PyInner<T> {
     ref_count: RefCount,
+    is_drop: PyMutex<bool>,
     // TODO: move typeid into vtable once TypeId::of is const
     typeid: TypeId,
     vtable: &'static PyObjVTable,
@@ -433,6 +440,7 @@ impl<T: PyObjectPayload> PyInner<T> {
         let member_count = typ.slots.member_count;
         Box::new(PyInner {
             ref_count: RefCount::new(),
+            is_drop: PyMutex::new(false),
             typeid: TypeId::of::<T>(),
             vtable: PyObjVTable::of::<T>(),
             typ: PyAtomicRef::from(typ),
@@ -849,7 +857,15 @@ impl<'a, T: PyObjectPayload> From<&'a Py<T>> for &'a PyObject {
 impl Drop for PyObjectRef {
     #[inline]
     fn drop(&mut self) {
+        if *self.0.is_drop.lock() {
+            error!(
+                "Double drop on PyObjectRef, typeid={:?}, type={:?}",
+                self.0.typeid,
+                ID2TYPE.lock().get(&self.0.typeid)
+            );
+        }
         if self.0.ref_count.dec() {
+            *self.0.is_drop.lock() = true;
             unsafe { PyObject::drop_slow(self.ptr) }
         }
     }
@@ -953,7 +969,21 @@ impl<T: PyObjectPayload> fmt::Debug for PyRef<T> {
 impl<T: PyObjectPayload> Drop for PyRef<T> {
     #[inline]
     fn drop(&mut self) {
+        if *self.as_object().0.is_drop.lock() {
+            error!(
+                "Double drop on PyRef, type={:?}",
+                std::any::type_name::<T>()
+            );
+        }
+
+        let tid = TypeId::of::<T>();
+        ID2TYPE
+            .lock()
+            .entry(tid)
+            .or_insert_with(|| std::any::type_name::<T>().to_string());
+
         if self.0.ref_count.dec() {
+            *self.0.is_drop.lock() = true;
             unsafe { PyObject::drop_slow(self.ptr.cast::<PyObject>()) }
         }
     }
@@ -1136,6 +1166,7 @@ pub(crate) fn init_type_hierarchy() -> (PyTypeRef, PyTypeRef, PyTypeRef) {
         let type_type_ptr = Box::into_raw(Box::new(partially_init!(
             PyInner::<PyType> {
                 ref_count: RefCount::new(),
+                is_drop: PyMutex::new(false),
                 typeid: TypeId::of::<PyType>(),
                 vtable: PyObjVTable::of::<PyType>(),
                 dict: None,
@@ -1148,6 +1179,7 @@ pub(crate) fn init_type_hierarchy() -> (PyTypeRef, PyTypeRef, PyTypeRef) {
         let object_type_ptr = Box::into_raw(Box::new(partially_init!(
             PyInner::<PyType> {
                 ref_count: RefCount::new(),
+                is_drop: PyMutex::new(false),
                 typeid: TypeId::of::<PyType>(),
                 vtable: PyObjVTable::of::<PyType>(),
                 dict: None,

The socket module is still incomplete. Many methods and types are missing. To add methods to this module, edit the file vm/src/stdlib/socket.rs accordingly.

I'm not sure about the name; I'm willing to change it to pretty much anything but I can't think of anything else right now. The main goal of this is to have a good Debug implementation for PyObject.payload, as discussed in #664.

Edit: this would close #664.

There's one TODO that I had - I'm not sure how to return different errors. It seems like the compiler always expects DecodeError::InvalidLength errors, so submitting this in a semi-draft state for some guidance. I'm also not super sure how the tests work, but I ran pytest -v in extra_tests/ and they all passed. Please let me know if there's something else that I should do!

Summary

Currently, RustPython takes SIGINT handler regradless of whether it's in repl mode or embedded mode, however in embedded mode, it's reasonable that the embedding application might not want RustPython to take the SIGINT handler, maybe we could add a option to disable SIGINT handler in embedded mode?

Detailed Explanation

Upon inited, RustPython takes SIGINT handler, the related code is basically as follow:

vm::stdlib::make_module->
signal::make_module ->
init_signal_handlers->
libc::signal(SIGINT,<custom handler>)

Drawbacks, Rationale, and Alternatives

Could interfer with code that depend on SIGINT/KeyBoardInterrput, so it might need to be a opt-out option

Unresolved Questions

The CPython tokenizer has both a NEWLINE and NL token.

Given this code:

x = (
  "a"
  "b"
)

x = "a" \
  "b"

RustPython currently outputs the same tokenization for these two snippets (modulo the left and right parens -- notice that in both cases, we have String followed by String):

Name {
    name: "x",
}
Equal
Lpar
String {
    value: "a",
    kind: Normal,
}
String {
    value: "b",
    kind: Normal,
}
Rpar
Newline
Name {
    name: "x",
}
Equal
String {
    value: "a",
    kind: Normal,
}
String {
    value: "b",
    kind: Normal,
}
Newline

Whereas in CPython, there are NL tokens in the first case that aren't present for continuations:

❯ python -m tokenize test.py
0,0-0,0:            ENCODING       'utf-8'
1,0-1,1:            NAME           'x'
1,2-1,3:            OP             '='
1,4-1,5:            OP             '('
1,5-1,6:            NL             '\n'
2,2-2,5:            STRING         '"a"'
2,5-2,6:            NL             '\n'
3,2-3,5:            STRING         '"b"'
3,5-3,6:            NL             '\n'
4,0-4,1:            OP             ')'
4,1-4,2:            NEWLINE        '\n'
5,0-5,1:            NL             '\n'
6,0-6,1:            NAME           'x'
6,2-6,3:            OP             '='
6,4-6,7:            STRING         '"a"'
7,2-7,5:            STRING         '"b"'
7,5-7,6:            NEWLINE        '\n'
8,0-8,0:            ENDMARKER      ''

Since Python 3.11, binascii.a2b_base64() function became to provide strict_mode keyword parameter. ^1

Related links

• Python docs: https://docs.python.org/3/library/binascii.html?highlight=binascii#binascii.a2b_base64
• RustPython implementation: https://github.com/RustPython/RustPython/blob/1d8269fb729c91fc56064e975172d3a11bd62d07/stdlib/src/binascii.rs#L147
• CPython implementation: https://github.com/python/cpython/blob/deaf509e8fc6e0363bd6f26d52ad42f976ec42f2/Modules/binascii.c#L388