Preparing for actors, and multiple "active" evaluation contexts, this PR introduces the Active trait, which provides an abstraction over what we had been calling Core, permitting us to abstract and reuse the current evaluation logic for two distinct situations in the new, redefined Core.

Because we also sometimes borrowed Core as read-only, we also introduce ActiveBorrow to capture that use case.

To do a demo with a single actor, Core needs to "enlarge" its scope to encompass:

• an agent's evaluation context (or several, but only one to start); it activates the actor, and scripts its tests/demos.
• an evaluation context for an actor that is activated by this agent; it provides a public interface, and has an internal store that persists across program edits.

Continues #126, implementing Actors.

This test demonstrates how to reproduce the following steps, all of which now work:

1. agent creates a local actor, with a name Counter.
2. agent sends a local actor Counter a message (calling a method)
3. Counter's methods execute with its internal state, and variables.
4. When each method completes, it responds to the agent, which resumes control.
5. agent upgrades actor Counter by re-using the same name later in the script (proof of concept form of upgrade, for unit testing here)
6. the upgrade "works correctly," in the sense that:
   1. the new method definitions run now, and the old ones are replaced.
   2. the store retains its variable's values from before, after the upgrade.

actor Counter = {
  var x = 0;
  public func get() /*: async Nat*/ { x };
  public func inc() { x := x + 1 };
};
assert (Counter.get() == 0);
Counter.inc();
assert (Counter.get() == 1);
actor Counter {
  var x = 0;
  public func get() /*: async Nat*/ { x };
  public func inc() { x := x + 2 };
};
assert (Counter.get() == 1);
Counter.inc();
assert (Counter.get() == 3);
#ok

A for loop whose iterator is an opaque object (see #120) can avoid the two-phase stepping logic of a general for loop, and even avoid popping the for loop's (new, special) stack frame.

Resolves #7.

Progress:

• [x] Find a good way to convert usize ranges to line/column numbers
• [x] Refactor Exp_, Type_, etc. to include source locations
• [x] Refactor the lexer to use the same source location pattern
• [x] Update usages throughout the codebase to account for these changes
• [x] Update the parser to preserve source locations from tokens

Adds a unit test which displays the status of each base library module. Once we can parse more of the modules, I'll simplify the output to show an aggregated view of the successes and errors.

A very basic, slightly usable, permissive lexer-based Motoko code formatter.

Example provisional CLI usage (using a line width of 30):

cargo run -- format 'actor     Main {public query func main():async Nat{let(a,b)=(1,2)/*{a;a;a/*))))*/;a{{{*/;ignore "abc";"Hello"}}; await Main.hello()' 30

The current formatter "configuration" lives at the bottom of format.rs in a function called get_space(), which determines what type of whitespace should be inserted between two given TokenTree values. This seems to be powerful enough to cover all the syntax in Motoko as long as the token trees are constructed correctly. Depending on how many rules we end up needing overall, I might try refactoring these definitions into a TOML or JSON configuration file.

For our project, we want comments to be retained in the AST, to retain them in the formatted output.

For lalrpop, need to preprocess input to make whitespace in comments parseable. Maybe other steps are needed too.

Consider this program

let rands = func(count){
  var c = 0;
  { next = func() {
     if (c == count) {
       null
     } else {
       c := c + 1;
       let (n, i) = prim "fastRandIterNext" rand_;
       rand_ := i;
       n
     }
    }
  }
};
let j = rands(5);
for (x in j) {
  // Do something iterative with x.
}

(also same as here https://github.com/dfinity/canister-profiling/pull/16/files#diff-ffc8c4b16dc3edfe909f12892610b23d05804b5fe2195d9f544b1933e347d3e0R13 )

When I run this program (no body in the for loop), I get 54 reductions, and 310 total steps before termination.

Most of the steps are evaluating the next function in the VM’s eval logic — What if I were to internalize all of these steps into the Rust side? This PR explores such an idea.

Adds a way to map Rust values directly to their corresponding Motoko values, preserving details such as variant names.

To do: implement a similar Motoko -> Rust conversion to replace the current usage of serde_json::Value.

Includes high-performance Value.to_nat(), Value.to_array(), etc. functions for unwrapping a Value as an alternative to using the (slightly slower) Value.convert() function.

This PR implements modules for the VM. (WIP)

1. For each module, combine names defined in the module to create a valid environment. Reusing the same name is an Interruption
2. combine names in nested modules, and also permit outer names to flow into these internal modules. Unlike inRust, in Motoko, all names in super scopes flow into subscopes.

Unlike other constructs, modules require doing a lot of steps that do not correspond to ordinary execution, but rather, correspond to phases just before or during type checking.

These steps are necessary to resolve identifiers and module projections in real Motoko programs, and involve resolving nested recursive definitions.

Each named module definition introduces a recursive scope that we construct incrementally, in a step-wise manner. When they nest, additional steps propagate the recursion between the nested levels.

Why?

We could eschew a step-based approach, and do the module-level resolution of paths in a batch style, with batch-style errors each time an identifier is used twice. However, that has two drawbacks:

1. Goes against the rest of the project, where we show internal progress at every possible step and interrupt it immediately when there is an issue. A batch algorithm is not like this, and would introduce another "mode" of interacting with programs that happens before they "run step by step". That would complicate things, including the UX design, potentially.
2. If we ever want to improvise with other pre-execution steps, this would introduce a place to do them. Such steps may include experimental intensional features, like let box; as with modules, they would step before the program evaluates.

Scoping using nested modules

Consider this nested module declaration:

module {
  module X {
    /* module Y { }; */ // will shadow Y sibling of X
    public module M {
      func f () : Nat { 
           g();h
           N.g();
           M.g();
           /* X.N.g() */
           x
      };
      public func g () { Y.y(); Y.Z.z() };
    };
    module N {
      public func g () { Y.y(); /* Z.z() /* error. */ */ };
    };  
    public let x = 5;
  };

  module Y {
    public module Z { public func z() { Y.y(); y() } };
    public func y() { X.M.g() };
  };
}

Notice:

• function X.M.g can "see" Y.y and Y.Z.z.
• function Y.Z.z can "see" function "y" and also see the same function as Y.y.
• function X.N.g can "see" Y.y but not Z.z (Y.Z.z() works though)

This example could be smaller and simpler. It's something I have been using as I poke at the compiler to test my understanding of the language, and having it be about this large as been helpful.

• [ ] prim "abs"
• [ ] prim "lsh_Nat"
• [ ] prim "rsh_Nat"
• [ ] prim "idlHash"
• [x] prim "print"
• [ ] prim "trap"
• [ ] prim "rts_version"
• [ ] prim "rts_memory_size"
• [ ] prim "rts_heap_size"
• [ ] prim "rts_total_allocation"
• [ ] prim "rts_reclaimed"
• [ ] prim "rts_max_live_size"
• [ ] prim "rts_callback_table_count"
• [ ] prim "rts_callback_table_size"
• [ ] prim "crc32Hash"
• [ ] prim "num_conv_Int_Int64"
• [ ] prim "num_conv_Int_Int32"
• [ ] prim "num_conv_Int_Int16"
• [ ] prim "num_conv_Int_Int8"
• [ ] prim "num_conv_Nat_Nat64"
• [ ] prim "num_conv_Nat_Nat32"
• [ ] prim "num_conv_Nat_Nat16"
• [ ] prim "num_conv_Nat_Nat8"
• [ ] prim "num_wrap_Int_Int64"
• [ ] prim "num_wrap_Int_Int32"
• [ ] prim "num_wrap_Int_Int16"
• [ ] prim "num_wrap_Int_Int8"
• [ ] prim "num_wrap_Int_Nat64"
• [ ] prim "num_wrap_Int_Nat32"
• [ ] prim "num_wrap_Int_Nat16"
• [ ] prim "num_wrap_Int_Nat8"
• [ ] prim "num_wrap_Int64_Nat64"
• [ ] prim "num_wrap_Nat64_Int64"
• [ ] prim "num_wrap_Int32_Nat32"
• [ ] prim "num_wrap_Nat32_Int32"
• [ ] prim "num_wrap_Int16_Nat16"
• [ ] prim "num_wrap_Nat16_Int16"
• [ ] prim "num_wrap_Int8_Nat8"
• [ ] prim "num_wrap_Nat8_Int8"
• [ ] prim "num_wrap_Char_Nat32"
• [ ] prim "num_conv_Nat32_Char"
• [ ] prim "conv_Char_Text"
• [ ] prim "char_to_upper"
• [ ] prim "char_to_lower"
• [ ] prim "char_is_whitespace"
• [ ] prim "char_is_lowercase"
• [ ] prim "char_is_uppercase"
• [ ] prim "char_is_alphabetic"
• [ ] prim "decodeUtf8"
• [ ] prim "encodeUtf8"
• [ ] prim "text_compare"
• [ ] prim "popcnt8"
• [ ] prim "clz8"
• [ ] prim "ctz8"
• [ ] prim "btst8"
• [ ] prim "popcnt16"
• [ ] prim "clz16"
• [ ] prim "ctz16"
• [ ] prim "btst16"
• [ ] prim "popcnt32"
• [ ] prim "clz32"
• [ ] prim "ctz32"
• [ ] prim "btst32"
• [ ] prim "popcnt64"
• [ ] prim "clz64"
• [ ] prim "ctz64"
• [ ] prim "btst64"
• [ ] prim "popcnt8"
• [ ] prim "clz8"
• [ ] prim "ctz8"
• [ ] prim "btst8"
• [ ] prim "popcnt16"
• [ ] prim "clz16"
• [ ] prim "ctz16"
• [ ] prim "btst16"
• [ ] prim "popcnt32"
• [ ] prim "clz32"
• [ ] prim "ctz32"
• [ ] prim "btst32"
• [ ] prim "popcnt64"
• [ ] prim "clz64"
• [ ] prim "ctz64"
• [ ] prim "btst64"
• [ ] prim "fabs"
• [ ] prim "fsqrt"
• [ ] prim "fceil"
• [ ] prim "ffloor"
• [ ] prim "ftrunc"
• [ ] prim "fnearest"
• [ ] prim "fmin"
• [ ] prim "fmax"
• [ ] prim "fcopysign"
• [ ] prim "num_conv_Float_Int"
• [ ] prim "num_conv_Int_Float"
• [ ] prim "num_conv_Float_Int64"
• [ ] prim "num_conv_Int64_Float"
• [ ] prim "fmtFloat->Text"
• [ ] prim "fsin"
• [ ] prim "fcos"
• [ ] prim "ftan"
• [ ] prim "fasin"
• [ ] prim "facos"
• [ ] prim "fatan"
• [ ] prim "fatan2"
• [ ] prim "fexp"
• [ ] prim "flog"
• [ ] prim "Array.init"
• [ ] prim "Array.tabulate"
• [ ] prim "blobToArray"
• [ ] prim "blobToArrayMut"
• [ ] prim "arrayToBlob"
• [ ] prim "arrayMutToBlob"
• [ ] prim "cast"
• [ ] prim "cast"
• [ ] prim "cast"
• [ ] prim "time"
• [ ] prim "cast"
• [ ] prim "cast"
• [ ] prim "cast"
• [ ] prim "cyclesBalance"
• [ ] prim "cyclesAvailable"
• [ ] prim "cyclesAccept"
• [ ] prim "setCertifiedData"
• [ ] prim "getCertificate"
• [ ] prim "stableMemorySize"
• [ ] prim "stableMemoryGrow"
• [ ] prim "stableMemoryLoadNat32"
• [ ] prim "stableMemoryStoreNat32"
• [ ] prim "stableMemoryLoadNat8"
• [ ] prim "stableMemoryStoreNat8"
• [ ] prim "stableMemoryLoadNat16"
• [ ] prim "stableMemoryStoreNat16"
• [ ] prim "stableMemoryLoadNat64"
• [ ] prim "stableMemoryStoreNat64"
• [ ] prim "stableMemoryLoadInt32"
• [ ] prim "stableMemoryStoreInt32"
• [ ] prim "stableMemoryLoadInt8"
• [ ] prim "stableMemoryStoreInt8"
• [ ] prim "stableMemoryLoadInt16"
• [ ] prim "stableMemoryStoreInt16"
• [ ] prim "stableMemoryLoadInt64"
• [ ] prim "stableMemoryStoreInt64"
• [ ] prim "stableMemoryLoadFloat"
• [ ] prim "stableMemoryStoreFloat"
• [ ] prim "stableMemoryLoadBlob"
• [ ] prim "stableMemoryStoreBlob"
• [ ] prim "stableVarQuery"
• [ ] prim "performanceCounter"

Suppose we introduce two fields, but use an annotation to "hide" the second one:

let r1 : { x : Nat } = { x = 3; y = 5 }; 
r1 == { x = 3 }

The equality check should evaluate to true, but currently we do not respect these type annotations, and the field y will not be hidden.

In Motoko, values of the same (non higher order) type can be compared for equality.

For records with mutable fields, the pointer identity of the field is dereferenced by the official Motoko comparison logic, not used to distinguish records whose values are the same, but whose pointers are not. In those cases, the records are identified by the implementation of ==.

However, in our Rust-based implementation, we currently do not dereference pointers when doing equality checks, but should.

For instance, this check should evaluate to true, but it will not currently:

{ var x = 0 } == { var x = 0 }

The two variables x are allocated in different pointers, but hold the same value.