It summarizes what we called the "non-consuming loop analysis" and the left-recursion analysis. Consult the article from Ford, section "Well-formed Grammars". It closes #22 #23 #24.

The following example gives the type ((String, PExpr), PExpr), we would like instead (String, PExpr, PExpr).

let_expr = let_kw let_binding in_kw expression
let_binding = identifier bind_op expression

Instead of:

 assert_eq!(calculator::parse_expression("7+(7*2)", 0).unwrap().data, 21);

We should propose a version without the 0 offset parameter:

assert_eq!(calculator::parse_expression("7+(7*2)").unwrap().data, 21);

I just started using oak. Probably I'm doing something wrong, but the following struck me as odd:

Code:

#![feature(plugin)]
#![plugin(oak)]

extern crate oak_runtime;
use oak_runtime::*;

grammar! my_grammar{
  quotation_mark = dquote / ("%22" > char_quotation_mark)
  dquote = ["\""] // %x22

  fn char_quotation_mark() -> char { '\"' }
}

fn main() {
  let state = my_grammar::parse_quotation_mark("%22".into_state());
  println!("{:?}", state.unwrap_data());
}

Output from from running cargo run

   Compiling oak_test v0.1.0 (file:///home/tstorch/src/oak_test)
error: Type mismatch between branches of the choice operator.
 --> src/main.rs:8:20
  |
8 |   quotation_mark = dquote / ("%22" > char_quotation_mark)
  |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |
note: char
 --> src/main.rs:8:20
  |
8 |   quotation_mark = dquote / ("%22" > char_quotation_mark)
  |                    ^^^^^^
note: char
 --> src/main.rs:8:36
  |
8 |   quotation_mark = dquote / ("%22" > char_quotation_mark)
  |                                    ^^^^^^^^^^^^^^^^^^^^^^

error: aborting due to previous error

error: Could not compile `oak_test`.

As both sides are char. This leaves me wondering a bit. Is this a bug or intentional behavior?

cargo and rust version:

> rustc --version
rustc 1.19.0-nightly (5b13bff52 2017-05-23)
> cargo --version
cargo 0.20.0-nightly (397359840 2017-05-18)

Update: If I do this it works "just fine":

#![feature(plugin)]
#![plugin(oak)]

extern crate oak_runtime;
use oak_runtime::*;

grammar! sum{
  quotation_mark = dquote / dquote_encoded
  dquote = "\"" > char_quotation_mark
  dquote_encoded = "%22" > char_quotation_mark

  fn char_quotation_mark() -> char { '\"' }
}

fn main() {
  let state = sum::parse_quotation_mark("%22".into_state());
  println!("{:?}", state.unwrap_data());
}

Somehow the bracket notation does not work well with functions returning a char... My current workaround is doing something like this in order to circumvent this:

a_to_f = ["A-F"] > char_id
fn char_id(c: char) -> char { c }

For example r1 / r2 in a typed context cannot be typed because we don't have name for the sum type. However r1 > make_r1 / r2 > make_r2 can be typed but the return types of make_r1 and make_r2 must be the same.

Hello, This is very cool, though I am not sure if I get it correctly. Does it mean that we the help of rust.peg, one could generate valid input sequences for a give grammar? For example, given a grammar for a calculator, then it can randomly generate input strings like '2+3'...?

Why string literal parsed to -> (^)? It makes no sense. If I want use string literal I need to look ahead with &"literal" and after get tuple of chars with same length and write a function to convert tuple of chars to vec or string or whatever. But if string literal return a value it can be easy dropped with -> (^) or maybe add new operator which can look ahead and consume value.

I tried realise it, but I don't really understand how oak works.

Sorry for my bad english.

When you write the context actions (such as e > f) in the grammar, they actually must be defined (not only declared) inside the grammar! macro. But it becomes uncomfortable when the grammar is large (for example, one for a whole programming language may require several tens of actions). So, it's currently impossible to write like this:

// in actions/part1.rs
pub fn action1() {
    // ...
}
// in actions/part2.rs
pub fn action2() {
    // ...
}

// in grammar
grammar! test {
    use super::actions::part1::*;
    use super::actions::part2::*;
    // ...
    some_test = e1 > action1 / e2 > action2
    // ...
}

So, for large grammar, allowing use functions from other modules would be useful.

Extend the parser with "tree-shape" atoms. This will allow to provide pattern matching on enum type and for example, to directly work with the Rust compiler Lexer. Several notes:

• We require rules and rust functions to start with a lower case (rules' names and semantic action) to distinguish these from tree-shaped data.
• Tree-shaped data structure start with an upper case. It would also work in semantic action, as suggested in #53.
• ~~We leave the type checking of the branches of the choice combinator (e1 / e2, T1 must equals T2) to the Rust compiler. It closes #73.~~ Finally, we chose to check the branches.
• We allow atom identifiers to be parsed by the host-language rule. The purpose is to allow identifiers with a path such as in mod::BinOp(mod2::Plus). It closes #54.
• We require the function parsing these "full-identifiers" to return the "core identifier" such as BinOp for mod::BinOp so we can distinguish function/rules VS. tree-data.

Some references on this topic of tree-pattern matching with PEG:

• Warth, Alessandro. “Experimenting with Programming Languages,” 2009.
• Warth, Alessandro, and Ian Piumarta. “OMeta: An Object-Oriented Language for Pattern Matching.” In Proceedings of the 2007 Symposium on Dynamic Languages, 11–19. ACM, 2007.

It's my first time in the internals of Rust, so I may have made a mistake here, but the code compiles and the tests all pass. The only thing I'm really unsure about is whether I'm using the correct constant for the ctxt member of the syntax::codemap::Span struct, but NO_EXPANSION probably seems the safest for now.

TL;DR: Because oak_runtime includes extern crate syntax, but is a runtime crate rather than a compiler plugin, rustc produced a dynamically linked executable instead of a statically-linked one, which makes distribution difficult/impossible.

According to Alex Chrichton at https://github.com/rust-lang/rust/issues/32996, it is intended behaviour for rustc to switch to using dynamic linking when syntax is referred to in a project, because the rlib for that crate is not included with the Rust standard library distribution, but only a .so library, presumably on the assumption that only the compiler will need to dynamically link to that library when it loads a compiler plugin.

However, because of Rust's linking strategy resolution (as detailed in the comments above), any of the final libraries referring to it will cause Rust to decide to dynamically link all libraries for the executable produced.

This becomes a problem at it precludes the use of the x86_64-unknown-linux-musl architecture, which is useful for packaging up the Rust executable with the very lightweight Alpine Linux in a docker container for deployment, or even using a full distribution like Ubuntu, as the produced executable won't run without the dynamic libraries in the rust toolchain - again making it almost impossible to distribute. In effect, any rust program that includes oak_runtime can only be executed with cargo run, which is not a suitable thing for production systems.

It seems the commit which introduced this issue is https://github.com/ptal/oak/commit/41e355feacd049cab6a525210f06eebf9d73633a

I'd really like to use oak in Differential Datalog, but unfortunately it has the requirement of stable Rust, allowing the nightly features to be toggled with a Cargo feature or removing them entirely to allow stable compilation would be really amazing

Initially, #80 was opened to be able to directly parsed on the rust token stream and, taking advantages of techniques already used in OMeta, being able to parse on arbitrary tree structure. However we are in a typed context and it seems more difficult than what I first thought. Similarly to #92, we propose to use the string literal to parse on enum's variants.

Design

• Provide a stream structure encapsulating the Rust parser, it will also be useful to interact with external parser.
• Implement the trait ConsumePrefix such that it consumes one or more tokens depending on the prefix.
• A literal, for example "=" will correspond to the token Eq. If the literal cannot be mapped, then an assert will be triggered at runtime. General parsing on tree proposed by #80 would avoid this runtime error, however, for now, we believe it's easy enough to find and fix.
• Tokens containing data must be parsed with an external parser (Ident(..), Literal(..), Lifetime(..),....) if you want to extract the contained data. Otherwise, the token data will be compared against the string literal. Note that Ident("a_variable") does not match "a_var" "iable" but only "a_variable".

A list of expressions is defined by e % sep which is equivalent to e (sep e)* in term of parsing but not equivalent in term of types. Indeed the first expression will have the type Vec<E> while the second (E, Vec<E>) or (E, Vec<(E, Sep)>) if the separator has a type different from () or (^).

• e % sep has the type Vec<E> and sep must have the type () (or (^)) or a typing error should be exposed.

For example take the grammar

predicate = (!"b" .)+ 

with the input b, it will output the wrong message:

unexpected `b`, expecting .

Other example could be find as well, this is because when logging errors we do not know we under in a not predicate.

Instead of building errors in different way and everywhere in the code, we take the approach of implementing a file error.rs in each module (such as more or less done in rustc). We use an error enum such as:

enum TypingError {
  TypeMismatch(TypeMismatchInfo),
  ...
}

struct TypeMismatchInfo {
  span: Span;
  ...
}

Each error has a struct containing the relevant information for later printing the error. An error enum must implement a trait to retrieve an error code:

trait ErrorCode {
  fn error_code(&self) -> String;
}

That returns the name of the variant, here TypeMismatch. It is used in the error attribute #29.

Each error structure must implement a trait for printing the error:

trait DisplayError {
  fn display_error(&self, cx: &ExtCtxt);
}

This is flexible enough to allow different kind of printing, even for a JSON output or for deep explanation (as with --explain in rustc).