$ git rev-parse --abbrev-ref HEAD ; git rev-parse HEAD
master
7a4ddb202cb457103827bab91a0bb2f745440b07

$ cargo build
   Compiling peg v0.2.2 (file:///home/n/3p/github.com/kevinmehall/rust-peg)
src/grammar.rs:4:5: 4:29 warning: use of deprecated item: use e.g. i32::from_str_radix, #[warn(deprecated)] on by default
src/grammar.rs:4 use std::num::from_str_radix;
                     ^~~~~~~~~~~~~~~~~~~~~~~~
src/grammar.rs:3970:73: 3970:94 warning: use of deprecated item: use e.g. i32::from_str_radix, #[warn(deprecated)] on by default
src/grammar.rs:3970                                                                         from_str_radix::<u32>(match_str,
                                                                                            ^~~~~~~~~~~~~~~~~~~~~
src/grammar.rs:4046:61: 4046:82 warning: use of deprecated item: use e.g. i32::from_str_radix, #[warn(deprecated)] on by default
src/grammar.rs:4046                                                             from_str_radix::<u32>(match_str,
                                                                                ^~~~~~~~~~~~~~~~~~~~~
src/grammar.rs:4134:97: 4134:118 warning: use of deprecated item: use e.g. i32::from_str_radix, #[warn(deprecated)] on by default
src/grammar.rs:4134                                                                                                 from_str_radix::<u32>(match_str,
                                                                                                                    ^~~~~~~~~~~~~~~~~~~~~
src/grammar.rs:4271:145: 4271:166 warning: use of deprecated item: use e.g. i32::from_str_radix, #[warn(deprecated)] on by default
src/grammar.rs:4271                                                                                                                                                 from_str_radix::<u32>(match_str,
                                                                                                                                                                    ^~~~~~~~~~~~~~~~~~~~~
src/peg_syntax_ext.rs:1:1: 1:107 error: unstable feature
src/peg_syntax_ext.rs:1 #![feature(plugin_registrar, quote, box_syntax, core, collections, rustc_private, box_patterns, str_char)]
                        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
note: this feature may not be used in the beta release channel
error: aborting due to previous error
Could not compile `peg`.

To learn more, run the command again with --verbose.

$ rustc --version
rustc 1.0.0-beta (9854143cb 2015-04-02) (built 2015-04-02)

When inside a precedence!{} block, if you define a rule like this:

l:position!() lhs:(@) "+" rhs:@ r:position!() { /* ... */ }

Instead of:

lhs:(@) "+" rhs:@ { /* ... */ }

It complains with the following error: `@` is only allowed in `precedence!{}`

Span capture is useful when your grammar generates an AST, when each node should have source mapping information, that can be used when validating the semantics of the AST after parsing.

When I try to use the following contents cargo test never returns:

#![feature(plugin)]
#![plugin(peg_syntax_ext)]

peg! line(r#"

#[pub]
line -> Vec<&'input str>
  = line_parts* 

line_parts -> &'input str
  = literal
  / command

#[pub]
literal -> &'input str
  = l:(!"$(" .)* { match_str }

command -> &'input str
  = "$(" c:inside_command ")" { c }

inside_command -> &'input str
  = [a-zA-Z0-9_-]+ { match_str }

"#);

#[test]
fn main() {
  assert_eq!(line::literal("foo ").ok().unwrap(), "foo ");
  assert_eq!(line::literal("foo \n").ok().unwrap(), "foo \n");
  assert_eq!(line::line("foo $(a)").ok().unwrap(), vec!["foo ", "a"]);
}

I think the problem is using the .* expression.

Edit: The example was simplified based on @jonas-schievink 's suggestions.

I'm hitting an error with rust-peg. It's expecting a newline in a statement that shouldn't need one:

main.rs:56:1: 123:5 error: Error at Line 58:12: Expected HashSet {"\u{2029}", "\r\n", "\u{2028}", "\n", "\r"}
main.rs:56 peg! dot(r#"
main.rs:57 #[pub]
main.rs:58 graph -> DotGraph
main.rs:59  = __ "strict"? __ gt:GraphType __ id:ID? __ "{" __ stmts:stmt_list __ "}" __
main.rs:60  {
main.rs:61      DotGraph::new(id, gt, stmts)
           ...

Allow an object passed to the top-level parse call to be threaded through all parse functions and accessible to the action code.

~~~With a #![context(T)] at the top of the grammar source, all exposed parse functions gain an extra argument of type T, which is exposed to actions as the context variable~~~.

Use cases: string interners, source mapping.

This example shows how to parse nested elements using enum. It's basically a very simple XML/HTML parser. The existing examples don't really show how to do this.

I'm trying to describe this regex \d+(\.\d+)? within a single rule, so I can support floats. I can't figure out how to express this without creating an entirely separate rule.

The primary issue seems to be the lack of expression groups, which would enable this: integer() ("." integer())?

Without expression groups, I thought maybe a delimiter with a restriction of a single repetition (integer() ** . but limited to two elements).

If there's an existing solution, that would be ideal.

I found this issue porting from 0.5.0 to 0.6.0.

When I run my parser test, it get stuck in an infinite loop (ie it never terminate).

This is the reduced test case:

extern crate peg;

peg::parser!{grammar p() for str {

pub rule string_literal() -> String
        = "\"" s:$((!['"'])*) "\"" { println!("literal {}", s); s.to_owned() }

}}

fn main() {

    let r = p::string_literal("\"literal\"");

    if let Ok(l) = r {
        println!("Parsed {}", l);
    } else {
        println!("Parse failed {:?}", r);
    }
}

Expected:

• either it doesn't compile because it's a syntax error (i.e. I'm doing the rule wrong with the update syntax)
• or it is a parse error

Instead it just get stuck when I cargo run.

https://github.com/kevinmehall/rust-peg/blob/8098fab1515dc38ca25cdebda9b17655aa5554e1/src/translate.rs#L337

It'd be great to be able to get the line and column of tokens (my AST diagnostics makes heavy use of this).

Is this already exposed in some way?

#position isn't quite what I'm looking as that is just the (byte?) offset into the source string.

This is the only thing I have seen so far that is preventing me from moving from pest to rust-peg. pest exposes this via input().line_col().

#85 is possibly related.

Example grammar:

peg! tsql_grammar(r#"
use ast;

whitespace0 = [ \t\r\n]*
whitespace1 = [ \t\r\n]+

#[pub]
top -> ast::TopStatement
    = kw_top:$("top"i)
      whitespace1
      paren_open:tok_paren_open?
      whitespace0
      lit:lit_i32
      whitespace0
      paren_close:tok_paren_close? {
          if (paren_open.is_some() && paren_close.is_none()) ||
             (paren_open.is_none() && paren_close.is_some()) {
                 return Err(ParseError {
                     // need to be able to create these fields
                 });
          }

          ast::TopStatement {
              top_keyword_pos: ast::Position::from((1, 1)),
              expr: ast::Node {
                  pos: ast::Position::from((1, 1)), // also need position (line, col) here
                  value: ast::Expression::Literal { lit: ast::Literal::Int(lit) },
              },

              paren_open: paren_open,
              paren_close: paren_close,
          }
      }"#);

Actions that consist of returning a single integer are ambiguous with the repetition count syntax introduced by #20.

number -> u32
  = "one" {1}
  / "two" {2}
  / "three" {3}

Those should be actions, but are parsed as repeat counts.

One option that minimally solves the problem is to simply drop x {5} in favor of x {5, 5}. A bare comma is not valid Rust, but the distinction between actions and the remaining { , } forms may be hard for humans to parse.

Does anyone have ideas for a new syntax for the bounded-repeat functionality?

I'm trying to write a parser that matches Lua's long string syntax. Basically, "[" followed by any number of "=" (including 0) followed by "[" opens the string, and "]" followed by the same number of "=" followed by "]" closes it.

Examples of long strings:

• [[simple long string]]
• [=["level 1" long string]]this is still part of the string]==]as is this]=]

I didn't come up with a parser that works, it's probably not even possible. This is what I currently have:

mlstring_char -> char
    = [^\]] { match_str.char_at(0) }

mlstring_inner -> String
    = "=" s:mlstring_inner "="      { s }
    / "[" chars:mlstring_char* "]"  { chars.into_iter().collect() }

string -> String
    = "[" s:mlstring_inner "]" { s }

This obviously doesn't work: Long strings end with the first "]" inside of them. If PEG wasn't greedy, I could just match any char inside mlstring_inner, but rust-peg doesn't seem to have any non-greedy mode. Another way to solve this would be to reference captured strings in the pattern (like some Regex implementations allow). That way, I could just capture "="* in the opening braces, and reference it in the closing brace. The match would only succeed if both strings are the same, ie. if both braces use the same number of "="s. The third way I've come up with is the most flexible solution: Run arbitrary rust code to determine if a rule should match. This way, I could just compare the number of "="s, and match the end of the long string if it has an equal number of "="s.

Is this somehow possible without modifying rust-peg?

The tokens I'm using in my grammar include info like text range, line, column, etc. It would be nice if I could match some of these tokens via the literal syntax and bind the resulting token to a variable for use within the rule's action expression.

Something like:

rule foo() -> u32
    = t:"blah" { t.line }

but when I try something like the above I get "cannot find value `t` in this scope".

I can work around this in a couple ways right now:

// I can use a rule like this:
rule nt() -> Token<'input> = ##next_token()

// using the undocumented ##member() syntax, with a suitable implementation on my impl Parser type:

impl<'input> TokenSlice<'input> {
    pub fn next_token(&self, pos: usize) -> RuleResult<Token<'input>> {
        match self.tokens.get(pos) {
            Some(t) => RuleResult::Matched(pos, *t),
            _ => RuleResult::Failed,
        }
    }
}

// and now I can do like so:
rule foo() -> u32
    = t:nt() "blah" { t.line }

(the above is what I'm using for now)

or

// I could create rules for each token kind:
rule BLAH() -> Token<'input>
    = token:[Token{ kind: TokenKind::Blah, .. }] { token }

rule foo() -> u32
    = t:BLAH() { t.line }

// But that gets tedious as I add new tokens, and I'd rather use the "blah" literal syntax instead of BLAH()

Context

I am writing a basic latex parser. For this I need to parse patterns such as: \textit{some text \texbft{some other text}}. I decided to recognize text by matching any character that is not syntax. With syntax I mean: \textit{ \textbf{and}`.

"Problem"

I was stuck on how to collect characters that are not syntax. Eventually I realized that putting a negative non consuming look-ahead in front of a match anything would get me any char that did not follow the rule.

I now use:

rule char() 
= !(syntax() / eof()) c:[_] { c }

This might be obvious to most users of rust-peg, I have limited exposure writing parsers.

Solution

I think a short example on how to match characters that do not follow some rule could be added to the docs under Pattern expressions. It would fit nicely after the idiom on matching <eof>

Alternatively a dedicated test/example could work. I prefer the first as I think this is an idiom. I am not a domain expert however. I can imagine you want to keep the docs as short as possible. In that case a clearly named test could be a good alternative.

Next steps

Let me know what you think, I am more then happy to open a draft PR for one or both of these options. I will probably need some help to get the terminology correct.

The documentation only shows examples of parsing numbers and single characters. Almost all the tests also don't parse strings, which makes it hard to know what one needs to do, especially if one is a bit of a beginner in Rust and rust-peg.

The information about the 'input lifetime annotation is a bit elusive (not documented), and it is not clear how this affects the lifetime annotations needed for any structs receiving the parsed str or Vec of str.

It would also be great if there were some recommendations as to how strings should be parsed, and if zero-copy can be achieved in any way.

Some proper documentation with a few examples of parsing singular or vec of strings (with operators such as ** and ++) would be really helpful.

Minimal example:

    pub rule mut_prec(a: &mut ()) -> () = precedence! {
        "a" { a; () }
    }

┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈
error[E0507]: cannot move out of `a`, a captured variable in an `Fn` closure
  --> $DIR/rule_args.rs:31:13
   |
30 |     pub rule mut_prec(a: &mut ()) -> () = precedence! {
   |                       -                   ---------- captured by this `Fn` closure
   |                       |
   |                       captured outer variable
31 |         "a" { a; () }
   |             ^^-^^^^^^
   |             | |
   |             | move occurs because `a` has type `&mut ()`, which does not implement the `Copy` trait
   |             move out of `a` occurs here
┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈┈

Discussed in https://github.com/kevinmehall/rust-peg/discussions/312

TODO:

• [ ] documentation
• [ ] change placeholder STACK OVERFLOW expected string to something nicer
• [ ] test failure case: non integer stack limit given

This PR implements a new stack_limit directive in the meta grammar. When specified, the generated parser will keep track of rule invocation depth and error out when the depth is beyond the specified maximum.

I'm not married to any of the naming and syntax here, feel free to suggest better alternatives.

A better way of raising errors from this might be to introduce a new RuleResult variant that immediately terminates parsing instead of carrying on with alternatives. I think that's a great future improvement, but didn't consider for this PR because it would expand the scope quite a bit.

Fixes #282. This PR depends on #307.

This PR introduces the following changes:

• replaces the rustfmt with a project wide cargo fmt
• updates missing formatted source code files from the path peg-macros

Background of this PR:

• I've noticed that some files are not corresponding to the rust format and I want to provide some additions and changes to your project in the near future and in order to keep the change set at minimal level IMHO the current master state should be updated.