A procedural macro for defining nom combinators in simple DSL

Andy Lok

Last update: Dec 12, 2022

Related tags

Parsing nom-rule

Overview

nom-rule

A procedural macro for defining nom combinators in simple DSL. Requires nom v5.0+.

Dependencies

[dependencies]
nom = "7"
nom-rule = "0.2"

Syntax

The procedural macro rule! provided by this crate is designed for the ease of writing grammar spec as well as to improve maintainability, it follows these simple rules:

TOKEN: match the token by token kind. You should provide a parser to eat the next token if the token kind matched. it will get expanded into match_token(TOKEN).
";": match the token by token text. You should provide a parser to eat the next token if the token text matched. it will get expanded into match_text(";") in this example.
#fn_name: an external nom parser function. In the example above, ident is a predefined parser for identifiers.
a ~ b ~ c: a sequence of parsers to take one by one. It'll get expanded into nom::sequence::tuple.
(...)+: one or more repeated patterns. It'll get expanded into nom::multi::many1.
(...)*: zero or more repeated patterns. It'll get expanded into nom::multi::many0.
(...)?: Optional parser. It'll get expanded into nom::combinator::opt.
a | b | c: Choices between a, b, and c. It'll get expanded into nom::branch::alt.
&a: Positive predicate. It'll get expanded into nom::combinator::map(nom::combinator::peek(a), |_| ()). Note that it doesn't consume the input.
!a: Negative predicate. It'll get expanded into nom::combinator::not. Note that it doesn't consume the input.
... : "description": Context description for error reporting. It'll get expanded into nom::error::context.

Example

Define match_text parser and match_token parser for your custom token type. You can use nom::combinator::fail as match_token if your parser use &str or &[u8] as input because you won't match on token kinds.

#[derive(Clone, Debug, PartialEq)]
struct Token<'a> {
    kind: TokenKind,
    text: &'a str,
    span: Span,
}

#[derive(Clone, Copy, Debug, PartialEq)]
enum TokenKind {
    Whitespace,

    // Keywords
    CREATE,
    TABLE,

    // Symbols
    LParen,
    RParen,
    Semicolon,
    Comma,

    Ident,
}

fn match_text<'a, Error: ParseError<'a>>>(
    text: &'a str,
) -> impl FnMut(Input<'a>) -> IResult<'a>, &'a Token<'a>, Error> {
    move |i| satisfy(|token: &Token<'a>| token.text == text)(i)
}

fn match_token<'a, Error: ParseError<'a>>>(
    kind: TokenKind,
) -> impl FnMut(Input<'a>) -> IResult<'a>, &'a Token<'a>, Error> {
    move |i| satisfy(|token: &Token<'a>| token.kind == kind)(i)
}

Then give the two parser to nom_rule::rule! by wrapping it into a custom macro:

macro_rules! rule {
    ($($tt:tt)*) => { 
        nom_rule::rule!($crate::match_text, $crate::match_token, $($tt)*)
    }
}

To define a parser for the SQL of creating table:

let mut rule = rule!(
    CREATE ~ TABLE ~ #ident ~ "(" ~ (#ident ~ #ident ~ ","?)* ~ ")" ~ ";" : "CREATE TABLE statement"
);

It will get expanded into:

let mut rule = 
    nom::error::context(
        "CREATE TABLE statement",
        nom::sequence::tuple((
            (crate::match_token)(CREATE),
            (crate::match_token)(TABLE),
            ident,
            (crate::match_text)("("),
            nom::multi::many0(nom::sequence::tuple((
                ident,
                ident,
                nom::combinator::opt((crate::match_text)(",")),
            ))),
            (crate::match_text)(")"),
            (crate::match_text)(";"),
        ))
    );

Auto Sequence (nightly only)

nom-rule is able to automatically insert ~ in the rule when necessary so that you get the example above working the same as the following:

let mut rule = rule!(
    CREATE TABLE #ident "(" (#ident #ident ","?)* ")" ";" : "CREATE TABLE statement"
);

To enable this feature, you need to use a nightly channel rust complier, and add this to the Cargo.toml:

nom-rule = { version = "0.2", features = ["auto-sequence"] }

A procedural macro for configuring constant values across crates

toml-cfg Rough ideas: Crates can declare variables that can be overridden Anything const, e.g. usize, strings, etc. (Only) The "root crate" can overri

43 Dec 24, 2022

Procedural macro to derive Serde serializer-deserializer for Prost

prost-serde-derive prost-serde-derive is a procedural macro to generate Serde serializers and deserializers for Prost-generated structs. Rationale Cur

4 Dec 15, 2022

A procedural macro to generate a new function implementation for your struct.

Impl New 🦀 A procedural macro to generate a new function implementation for your struct. 🚀 Add to your project Add this to your Cargo.toml: [depende

4 Sep 8, 2023

A procedural macro that copy-pastes match arms for new type variant enums.

All the same! If you ever had code that looks like this: use std::io; use std::pin::Pin; use std::task::{Context, Poll}; use tokio::io::AsyncWrite; us

15 Feb 20, 2024

Parse command line arguments by defining a struct.

StructOpt Parse command line arguments by defining a struct. It combines clap with custom derive. Documentation Find it on Docs.rs. You can also check

2.6k Jan 5, 2023

This is an implementation defining standard for client-side-validation

Client-side-validation Foundation Libraries This is an implementation defining standard of client-side-validation representing a set of its Foundation

8 Dec 3, 2022

A Rust crate for LL(k) parser combinators.

oni-comb-rs (鬼昆布,おにこんぶ) A Rust crate for LL(k) parser combinators. Main project oni-comb-parser-rs Sub projects The following is projects implemented

24 Nov 3, 2022

Composable n-gram combinators that are ergonomic and bare-metal fast

CREATURE FEATUR(ization) A crate for polymorphic ML & NLP featurization that leverages zero-cost abstraction. It provides composable n-gram combinator

3 Aug 25, 2022

PEG parser combinators using operator overloading without macros.

pom PEG parser combinators created using operator overloading without macros. Document Tutorial API Reference Learning Parser Combinators With Rust -

412 Dec 31, 2022

Attribute for defining `macro_rules!` macros with proper visibility and scoping

macro-vis This crate provides an attribute for defining macro_rules! macros that have proper visibility and scoping. The default scoping and publicity

2 Aug 29, 2022

A tool for defining and running multi-container Docker applications

Ikki Ikki is a tool for defining and running multi-container Docker applications. It is similar to Docker Compose but comes with some differences. Goa

39 Dec 21, 2022

ShakeFlow: Functional Hardware Description with Latency-Insensitive Interface Combinators

ShakeFlow: Functional Hardware Description with Latency-Insensitive Interface Combinators This repository contains the artifact for the following pape

KAIST Concurrency & Parallelism Laboratory

36 Feb 16, 2023

AI-TOML Workflow Specification (aiTWS), a comprehensive and flexible specification for defining arbitrary Ai centric workflows.

AI-TOML Workflow Specification (aiTWS) The AI-TOML Workflow Specification (aiTWS) is a flexible and extensible specification for defining arbitrary wo

20 Apr 8, 2023

Crate extending futures stream combinators, that is adding precise rate limiter

Comments

Complete example

Is there any way that you could put the example into a separate module? I can't get either the example in the readme or the tests module to compile.

error: proc macro panicked
  --> src/main.rs:71:9
   |
71 |           nom_rule::rule!($crate::match_text, $crate::match_token, $($tt)*)
   |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...
76 |       let mut rule = rule!(
   |  ____________________-
77 | |         CREATE ~ TABLE ~ #ident ~ ^"(" ~ (#ident ~ #ident ~ ","?)* ~ ")" ~ ";" : "CREATE TABLE statement"
78 | |     );
   | |_____- in this macro invocation

   = help: message: called `Option::unwrap()` on a `None` value
   = note: this error originates in the macro `rule` (in Nightly builds, run with -Z macro-backtrace for more info)

Also is it possible to use rules inside of other rules?

opened by slyons 3