Prerequisites

Here are a few things you should provide to help me understand the issue:

• Rust version : 1.44
• nom version : nom7
• nom compilation features used: basic

Idea

Nom has many1, many0, and many_m_n functions. Similar with other parts of the API.

What if instead we had a IntoRange trait that took in the different range types and single numbers.

Example:

many(tag("abc"), 0..) -> many0(tag("abc")) 
many(tag("abc"), 1..) -> many1(tag("abc")) 
many(tag("abc"), 1) -> many_m_n(tag("abc"), 1, 1) 

This PR intends to fix #1393.

Description

It replaces existing parser variations that only differ by how many times a subparser may run (like the different many_* variants) with a single parser that takes a range.

In addition to a range parsers can also take a single usize value as parameter which evaluates to a range containing that value only (value..=value). This is primarily done for convenience.

Migration remains fairly simple as the new parsers are a full replacement and just require the respective parameters to be rephrased as a single range.

Reasoning

Using ranges makes the API cleaner by removing unnecessary function duplication. It also allows nom to integrate with more of Rusts language features (specifically the range syntax).

Example

many(2..=4,
  tag("Abcd")
)(i)

With the old parsers this would be expressed using the many_m_n parser:

many_m_n(2, 4,
  tag("Abcd")
)(i)

Special attention should be drawn to the fact that the other variations, such as many0 are also possible by using open ended ranges (0.., 1..).

Notes

The current implementation allows individual parsers to take ranges as parameters but requires them to document how they interpret open ended ranges. This was done to maximize flexibility and allow multiple, potentially contradicting, interpretations to coexist within the codebase.

A good example of this in action is the difference between many and fold:

• many (a.. -> a..=usize::MAX) many packages its results into a Vec. Therefore going beyond the usize limit makes no sense and any open ended range is capped at usize::MAX.
• fold (a.. -> a..=∞) fold has static memory requirements since there is only one accumulator. This accumulator can be used indefinitely and as such the amount of iterations can go beyond even the usize::MAX limit (the current implementations of fold_* also support such cases). Because of this the parser interprets open ended ranges to mean "can run for an infinite number of times".

TODOs

Candidates

The following parsers have been proposed for merging.

• [x] many_* (many0, many1, many_m_n) Resolution: Done
• [x] fold_* (fold_many0, fold_many1, fold_many_m_n) Resolution: Done
• [ ] take_till* (take_till, take_till1)
• [ ] take_while* (take_while, take_while1, take_while_m_n)
• [ ] take_until* (take_until, take_until1)
• [ ] many_till

Open questions

• [x] ~~How should the obsolete parsers be handled? Currently the obsolete parsers (like many_m_n) are deprecated using #[deprecated=""] with a message pointing the developer to the replacement. (Usually in the form of a simple Replaced by <new_parser>). This allows the changes to remain backwards compatible while steering developers towards using the replacement parsers. Tests for the deprecated parsers are left as is but are annotated with #[allow(deprecated)] to suppress the warnings. I would recommend keeping them in for as long as the parsers still exist to make sure that they arent broken accidentally.~~ Resolution: No deprecations for this release. The old parsers and the new range based parsers will exist side-by-side.
• [ ] Resolve open questions of the individual candidates. See here

As said on Gitter, this is the first step for a memchr usage in the FindToken implementation. So far, no performance boost has been observed. Maybe benchmarks are not designed to show this kind of performance change yet.

This is a WIP.

There have been a lot of demands that I change nom's basic type from IResult to std::result::Result. IResult has the following definition:

pub enum IResult<I,O,E=u32> {
  Done(I,O),
  Error(Err<I,E>),
  Incomplete(Needed)
}

This was originally inspired from attoparsec's IResult in which the Partial branch contained a closure to be called when more data is available. For various reasons, I was not able to make the closure idea work (note that Rust was very far from 1.0 at the time), so I chose to show how much data was needed to ask the user to parse again.

I open this issue to study what a change from IResult to Result would entail. I make no promise to do that change, and I will not put the issue to a vote. I will however take into account the responses I get.

The proposal is to make the Incomplete branch part of the Error branch, which would allow employing Result. I do not know yet what the end type would look like. I see two possibilities for the Err type: containing either a Needed or the other error branches, or flattening Needed at the same level.

So, to detail the arguments:

pro:

• can reuse all of the Result methods and the code relying on it
• less surprising for developers
• nom users can easily ignore all of the Incomplete usage, parsers are easier to write
• it might make the macros simpler, because in pattern matching, most combinators would handle 2 cases instead of 4: Done and Error. In some combinators, there will be only three of them: Done, Error or Incomplete(Needed::Unknown), Incomplete::Needed::Size(sz), because the calculation of needed data must still happen
• a lot of nom parsers are stuck on an old version and likely will never update (because it does the job as is), so less code to rewrite, maybe?
• a lot of people find Incomplete confusing, since they work on complete data (like a file completely read in memory)
• I could add in there a "non backtrackable error" that can contain the same thing as a normal error, but would make everything return instead of testing other branches
• it might make compilation faster (compilation time is the reason for the nom fork in syn)

con:

• it means a nearly complete rewrite of nom. It's not necessarily an issue, I'm willing to make the time for it if needed
• there are still a lot of nom users relying on Incomplete, and this is a big breaking change for them
• even parsers that do not use Incomplete would need to be updated to use Result
• it might make some combinators confusing. All of the backtracking combinators like alt! currently return on Incomplete instead of testing the next branch. So, instead of alt! and alt_complete!, do a alt! and alt_incomplete! ?
• nom was designed with streaming in mind, I am worried hiding Incomplete will hide this benefit of the library

also, I am not sure about the timeline here. I am doing nom 2.0 very soon and it introduces some breaking changes, but I'm worried this change might be too big and drive users away. On the other hand, a 3.0 would likely happen far in the future, and there would be even more code (in nom and in code relying on nom) to update.

so, what do people think?

Hi! Long time listener, first time caller. Love nom.

Would you be interested in a PR for a "recipes" section of the docs that contains relatively short snippets to do common tasks? I'm thinking of stuff that's less than a complete example but more than a single combinator:

• [X] EOL comments
• [X] C-style comments, unnested and nested
• [X] whitespace-eating combinators
• [ ] ~Python-style~ ~indentation~ Already exists in the Python parser implementation.
• [X] identifiers
• [x] floats (an arbitrary but named syntax that a user could modify to their language)
• [x] common integer forms (0x3f5a, 34u, 0b11010011, etc.)

Also, are there more detailed docs or examples for verbose errors with row/col info? If not, and if you are agreeable to a PR along those lines, maybe you could point me to a nom parser that uses it, and I can extract bits to make a minimal but thorough tutorial.

After some thought, I reached a satisfying new design for nom 5.0, that I tried in the nomfun repository. This design uses functions instead of macros, with the same signature as macros combinators, mostly having I -> IResult<I,O,E> functions as arguments, and returning other functions, or applying them directly on some input. As an example, here is how the pair combinator would be written:

pub fn pair<I, O1, O2, E, F, G>(first: F, second: G) -> IResult<I, (O1, O2), E>
  where F: Fn(I) -> IResult<I, O1, E>,
        G: Fn(I) -> IResult<I, O2, E> {

  move |input: I| {
    let (input, o1) = first(input)?;
    second(input).map(|(i, o2)| (i, (o1, o2)))
  }
}

pub fn pairc<I, O1, O2, E, F, G>(input: I, first: F, second: G) -> IResult<I, (O1, O2), E>
  where F: Fn(I) -> IResult<I, O1, E>,
        G: Fn(I) -> IResult<I, O2, E> {

  pair(first, second)(input)
}

This way we have two versions, one that combines two parsers and makes another one, and another that can take some input.

The macro version can then be rewritten that way:

macro_rules! pair(
  ($i:expr, $submac:ident!( $($args:tt)* ), $submac2:ident!( $($args2:tt)* )) => (
    pair!($i, |i| $submac!(i, $($args)*), |i| $submac2!(i, $($args2)*))
  );

  ($i:expr, $submac:ident!( $($args:tt)* ), $g:expr) => (
    pair!($i, |i| $submac!(i, $($args)*), $g);
  );

  ($i:expr, $f:expr, $submac:ident!( $($args:tt)* )) => (
    pair!($i, $f, |i| $submac!(i, $($args)*));
  );

  ($i:expr, $f:expr, $g:expr) => (
    $crate::pairc($i, $f, $g)
  );
);

As we can see currently in the 5.0 branch, most combinators are easy to replace:

• extract the macro's code
• put it in a function
• add the proper trait bounds to the function
• replace the macro's code with a function call

The resulting code is functionally equivalent, has less type inference issues and is much faster when built with link time optimization (I have seen results like 20% faster with the new design on some benchmarks).

Another benefit of this system is that it benefits from better import behaviour. Right now, even in edition 2018, macros that are exported are put at the top level (so the module import like macros use is actually a lie). So I cannot make variants of macros 'except by changing their name, to do stuff like separated_list and separated_list_complete. This was an issue because we expect slightly different behaviour in streaming parsers (where we're not sure we'll get the whole data at once) or in complete parsers (where we're sure we have the whole input). In nom 4, I tried to solve this by introducing the CompleteByteSlice and CompleteStr input types, that would behave differently, so you could use the same macros but have different parsers depending on the input. This proved difficult to use, especially considering that we might want to switch back and forth between behaviours (streaming parsers using TLV will know that they have the complete data inside the TLV, complete parsers might want to use methods that work directly on &[u8] and &str. Also, most people did not bother reading the documentation about it and started directly using &[u8] and &str as input when they expected the other behaviour, which resulted in quite some time spent explaining it.

So with functions, we can actually make specialized versions of combinators. We could imagine having streaming and complete versions of many0, tag, etc. And we would let people use those versions by importing them directly (use nom::multi::streaming::many0, etc), and they could even use both versions in the same file.

The downside is that there's an enormous amount of work for this:

• for most combinators (that are not affected by streaming or not), two functions to add:
  • a function returning a closure
  • a function that is called directly
• for combinators affected by streaming, multiply that by 3:
  • a legacy function to keep backward compatibility inside the macro (although I'm not yet sure we should keep the old behaviour, I might just remove CompleteByteSlice and CompleteStr)
  • a function that works in streaming
  • a function that does not work in streaming

These functions will also require their own documentation and tests, and all of nom's documentation and examples should probably be adapted to this. I'm making steady progress on converting the combinators, but there's still a lot to do. (TODO: make a checklist of which combinators were ported over or not)

Questions I have to solve now:

• do I keep nom 4's behaviour with CompleteByteSlice and CompleteStr ?
• do I keep macros related to streaming backward compatible with nom 4?
• if not, which behaviour should I go with? Streaming or not?
• do I make two versions of each combinators, or can I assume people will be able to write pair(first, second)(input) without any issues?
• how do I port combinators that have a variable number of arguments, like do_parse (this one could probably be written directly with the ? syntax like this: https://github.com/Geal/nomfun/blob/master/benches/http.rs#L93-L102 ), tuple, permutation, alt, switch?
• how do I reimplement ws?
• should I extract method and ws in their own crates? They're not strictly necessary to nom and would make sense as separate libraries

Hello there, I'm trying to build nom for a RISC-V target which will use no_std. But after adding it to my Cargo.toml rust complained that a dependency required std.

However looking at the previous issues, I found https://github.com/Geal/nom/issues/1370 and saw that there was a fix so I switched my dependency to point directly at github:

[dependencies.nom]
# version = "7.0.0"
git = "https://github.com/Geal/nom"
default-features = false
features = []

However this is still producing:

error[E0463]: can't find crate for `std`
  |
  = note: the `riscv32imac-unknown-none-elf` target may not support the standard library
  = note: `std` is required by `memchr` because it does not declare `#![no_std]`
  = help: consider building the standard library from source with `cargo build -Zbuild-std`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0463`.
error: could not compile `memchr`

Looking at the nom Cargo.toml features, it seems like turning off all default features (like I'm doing above) should turn off std in all deps. And it did help, as I fixed the error I was getting in the issue above but I now get an error for memchr.

Any ideas why setting default-features = false wouldn't do the trick for memchr?

alpha1 should accept any kinds of alphabetical characters, and because &strs natively support UTF-8, it is counterintuitive and just less useful that alpha1 only takes characters a-zA-Z (correct me if I'm mistaken).

I am exploring the possibility of switching to nom in a project I am working on. I am not fully familiar with nom yet, so please bear with me.

For starters, I was trying to come up with a parser that matches strings of the form [a-zA-Z][-a-zA-Z0-9_]*. I wrote this:

#[macro_use]
extern crate nom;

use std::str::from_utf8;

use nom::{alpha, alphanumeric};
use nom::{IResult, Needed};
use nom::IResult::*;

named!(identifier<&[u8], String>,
       chain!(
           h: map_res!(alpha, from_utf8) ~
           t: many0!(alt!(alphanumeric | tag!("-") | tag!("_"))),
           || {
               let  s = h.to_string();
               t.into_iter().fold(s, |mut accum, slice| {
                   accum.push_str(from_utf8(slice).unwrap()); accum })}));

And I tested it with:

    #[test]
    fn id_name() {
        let a_setting = &b"miles"[..];
        let res = setting_name(a_setting);
        assert_eq!(res, Done(&b""[..], "miles".to_string()));
    }

When I run cargo test my PC completely hangs. With top I can see that it starts consuming more and more CPU and memory until the entire system is completely unusable and I have to hard reset.

Am I doing something wrong? Is this the best way to make a parser to match this type of strings?

• Err enum is a terrible name very confusing
• Err is not in error module
• ErrorConvert is not in error module
• IResult is not in error module
• Needed is not in error module
• both to_owned() method now make no sense in v6 https://docs.rs/nom/6.2.1/nom/enum.Err.html#method.to_owned
• no to_owned() wait to convert a Err<Error<&str>> to Err<Error<String>>
• convert() from Error seem useless for user https://docs.rs/nom/6.2.1/nom/trait.ErrorConvert.html#tymethod.convert
• most utils method don't work with new V6 default error type.

And that just the few problem I hit doing a little parsing of bearer authorization http today. Keep it simple please, I already raise this problem several time in the past, it's worse in V6 ! Or I'm missing and so please enlighten me https://github.com/Geal/nom/blob/master/doc/error_management.md doesn't seem up to date.

We really need clear example of how to make error type owned, clear guide line on how nom parser should handle error properly. Maybe some idea here https://github.com/rust-lang/project-error-handling

• nom version: 6.2.1
• nom compilation features used: none

The following code:

fn main() {
  let res: nom::IResult<&str, Vec<char>> =
    nom::multi::many_m_n(4, 2, nom::character::complete::char('a'))("aaa");
  dbg!(res);
}

will succeed, consuming two 'a':

[src/main.rs:4] res = Ok(
    (
        "a",
        [
            'a',
            'a',
        ],
    ),
)

While it is unlikely that someone would hardcode a minimum value greater than a maximum value, they can be the result of more complex code, and a call to many_m_n in such conditions should probably systematically fail (as the constraint of parsing something at least min but at most max times would be impossible to satisfy).

Have a nice day

this reduces duplication a bit

the implementation comes from:

• https://github.com/epage/nom-experimental/pull/2
• https://github.com/epage/nom-experimental/pull/25

This is a combinator I use all the time, might be useful to see something like it in this crate.

It drops a byte at a time until the given parser matches, then returns the result.

I don't do parsing in any really performance sensative contexts, this can probably be better implemented. This impl demonstrates the idea.

fn drop_until<'a, T>(
    parser: fn(&'a str) -> IResult<&'a str, T>,
) -> impl FnMut(&'a str) -> IResult<&'a str, T> {
    map(many_till(take(1u8), parser), |(_, matched)| matched)
}

If you use char parser, nom automatically reports the expected char but many other parsers (e.g. one_of) don't. This leads to the use of alt((char(.), char(.))) in places where a one_of would make more sense / be cleaner.

Ideally, the error reporting would be made more flexible, which will be made more complicated by #1580.

A (very)( rough proposal

• Split out VerboseErrorKind::Char to an enum Value with
  • Token(T)
  • Set(<I as IntoOutput>::Output)
  • Range(T.=T)
  • Next(usize) (how much of the Input is to be included)
  • Descriptor(&'static str) (for more descriptive names)
• Rename FindTokens to TokenPattern to be more explicit in how its used
• Add a TokenPattern::values iterator that returns all of the error reporting Values contained in it
• Add TokenPattern::named that creates a NamedPattern that exclusive returns Descriptor
• Extend the Error trait for adding expected and unexpected "Value"s
• Add Parser::expected and Parser::unexpected combinators to allow a user to set this on any combinator explicitly

Some inspiration is coming from combine

• https://docs.rs/combine/latest/combine/trait.Parser.html#method.expected
• https://github.com/Marwes/combine/issues/333

I'm doing the example from the docs here: https://docs.rs/nom/latest/nom/combinator/fn.map_res.html

let mut parse = map_res(digit1, |s: &str| s.parse::<u8>());

That gives:

error[E0283]: type annotations needed
   --> src/main.rs:31:24
    |
31  |     let base = map_res(digit1, |s: &str| s.parse::<u32>());
    |                ------- ^^^^^^ cannot infer type of the type parameter `E` declared on the function `digit1`
    |                |
    |                type must be known at this point
    |
    = note: cannot satisfy `_: FromExternalError<&str, ParseIntError>`
note: required by a bound in `map_res`

No idea why the example from the documentation would be failing here.

I keep getting this error; Err value: Error(())` Perhabs am doing something wrong. Please advise. Thanks in advance.

parse_string taken from the examples

fn main() {
  let dd = "\u{1e}";
  let result = parse_string::<()>(dd);
  println!("Result:\n\n{}", result.unwrap().1);
}