NixEl is a Rust library that turns Nix code into a variety of correct, typed, memory-safe data-structures

Kevin Amado

Last update: Dec 29, 2022

Related tags

Data structures parser nix nixos lexer abstract-syntax-tree concrete-syntax-trees

Overview

🐉 NixEL

Lexer, Parser, Abstract Syntax Tree and Concrete Syntax Tree for the Nix Expressions Language.

NixEl is a Rust library that turns Nix code into a variety of correct, typed, memory-safe data-structures.

It's based on the original lexer and parser of Nix, using the same algorithms and methods, which means guaranteed correctness. Additionally, it's been tested against every file in Nixpkgs, and also continuously on our infrastructure.

It also provides a command line application called $ nixel which calls the library and pretty-prints the results.

For example, given some code:

let
  greeting = "Hello World!";
in
  greeting

NixEL can generate an Abstract Syntax Tree for you:

LetIn {
    bindings: [
        KeyValue(
            AttributePath {
                attributes: [
                    Raw {
                        content: "greeting",
                        position: (2, 3),
                    },
                ],
            },
            String {
                parts: [
                    Raw {
                        content: "Hello World!",
                        position: (2, 15),
                    },
                ],
            },
        ),
    ],
    target: Variable {
        identifier: "greeting",
        position: (4, 3),
    },
    position: (1, 1),
}

Or perform Lexical Analysis:

LET "let" (1, 1)
ID "greeting" (2, 3)
= "=" (2, 12)
" "\"" (2, 14)
STR "Hello World!" (2, 15)
" "\"" (2, 27)
; ";" (2, 28)
IN "in" (3, 1)
ID "greeting" (4, 3)

And produce a Parse Tree:

Γ := rules "expr"
  expr := rules "expr_function"
    expr_function := rules "LET" "binds" "IN" "expr_function"
      LET := lexemes "LET"
        LET "let" (1, 1)
      binds := rules "binds" "attrpath" "=" "expr" ";"
        binds := rules
        attrpath := rules "attr"
          attr := rules "ID"
            ID := lexemes "ID"
              ID "greeting" (2, 3)
        = := lexemes "="
          = "=" (2, 12)
        expr := rules "expr_function"
          expr_function := rules "expr_if"
            expr_if := rules "expr_op"
              expr_op := rules "expr_app"
                expr_app := rules "expr_select"
                  expr_select := rules "expr_simple"
                    expr_simple := rules "\"" "string_parts" "\""
                      " := lexemes "\""
                        " "\"" (2, 14)
                      string_parts := rules "STR"
                        STR := lexemes "STR"
                          STR "Hello World!" (2, 15)
                      " := lexemes "\""
                        " "\"" (2, 27)
        ; := lexemes ";"
          ; ";" (2, 28)
      IN := lexemes "IN"
        IN "in" (3, 1)
      expr_function := rules "expr_if"
        expr_if := rules "expr_op"
          expr_op := rules "expr_app"
            expr_app := rules "expr_select"
              expr_select := rules "expr_simple"
                expr_simple := rules "ID"
                  ID := lexemes "ID"
                    ID "greeting" (4, 3)

This library is based on Santiago and can be used to implement tools like Code Formatters, documentation generators, and even alternative evaluators of the Nix expression language.

You can checkout more examples in the tests folder.

We hope you find NixEL useful!

And don’t forget to give us a star ⭐

Comments

Parsing invalid input

One attractive feature of rnix-parser is, through its use of rowan, it is able to recover from errors and provide partial parses for input. This is especially useful in the context of editor tooling where the code is frequently invalid while it is being edited. How does nixel/santiago stack up in this respect?

opened by darichey 6
Comparison with rnix-parser

There’s already a Nix parser written in Rust at https://github.com/nix-community/rnix-parser (but from what I could see based on a very different parsing method and producing a somewhat different result), how do the two of them compare? Do you have any benchmark or qualitative comparison?

opened by thufschmitt 3
Allow parsing partially invalid Nix code (Error recovery)
We currently use GNU Bison, and it would be possible to offer this feature by using Bison's native error recovery mechanisms: https://www.gnu.org/software/bison/manual/bison.html#Error-Recovery

There are a few sensible places to recover from errors, for instance:

Between parentheses (lists, attribute sets, etc), like '(' error ')', or

In unfinished bindings like binds attrpath '=' error

etc

More information in: https://github.com/kamadorueda/nixel/issues/2

enhancement help wanted good first issue
opened by kamadorueda 0