What if we could check declarative macros before using them?

Overview

expandable

An opinionated attribute-macro based macro_rules! expansion checker.


Sylvester Stallone in Rambo: First Blood Part II. The image has been edited such that his two hands are thumbsup-ing. His body is covered with dust and sweat and a bit of blood (not his). At the bottom of the image is written 'POV: #[expandable::expr] stops complaining'.

Textbook example

rustc treats macro definitions as some opaque piece of tokens and don't do any check on them. For instance, the following macro definition is valid:

macro_rules! js_concat {
    ($left:expr, $right:expr) => {
        $left ++ $right
    };
}

However, any call to the js_concat macro is invalid, as the ++ operator does not exist in Rust. Luckily for us, this crate provides the expandable::expr macro, that checks that the macro expands to a valid expression. Let's use it on js_concat:

#[expandable::expr]
macro_rules! js_concat {
    ($left:expr, $right:expr) => {
        $left ++ $right
    };
}

This emits the following error 1:

error: Potentially invalid expansion. Expected an identifier, a literal, `if`, a `[`, a `{`.
 --> tests/ui/fail/js_concat.rs:5:16
  |
5 |         $left ++ $right
  |                ^

Expansion context

Macros can expand to different things depending on where they are called. As a result, expandable must know what the macro expands to. To do so, multiple macros are available:

  • Macros that expand to expressions are checked by expandable::expr,
  • Macros that expand to items are checked by expandable::item,
  • TODO: pattern, statements, type.

What can it detect?

This section briefly describes what the macros can detect and what they can't. All the terminology used in this section is taken from the Ferrocene Language Specification.

This crate aims to detect all the things that are not considered by rustc as an invalid macro definition. It may or may not detect what rustc already detects. Some errors can't be detected because they are not possible to detect in the first place.

Invalid matcher

rustc is quite good at detecting invalid matchers. This crate deliberately does not try to compete with it. Whether if this crate returns an error on an invalid matcher is left unspecified.

Invalid transcription

Some macros can't be expanded because they don't respect the rules for a transcription to happen. For instance, the following macro exhibits a transcription error:

macro_rules! invalid_expansion {
  ( $( $a:ident )? ) => { $a }
}

In this example, the repetition nesting of $a used in the macro transcription does not match the repetition nesting of $a defined in the matcher.

This crate aims to detect all the possible invalid transcription.

Invalid produced AST

Some macro expand correctly (ie: they respect the transcription rules), but the produced AST is invalid. For instance, the following macro expands to an invalid AST:

macro_rules! invalid_expansion {
    () => { * }
}

(* is an invalid item, an invalid expression, an invalid pattern, an invalid statement and an invalid type -- there is no context in which it can be called).

rustc doesn't (and can't) detect any potentially invalid AST. This is the raison d'être of this crate.

Opinionated?

In the general case, proving that a macro is well-formed is impossible. This crates has to make assumptions about the macros it is checking. This section lists them and gives a short rationale of why they are needed.

No recursive macro definition

This crate assumes that the macro expansion does not contain any macro definition. For instance, the following macro definition can't be checked with expandable:

macro_rules! foo {
    () => {
        macro_rules! bar {
            () => { 42 }
        }
    }
}

This limitation is caused by the fact that it's impossible to tell if a sequence of tokens is a macro definition or not. For instance, the macro_rules token may be passed by the user when invoking the macro. Having no guarantee of what's a macro and what is not makes it impossible to ensure that a metavariable is properly defined.

No macro call (for now)

This crate assumes that one expansion of a macro works. It does not check that the recursive expansion of the macro works. It checks that the macro invocation itself is legal, but don't check that it matches any rule.

This is caused by the fact that other macros may add more constrains on the macro invocation. This requirement may be lifted in the future for macros that call themselves.

The repetition stack must match

This crate requires a metavariable indication used in a transcriber to have the same repetition stack as the corresponding metavariable that appears in the matcher.

For instance, the following code is rejected by expandable:

#[expandable::expr]
macro_rules! fns {
    ($($a:ident)*) => {
        $(
            fn $a() {}
        )+
    }
}

Minimal Supported Rust Version (MSRV), syntax support and stability

expandable supports Rust 1.65 and above. Bumping the MSRV is considered a breaking change.

Note that the embedded parser will support syntax that was introduced after Rust 1.65.

Adding support for newer syntax is not considered a breaking change.

The error messages generated by expandable are not stable and may change without notice.

Any change that may trigger errors on previously accepted code is considered a breaking change.

License

Licensed under the MIT license.

Footnotes

  1. The Rust grammar is not fully implemented at the moment, leading to incomplete "expected xxx" list this will be fixed before the first non-alpha release of this crate.

Comments
  • Update README and crate docs to show 1.65 as MSRV, not 1.56

    Update README and crate docs to show 1.65 as MSRV, not 1.56

    Based on the GitHub Actions config and the justfile, the tested MSRV appears to be 1.65, not 1.56 like the README says on the current main branch. I'm guessing the code is correct and the README just swapped the numbers by accident.

    opened by obi1kenobi 4
  • Improve the crate-level doc and readme

    Improve the crate-level doc and readme

    This commit adds even more documentation explaining what is supported and what is not.

    The documentation is added at the top of the expandable crate. We may want to use some shenanigans in order to move part of it to the expandable-impl crate.

    opened by scrabsha 1
  • feat: parse block expressions and const generics

    feat: parse block expressions and const generics

    This PR adds support for BlockExpression* **, and its parsing in generic argument context. In addition, literals (optionally prefixed by -) are parsed in generic arguments.

    *: does not include labels **: we still don't handle statements, it is just { <expr> } actually.

    opened by scrabsha 0
  • feat: parse generics in function and method calls

    feat: parse generics in function and method calls

    #8 and #22 added support for function and method call with 0 or more arguments. This MR adds support for generic arguments using the turbo::<fish> syntax.

    For now, we only support generic type arguments. This will be improved later.

    Additionally, some occurrence of First- and Then-suffixed stack symbols were merged in order to reduce the amount of required transitions and stack symbols.

    opened by scrabsha 0
  • feat: parse <expr> . <something>

    feat: parse .

    This MR adds support for AwaitExpr, FieldAccessExpression* and MethodCallExpression** parsing.

    *: We currently don't recognize literal fragments, so it does not handle <expr> . $literal **: This does not include method call with generics - they are too scary for now >.<

    opened by scrabsha 0
  • refactor: add a transition inheritance system

    refactor: add a transition inheritance system

    Many state transitions set are a superset of another state's transition set. The current solution for this is to write these transitions one time for each state. This is very suboptimal and error-prone (see #17).

    This PR fixes the situation by allowing a state to "extend" another set, and inherit its states. This allows us to remove almost all the transitions in the AfterIf state, and will make adding new supersets of ExprStart easier.

    opened by scrabsha 0
  • feat: add support for array expressions

    feat: add support for array expressions

    opened by scrabsha 0
  • fix: Add the missing AfterIf transitions

    fix: Add the missing AfterIf transitions

    (This is just another instance of me not being able to update multiple things at the same time)

    #14 added a bunch of transitions that occurred in the AfterExpr state. I forgot to add them in the AfterIf state as well. This PR fixes that.

    Additionally I added a test so that the CI will break if I ever forget to update the AfterIf state. Great!

    opened by scrabsha 0
  • feat: parse all the tokens of the Rust language

    feat: parse all the tokens of the Rust language

    I was tired adding the punctuation tokens one by one to the language, so I wrote this mega-PR which adds all of them. Every punctuation, everywhere, all at once.

    I followed the Ferrocene naming for the punctuators. I hope it will help future contributors.

    Also: previous implementation of quote translated @ as a $. This has been changed such that # now translates to $ (@ translates to @, cool cool). This makes us more consistent with how the actual quote works.

    opened by scrabsha 0
  • refactor: compute the token descriptions before checking the expansion

    refactor: compute the token descriptions before checking the expansion

    This allows the transition function to be "more branchless", which will severely reduce the pressure on the "Token -> TokenDescription" conversion function when checking a repetition.

    opened by scrabsha 0
  • feat: add arithmetic, bit and comparison expression parsing

    feat: add arithmetic, bit and comparison expression parsing

    Two things to note:

    • We should establish a naming convention for the Terminals and the TokenDescriptions. This is a huge mess.
    • On the frontend side, we will need to limit the number of TokenDescription printed in the error messages.

    Relevant links:

    opened by scrabsha 0
Owner
Sasha Pourcelot
POSIX sorceress by day, Rustacean by night.
Sasha Pourcelot
Which words can you spell using only element abbreviations from the periodic table?

Periodic Words Have you ever wondered which words you can spell using only element abbreviations from the periodic table? Well thanks to this extremel

J Spencer 11 Apr 26, 2021
A crate using DeepSpeech bindings to convert mic audio from speech to text

DS-TRANSCRIBER Need an Offline Speech To Text converter? Records your mic, and returns a String containing what was said. Features Begins transcriptio

null 32 Oct 8, 2022
Probabilistically split concatenated words using NLP based on English Wikipedia unigram frequencies.

Untanglr Untanglr takes in a some mangled words and makes sense out of them so you dont have to. It goes through the input and splits it probabilistic

Andrei Butnaru 15 Nov 23, 2022
A Markdown to HTML compiler and Syntax Highlighter, built using Rust's pulldown-cmark and tree-sitter-highlight crates.

A blazingly fast( possibly the fastest) markdown to html parser and syntax highlighter built using Rust's pulldown-cmark and tree-sitter-highlight crate natively for Node's Foreign Function Interface.

Ben Wishovich 48 Nov 11, 2022
A lightweight platform-accelerated library for biological motif scanning using position weight matrices.

?? ?? lightmotif A lightweight platform-accelerated library for biological motif scanning using position weight matrices. ??️ Overview Motif scanning

Martin Larralde 16 May 4, 2023
Implementation of sentence embeddings with BERT in Rust, using the Burn library.

Sentence Transformers in Burn This library provides an implementation of the Sentence Transformers framework for computing text representations as vec

Tyler Vergho 4 Sep 4, 2023
Automatically check for SPF misconfigurations that could result in email spoofing

SPFJack Email spoofing is dead, but misconfiguration never dies. Purpose This project is designed to take in domain names and review their SPF records

Alex (LunarCA) 2 Mar 27, 2022
FeignHttp is a declarative HTTP client. Based on rust macros.

FeignHttp is a declarative HTTP client. Based on rust macros. Features Easy to use Asynchronous request Configurable timeout settings Suppor

null 46 Nov 30, 2022
Pretend is a macros-based declarative Rust HTTP client

pretend is a modular, Feign-inspired HTTP, client based on macros. It's goal is to decouple the definition of a REST API from it's implementation.

null 24 Aug 3, 2022
The utility is designed to check the availability of peers and automatically update them in the Yggdrasil configuration file, as well as using the admin API - addPeer method.

Yggrasil network peers checker / updater The utility is designed to check the availability of peers and automatically update them in the Yggdrasil con

null 6 Dec 25, 2022
Js-macros - Quickly prototype Rust procedural macros using JavaScript or TypeScript!

js-macros Quickly prototype Rust procedural macros using JavaScript or TypeScript! Have you ever thought "this would be a great use case for a procedu

null 15 Jun 17, 2022
Program to check if stereo wav files have identical channels (faux-stereo) and convert them to mono.

zrtstr Command line application for checking WAV-files for identical channels, detecting faux-stereo files generated by some audio-editing software an

Kirill 22 Nov 6, 2022
vault client using jwt authentication that define environment variables from vault secrets before executing into something else

envlt envlt, like env, allows you to define environment variables and then execute into something else, but instead of static values, it uses using si

Eric Burghard 6 Nov 13, 2022
svgcleaner could help you to clean up your SVG files from the unnecessary data.

svgcleaner svgcleaner helps you clean up your SVG files, keeping them free from unnecessary data. Table of Contents Purpose Goals Alternatives Charts

Evgeniy Reizner 1.5k Jan 9, 2023
This is a smart contract running on NEAR Protocol. It could be used to run a token sale.

Token Sale This is a smart contract running on NEAR Protocol. It could be used to run a token sale. Sale rules There are 2 periods: Sale and Grace. In

Harry Nguyen 0 Dec 5, 2021
Finds imports that could be exploited, still requires manual analysis.

drv-vuln-scanner Vulnerable driver scanning tool for win64, put drivers to scan in drv/. Finds imports that could be exploited, still requires manual

selene 24 Dec 10, 2022
Little 2D physics engine used for my game Crate Before Attack.

Circle2D Circle2D is a little physics library used for my game CrateBeforeAttack. Live demo: https://koalefant.github.io/circle2d/ It is not productio

Evgeny Andreeshchev 24 Jan 14, 2022
Discover GitHub token scope permission and return you an easy interface for checking token permission before querying GitHub.

github-scopes-rs Discover GitHub token scope permission and return you an easy interface for checking token permission before querying GitHub. In many

null 8 Sep 15, 2022
A canvas on which you can draw anything with ease before drawing the pixels on your small hardware display.

embedded-canvas    canvas - a piece of cloth backed or framed as a surface for a painting NOTE: This crate is still in development and may have breaki

Lechev.space 13 Aug 31, 2022
I will be attempting Advent of Code 2022 with Rust, a language I have never learned before.

Advent of Code 2022 This year, I will be attempting Advent of Code with Rust, a language I have never learned before. I will also be taking some notes

null 4 Jan 7, 2023