ArchLinux users can install Rulex from AUR:

• https://aur.archlinux.org/packages/rulex-rs-bin

E.g.: yay -S rulex-rs-bin

It will fetch the latest binary release from Microsoft Github and install it along with the CHANGELOG, the README and the licenses.

Description

This PR removes the dependency on thiserror, which can be replaced by a manual implementation of std::error::Error and core::fmt::Display. This makes pomsky and pomsky-macro external dependency free! :tada:

I made these changes to avoid adding proc-macro dependencies to a small project where I used pomsky-macro. I used Copilot to copy most of error messages, they should be checked again to ensure I didn't changed anything.

~~Fixes # (issue)~~ No issue was created.

Checklist:

• [x] My code is formatted with cargo fmt
• [x] My code compiles with the latest stable Rust toolchain
• [x] All tests pass with cargo test
• [x] My changes generate no new warnings with cargo clippy
• [ ] ~~I have commented my code, particularly in hard-to-understand areas~~
• [ ] ~~My changes are covered by tests~~

As explained in the blog post, we need to rename the project to pomsky.

What needs to be done

• [x] announce the new name
• [x] rename the repository
• [x] rename the types in the source code containing Rulex
• [x] adjust the README
• [x] adjust the documentation on the website
• [x] adjust the source code documentation
• [x] contact the maintainer of the AUR package
  • [x] rename the AUR package or publish a new one and delete the existing one
  • [ ] announce the AUR package again
• [x] publish all the crates under their new names
• [x] publish minor versions of the old crates that warn the user in the README and the documentation that they are unmaintained and should be replaced
• [ ] rename the GitHub organization
  • this won't be done until we're sure that the old URLs will continue to work, so we can redirect them to the new website

Status Quo

The meaning of <% and %> is not intuitively clear, and even confusing to some: Someone suggested that <% should indicate the end of the string, because the < angle should point towards the string, and vice versa for %>. Even worse, in right-to-left (RTL) languages the directions are reversed. Therefore it doesn't make sense to speak about the "left" and "right" end of a string, only the start and end of the string.

In the latest version, two built-in variables, Start and End, were added that are currently aliases for <% and %>.

Solution

Deprecate <% and %>. Suggest using Start and End instead.

Deprecation schedule

• [x] Add the built-ins Start and End
• [x] Update the documentation to recommend these built-ins
• [x] Emit a deprecation warning in the CLI and the playground when the old syntax is used
• [x] Remove documentation for the old syntax
• [ ] Remove the old syntax in the next breaking release [target: 0.7]

I opened issue #17, but decided to just write a PR myself.
This adds a new flag no-new-line to rulex-bin, that suppresses a new-line that is currently printed by using println!("{compiled}");

Is your feature request related to a problem? Please describe.

When I typed [d], I thought rulex would output \d, but it output \p{Nd}.

Describe the solution you'd like

1. output \d
2. Prompt users to use Unicode /\p{Nd}/u

Describe alternatives you've considered

Additional context

Description

I discovered three panic scenarios while fuzzing the rulex parser, all related to indexing (by byte offset) into a str and landing within a multi-byte utf8 codepoint. The following queries trigger panics (where MB is a single multibyte codepoint):

• "MB": MB within a quoted section
• \MB: an escaped MB
• "\MB: an escaped MB within a quoted section

Checklist:

• [x] My code is formatted with cargo fmt
• [x] My code compiles with the latest stable Rust toolchain
• [x] All tests pass with cargo test
• [x] My changes generate no new warnings with cargo clippy
• [x] I have commented my code, particularly in hard-to-understand areas
• [x] My changes are covered by tests

Describe the bug

When trying to write a regex lookahead, you get a nice help message:

(?=Test)

This syntax is not supported

Help: Lookahead uses the `>>` syntax. For example, `>> 'bob'` matches if the position is followed by bob.

However, no help message is shown for lookbehind (positive or negative).

To Reproduce

(?<=Test)

or

(?<!Test)

Expected behavior

A help message appears in the lines of

Help: Lookbehind uses the `<<` syntax. For example, `<< 'bob'` matches if the position is preceded by bob.

Additional context

Tried it in the playground

Description

Fixes #24

Checklist:

• [x] My code is formatted with cargo fmt
• [x] My code compiles with the latest stable Rust toolchain
• [x] All tests pass with cargo test
• [x] My changes generate no new warnings with cargo clippy
• [x] I have commented my code, particularly in hard-to-understand areas
• [x] My changes are covered by tests

Support regex back and forward references.

A reference is a number or identifier prefixed with two dots:

:('hello world') '!'* ::1
:name('hello world') '!'* ::name

This matches the string hello world!!!hello world.

In repetitions, some regex flavors allow forward references, i.e. references to a group that appears later in the regex. They match the text that was matched by that group in the previous repetition:

( ::1* '.' :('a') 'b' )+

This matches the string .ab, but also .abaaa.ab.

JavaScript supports backreferences, but only numbered. It doesn't support forward references.

Is your feature request related to a problem? Please describe.

Pomsky doesn't implement all regex features, and probably never will, given the vast number of regex engines.

Describe the solution you'd like

A special syntax to embed a regex in a pomsky expression, which is emitted unchanged. This looks like this:

regex "Hello [world]"
regex '\bHello\B'

Describe alternatives you've considered

Instead of a keyword, backticks could be used like in melody. I'm not a fan of this, because this feature should be available, but not encouraged. There should be "syntactic salt" to make it less convenient.

Another, perhaps more fitting name for the keyword would be "raw", since the string is not escaped. However, the meaning of "raw" strings in PL is usually inverted: It refers to escaping in the input, not the compiled output. The keyword regex better describes what use cases this syntax enables.

Roadmap

• [x] Reserve keyword for 0.7
• [x] Implement syntax for 0.8

We need a code of conduct that requires participants in discussions to adhere to certain standards. The goal is to foster a friendly and welcoming community where anyone can feel safe to participate.

Most importantly, the following behaviors are prohibited:

• harassment and violent language
• racism, sexism, and discrimination based on gender, sexual orientation, disability, religion, etc.

This is not in response to any specific incident. My discussions so far have been pleasant, and I want them to stay that way.

I'm inclined to copy the CoC from Rust:

• We are committed to providing a friendly, safe and welcoming environment for all, regardless of level of experience, gender identity and expression, sexual orientation, disability, personal appearance, body size, race, ethnicity, age, religion, nationality, or other similar characteristic.
• Please avoid using overtly sexual aliases or other nicknames that might detract from a friendly, safe and welcoming environment for all.
• Please be kind and courteous. There’s no need to be mean or rude.
• Respect that people have differences of opinion and that every design or implementation choice carries a trade-off and numerous costs. There is seldom a right answer.
• Please keep unstructured critique to a minimum. If you have solid ideas you want to experiment with, make a fork and see how it works.
• We will exclude you from interaction if you insult, demean or harass anyone. That is not welcome behavior. We interpret the term “harassment” as including the definition in the Citizen Code of Conduct; if you have any lack of clarity about what might be included in that concept, please read their definition. In particular, we don’t tolerate behavior that excludes people in socially marginalized groups.
• Private harassment is also unacceptable. No matter who you are, if you feel you have been or are being harassed or made uncomfortable by a community member, please contact one of the channel ops or any of the Rust moderation team immediately. Whether you’re a regular contributor or a newcomer, we care about making this community a safe place for you and we’ve got your back.
• Likewise any spamming, trolling, flaming, baiting or other attention-stealing behavior is not welcome.

A page with this code of conduct (adapted to Pomsky) will be added to the website footer and linked from the README. As (currently) the only maintainer I'm responsible for moderating discussions. This means that people who violate the code of conduct will face consequences, ranging from a stern warning to a permanent ban, depending on the severity.

Is your feature request related to a problem? Please describe.

Tools such as IDE plugins need Pomsky's output in a machine-readable format. There are multiple ways to achieve this:

1. Publish a native library that other tools can dynamically link to
2. Let tool authors create the bindings they need themselves by using the pomsky crate and distribute it as part of their tool
3. Make the CLI more powerful, so it can be used by tools more effectively

The solution I'm proposing is number 3 since I think it offers the best experience both for users and tool authors: Users need to have the CLI installed, but they don't have to download anything else, and tool authors don't need to update their tools every time a new Pomsky version is released.

Describe the solution you'd like

Add a --json flag to print all compilation results as JSON rather than plain text. The JSON is written to stdout on a single line. It's an object with the following structure:

{
  version: "1"  // schema version
  success: bool  // true if no errors occurred
  output? : string  // compiled regex, only present if the compilation was successful
  diagnostics: object[]  // array of errors and warnings
    severity: "error" | "warning"
    kind: string  // e.g. "parse" for parse errors or "compat" for compatibility warnings
    spans: object[]  // source code locations that should be underlined
      // initially this array will always contain exactly one object
      start: int  // start byte offset, inclusive
      end: int  // end byte offset, exclusive
      label? : string  // optional additional information to this span
    description: string  // explanation of the error or warning
    help: string[]  // additional information to help the user fix the issue
    fixes: object[]  // "quick fixes", automatic source code transformations; may be displayed as a light bulb
      // since this feature isn't implemented in Pomsky yet, this array will always be empty initially
      description: string  // text to display 
      replacements: object[]  // array of source code locations that need to be modified
        start: int  // start byte offset, inclusive
        end: int  // end byte offset, exclusive
        insert: string  // text to replace the source code location with
}

Example:

{
  "version": "1",
  "success": false,
  "diagnostics": [
    {
      "severity": "error",
      "kind": "lexer",
      "spans": [
        { "start": 3, "end": 4 }
      ],
      "description": "unexpected `$`",
      "help": "literal text must be wrapped in quotes, i.e. '$' or \"$\"",
      "fixes": []
    }
  ]
}

Byte offsets are zero-indexed, e.g. the first byte has the span { "start": 0, "end": 1 }.

The input is always treated as UTF-8, so the characters in the string a💩ø have the spans { "start": 0, "end": 1 }, { "start": 1, "end": 5 } and { "start": 5, "end": 7 }.

Describe alternatives you've considered

There are other formats than JSON, like XML and MessagePack, but they're less widely supported, and JSON should be fast enough.

Instead of reporting byte offsets, Pomsky could return row and column positions. However, that requires more work, and may not even work for every tool, since different IDEs have different plugin interfaces. Furthermore, UTF-8 has an exact definition, whereas rows and columns aren't well defined in the context of Unicode.

Future possibilities

We can keep the CLI running in the background, so tools can write to stdin and get a response at stdout, without having to spawn a process each time.

We can add a --check flag that returns diagnostics but doesn't actually compile the expression, which should be faster. This could then be used to report errors on every keystroke.

Distributing Pomsky not only in the CLI format, but also as a native library (.so, .dll), allows tools to compile and receive an output in a standardized format which can be interpreted through interfaces like JNI or JNA. Example usage: compilation and error reporting inside of an IDE.

I think a project like this https://github.com/Dushistov/flapigen-rs might come handy.

Is your feature request related to a problem? Please describe.

Duplicate alternatives can happen when combining different variables, or simply by accident.

Example: A subset of C keyword with while appearing twice:

% (
  | "break"
  | "case"
  | "char"
  | "const"
  | "continue"
  | "default"
  | "do"
  | "double"
  | "else"
  | "float"
  | "for"
  | "if"
  | "int"
  | "long"
  | "return"
  | "short"
  | "sizeof"
  | "struct"
  | "switch"
  | "void"
  | "while"
  | "while"
) %

Describe the solution you'd like

Assuming no side effects (e.g. capturing groups), duplicate alternatives can be removed without affecting the behavior of the pattern (negatively).

More specifically, given an alternation, let index(x) -> int be a function that returns the index of a given alternative of the alternation in its list of alternatives. If there exist two alternatives a and b such that index(a) < index(b), then b can be removed without effecting the pattern.

This optimization might even prevent very simple cases of exponential backtracking. E.g. ( "a" | "a" )+ "b".

Describe alternatives you've considered

A more complete solution would be to implement a more complete DFA-based solution to determine whether a given alternative is a subset of the union of all previous alternatives in the alternation. However, this is significantly more computationally expensive and does not support irregular features such as assertions.

Additional context

The regexp/no-dupe-disjunctions rule is an implementation of this optimization. This rule uses the DFA approach along if a simpler regex-comparison approach to determine duplicate alternatives.

Is your feature request related to a problem? Please describe.

Pomsky currently does not remove unnecessary elements in character classes. E.g. [ w "abc" ] compiles to [\wabc] (Java). However, the abc is unnecessary because [\wabc] == \w.

Describe the solution you'd like

Remove unnecessary elements in character classes to optimize and simplify them.

Additional context

This requires knowing the precise set of characters accepted by each character class element. For an example implementation of this, checkout the regexp/no-dupe-characters-character-class rule.