Vue's template compiler reimplemented in Rust!

Related tags

rust vuejs vue compiler
Overview

vue template compiler in Rust

https://github.com/vuejs/rfcs/discussions/369#discussioncomment-1192421

Maybe in the long run we want the whole transform (and even the Vue compiler!) to be implemented in native Go/Rust so performance would no longer be a concern ;)

Future is now!

Architecture

The original design in vue-next mixes code generation and ast parsing in the same data structure. As we can see, the transform pass will in-place mutate ast nodes, leaving the node with both code generation node and ssr code generation node.

This is typically a sign of leaky abstraction.

So in the Rust version I decided to take another approach.

The design targets at three different levels of developers in Vue ecosystem:

  • Lib/App creator: every Vue developers who write component library or application code.
  • Platform developer: Devs who write compiler implementation for DOM/SSR/Native platform.
  • Framework author: Vue core lib author a.k.a Evan.

The core framework targets multiple platforms and can be extended to support more. Core framework components span all platforms and are hardwired to the core lib runtime.

Platforms are usually DOM or SSR environment. Hosts are browser and node, respectively. Developing a platform needs to write code for both vue-compiler and vue-runtime. Optionally platform developer can write code in host, e.g. in hybrid app or mini-program.

And finally lib or app creators can write component library, business code or application components targeted to certain platforms.

The compilation has several phases:

  • Scan (output: Tokens): Hardwired in the compiler at framework level.
  • Parse (output: template AST): Hardwired in the compiler at framework level.
  • Convert (output: intermediate representation): Customizable for platform developers with sensible default.
  • Transform (input/output: customizable IR): Customizable with default by using generic/traits.
  • Code Generate (customizable output: e.g. JS/TS): Customizable with default.

Other Design different from the original compiler

  • Directive parsing is implemented manually instead of by regexp.
  • nodeTransforms is not supported. It's too hard for app creator to use and maintain IR invariants. Platform devs can still customize by implementing converter/transformer.
  • directiveTransforms now can returns not only Props but also SimpleExpression. The extra flexibility makes a more cohesive v-bind/v-on conversion: the logic for processing the directives now resides in one single file without regard to the presence of an argument.
  • Runtime helper collection context.helper/helperString is moved out from convert and tracked in transform phase, avoiding several methods and reducing HashMap to a bitflag.

Intended Usage

  • Rust library
  • CLI binary
  • napi based nodejs library
  • wasm based npm package: a fallback if napi fails to work and a toy for browser.
  • No Browser build No support since most features in full build are additional except for browser based expression checking or HTML escaping. Browser build removed them for size. But template compiler in browser is already for toy project. For browser specific summary see this google sheet.

Implementation Detail

  • Plenty of debug_asserts to maintain compiler state invariants.
  • The library seeks minimal allocation by using &str, Cow<'_, str> and smallvec.
  • A customized VStr is used to minimize string manipulation.
  • Fxhash is preferred over default hasher since hash collision is not a concern.
  • The bitflags crate is used to represent runtime helper and vnode patch flags.
  • Use heavily optimized routines for string search primitives. (Perf reference)
  • Benchmark with criterion.rs.
  • Test compiler output by snapshot test.
  • Use alternative allocator like wee_alloc.

Reference

  • vue-next: ご本家様
  • html spec is the definitive guide for parsing HTML-like files.
  • Vue Template Explorer gives instant results for code generation and error reporting.
  • Nu html checker is the official html validator from W3C. This is the canonical error reporter for html parsing, when there is a discrepancy between the framework and the spec.
  • AST explorer can inspect AST nodes interactively.

Roadmap

Todo tasks grouped by scopes.

[util]

  • VStr
    • string intern
    • camel/pascal cache
    • str ops

[core]

  • tokenizer
    • UTF8 support
  • parser
  • IR converter
    • v-if
    • v-for
    • v-slot
    • slot outlet
    • element
    • build props
  • transformer
  • code generator

[dom]

  • transformer
  • code generator

[ssr]

  • TODO

[sfc]

  • TODO

[test]

  • tokenizer test
  • parser test
    • dir parser test
  • Add insta snapshot

[bench]

  • Add benchmark framework
  • Micro benchmarks for compiler components
  • Integrated benchmarks using repos like Element-Plus

[infra]

  • Add pre-commit hooks.
  • Add Github Actions for various checks.
  • Change single lib to cargo workspaces.

[community]

  • TODO. not ready for contribution for now.
Issues
  • Setup github actions

    Setup github actions

    null

    opened by zhmushan 4
  • add new implementation detail

    add new implementation detail

    null

    opened by IWANABETHATGUY 2
  • Add pre-commit for better engineering

    Add pre-commit for better engineering

    Adding a pre-commit hook for

    • fmt
    • clippy
    • commitlint
    enhancement 
    opened by HerringtonDarkholme 1
  • tracking snapshot testing library

    tracking snapshot testing library

    https://github.com/mitsuhiko/insta

    enhancement 
    opened by IWANABETHATGUY 0
  • 🎸 refactor

    🎸 refactor

    refactor

    opened by IWANABETHATGUY 0
  • refactor: 💡 maybe perf a bit

    refactor: 💡 maybe perf a bit

    null

    opened by IWANABETHATGUY 0
  • refactor: 💡 refactor

    refactor: 💡 refactor

    refactor parser

    opened by IWANABETHATGUY 0
  • Should RE used in the crate?

    Should RE used in the crate?

    Hard to migrate but possible

    • [x] compiler-core/src/transforms/vFor.ts|304 col 15| const forAliasRE = /([\s\S]*?)\s+(?:in|of)\s+([\s\S]*)/
    • [x] compiler-core/src/transforms/vFor.ts|307 col 18| const forIteratorRE = /,([^,\}\]]*)(?:,([^,\}\]]*))?$/
    • [ ] compiler-core/src/transforms/vOn.ts|19 col 12| const fnExpRE = /^\s*([\w$_]+|\([^)]*?\))\s*=>|^\s*function(?:\s+[\w$]+)?\s*\(/
    • [ ] compiler-core/src/utils.ts|79 col 17| const whitespaceRE = /\s+[.[]\s*|\s*[.[]\s+/g
    • [ ] shared/src/normalizeProp.ts|30 col 20| const listDelimiterRE = /;(?![^(]*\))/g // ; followed by no single )

    Easy to migrate

    compiler-core/src/parse.ts|60 col 13| const decodeRE = /&(gt|lt|amp|apos|quot);/g
    compiler-core/src/transforms/vFor.ts|308 col 18| const stripParensRE = /^\(|\)$/g
    compiler-core/src/utils.ts|66 col 20| const nonIdentifierRE = /^\d|[^\$\w]/
    compiler-core/src/utils.ts|77 col 26| const validFirstIdentCharRE = /[A-Za-z_$\xA0-\uFFFF]/
    compiler-core/src/utils.ts|78 col 21| const validIdentCharRE = /[\.\?\w$\xA0-\uFFFF]/
    shared/src/normalizeProp.ts|31 col 24| const propertyDelimiterRE = /:(.+)/
    shared/src/escapeHtml.ts|1 col 13| const escapeRE = /["'&<>]/
    compiler-dom/src/transforms/stringifyStatic.ts|142 col 15| const dataAriaRE = /^(data|aria)-/
    

    use RE in SFC crates

    compiler-sfc/src/parse.ts|333 col 12| const splitRE = /\r?\n/g
    compiler-sfc/src/parse.ts|334 col 12| const emptyRE = /^(?:\/\/)?\s*$/
    compiler-sfc/src/parse.ts|335 col 14| const replaceRE = /./g
    compiler-sfc/src/rewriteDefault.ts|4 col 20| const defaultExportRE = /((?:^|\n|;)\s*)export(\s*)default/
    compiler-sfc/src/rewriteDefault.ts|5 col 25| const namedDefaultExportRE = /((?:^|\n|;)\s*)export(.+)as(\s*)default/s
    compiler-sfc/src/rewriteDefault.ts|6 col 25| const exportDefaultClassRE =
    compiler-sfc/src/stylePluginScoped.ts|5 col 20| const animationNameRE = /^(-\w+-)?animation-name$/
    compiler-sfc/src/stylePluginScoped.ts|6 col 16| const animationRE = /^(-\w+-)?animation$/
    compiler-sfc/src/cssVars.ts|15 col 20| export const cssVarRE = /\bv-bind\(\s*(?:'([^']+)'|"([^"]+)"|([^'"][^)]*))\s*\)/g
    ref-transform/src/refTransform.ts|28 col 21| const transformCheckRE = /[^\w]\$(?:\$|ref|computed|shallowRef)?\s*(\(|\<)/
    
    opened by HerringtonDarkholme 0
Owner
Herrington Darkholme
闇と森の妖精
Herrington Darkholme
The sysroot manager that lets you build and customize `std`

PSA: Xargo is in maintenance mode xargo The sysroot manager that lets you build and customize std Cross compiling `std` for i686-unknown-linux-gnu Xar

Jorge Aparicio 882 Sep 19, 2021
[OUTDATED] Instructions for how to cross compile Rust projects for the Raspberry Pi

Cross Compiling for Raspberry Pi This guide will show how Rust programs can be cross compiled for the Raspberry Pi using Cargo. These instructions may

Erik Hedvall 293 Aug 30, 2021
Operating System development tutorials in Rust on the Raspberry Pi

Operating System development tutorials in Rust on the Raspberry Pi

Rust Embedded 5.2k Sep 19, 2021
Reusable components for the Arduino Uno.

Ruduino This library provides a set of reusable components for the Arduino Uno. Overview Register and bit definitions use ruduino::cores::current::POR

The AVR-Rust project 503 Sep 15, 2021
Driver to work with the esp8266 module over the serial port.

esp8266-wifi-serial (WIP) Driver to work with the esp8266 module over the serial port. By using this module you can join the existing access point or

Aleksey Sidorov 15 Aug 12, 2021