syntect

syntect is a syntax highlighting library for Rust that uses Sublime Text syntax definitions. It aims to be a good solution for any Rust project that needs syntax highlighting, including deep integration with text editors written in Rust. It's used in production by at least two companies, and by many open source projects.

If you are writing a text editor (or something else needing highlighting) in Rust and this library doesn't fit your needs, I consider that a bug and you should file an issue or email me. I consider this project mostly complete, I still maintain it and review PRs, but it's not under heavy development.

Important Links

API docs with examples: https://docs.rs/syntect
Changelogs and upgrade notes for past releases

Getting Started

syntect is available on crates.io. You can install it by adding this line to your Cargo.toml:

syntect = "4.4"

After that take a look at the documentation and the examples.

If you've cloned this repository, be sure to run

git submodule update --init

to fetch all the required dependencies for running the tests.

Features/Goals

Work with many languages (accomplished through using existing grammar formats)
Highlight super quickly, faster than nearly all text editors
Include easy to use API for basic cases
API allows use in fancy text editors with piece tables and incremental re-highlighting and the like.
Expose internals of the parsing process so text editors can do things like cache parse states and use semantic info for code intelligence
High quality highlighting, supporting things like heredocs and complex syntaxes (like Rust's).
Include a compressed dump of all the default syntax definitions in the library binary so users don't have to manage a folder of syntaxes.
Well documented, I've tried to add a useful documentation comment to everything that isn't utterly self explanatory.

Built-in output to coloured HTML

 tags or 24-bit colour ANSI terminal escape sequences.

Nearly complete compatibility with Sublime Text 3, including lots of edge cases. Passes nearly all of Sublime's syntax tests, see issue 59.
Load up quickly, currently in around 23ms but could potentially be even faster.

Screenshots

There's currently an example program called syncat that prints one of the source files using hard-coded themes and syntaxes using 24-bit terminal escape sequences supported by many newer terminals. These screenshots don't look as good as they could for two reasons: first the sRGB colours aren't corrected properly, and second the Rust syntax definition uses some fancy labels that these themes don't have highlighting for.

Example Code

Prints highlighted lines of a string to the terminal. See the easy and html module docs for more basic use case examples.

use syntect::easy::HighlightLines;
use syntect::parsing::SyntaxSet;
use syntect::highlighting::{ThemeSet, Style};
use syntect::util::{as_24_bit_terminal_escaped, LinesWithEndings};

// Load these once at the start of your program
let ps = SyntaxSet::load_defaults_newlines();
let ts = ThemeSet::load_defaults();

let syntax = ps.find_syntax_by_extension("rs").unwrap();
let mut h = HighlightLines::new(syntax, &ts.themes["base16-ocean.dark"]);
let s = "pub struct Wow { hi: u64 }\nfn blah() -> u64 {}";
for line in LinesWithEndings::from(s) {
    let ranges: Vec<(Style, &str)> = h.highlight(line, &ps);
    let escaped = as_24_bit_terminal_escaped(&ranges[..], true);
    println!("{}", escaped);
}

Performance

Currently syntect is one of the faster syntax highlighting engines, but not the fastest. The following perf features are done:

Pre-link references between languages (e.g Comments [WIP] Kinda-working fancy-regex support This branch switches the regex engine to fancy-regex or more specifically my fork of it. Currently it only works for a few syntaxes because of a few different features fancy-regex doesn't support: [x] The \n escape (Everything, but fixed it my fork) [x] Unnecessary escapes in character classes like [\<] [x] The \h escape in character classes (Rust) [x] Fix #76 so nonewlines mode doesn't produce weird regexes. [x] Named backrefs \k<marker> (Markdown) [x] Fancy character class syntax [a-w&&[^c-g]z] [ ] Add support for match limit to fancy-regex: https://github.com/google/fancy-regex/issues/44 The jQuery highlighting benchmark now takes 1s instead of 0.66s. Which is super unfortunate given that I'd hoped it would be faster than Oniguruma. I have no idea why it is substantially slower. @raphlinus @robinst opened by trishume 67 Highlighting Scintilla widget with Syntect Here's the thing, in the past I'd been playing around with lark-parser to highlight the syntax of a Scintilla widget with +/- succes. here you can see a very little example where you can see how lark & qscintilla are used together Thing is, syntax highlighting with Scintilla is really painful (as well as creating new lexers)... So I was considering this idea of being able to use textmate syntax&themes on a Scintilla widget by using syntect. The idea would be creating some sort of generic Scintilla Lexer that would allow me to do so... At this point I've created python bindings out of mostly of the public syntect API but I'm not sure if this idea of mine is a good idea or not. If you don't know too much about how QScintilla syntax highlighting works, here's a very good explanation about it. So, before trying anything and wasting any time I'd like to listen your thoughts about it, whether this is a good idea or not... and if it is, what'd be the best way to execute it and how long could take me. Thanks. Ps. I'm asking you before trying anything by myself cos I'm considering all your experience on this area & the fact you'll probably know all the pros & cons about using syntect highlighting with another text editors... It's not like I'm asking explicitely which or what function using... at this point I've already become familiar with the whole public API exposed by syntect, so... ;) opened by brupelo 65 SublimeText view.extract_scope feature Hi, first of all, let me tell you this project is really awesome and promising :D Right now I can proudly say I don't know anything about rust... even I don't understand how to use rust libraries on apps (definitely need to take some rust tutorial) but I'd really like to be able to use this library. Again, I don't know rust at all but I definitely know quite a lot about python and that's why I've decided I'm gonna create some python bindings out of your library so I'll be able to use it in some of my apps. Anyway, the first thing I'd like to know is whether SublimeText view.extract_scope function is available from your library (if not exactly... some functions that allow me to achieve the same)? Could you please post a little minimal example I can play with or providing some references to the functions I'd need to use? Btw, at first I thought implementing my own python library to duplicate Sublime's features would be my best chance to duplicate some Sublime's features on a Scintilla component... but that was before knowing there was this amazing library available ;) . I've known about this project thanks to ask about it on stackoverflow. Thanks in advance! Ps. I'm assuming you're already an expert about Sublime's internal API but if that's not the case, this particular function is intended to: extract_scope(point) Returns the extent of the syntax scope name assigned to the character at the given point. opened by brupelo 30 Enable multi-threaded use Background: I'm trying to write a service that receives source code via HTTP, parses it using syntect and then returns some information about it. I'm using Iron for it, but this issue would also occur with other uses. In Iron, the Handler trait which handles requests is required to be Send and Sync, meaning it can be reused between threads. So, the simplest implementation is to just call SyntaxSet::load_defaults_nonewlines() on every request, then figure out which SyntaxDefinition to use, and then parse the text. Unfortunately, this means that the syntax definitions have to be loaded for each request, which is quite a bit of work that could be avoided. You can see where this is going. I'd like to have a way to only load SyntaxSet once at startup and then share it between all the request handlers. Or if that's not possible, at least share the SyntaxDefinition between threads (and have a mutex protecting SyntaxSet). If we try to write that with the current syntect, it would look something like this: #[test] fn threading_with_shared_syntax_set() { let syntax_set = SyntaxSet::load_defaults_nonewlines(); let syntax = syntax_set.find_syntax_by_name("rust").unwrap(); thread::spawn(|| { let parse_state = ParseState::new(&syntax); let mut f = File::open("testdata/parser.rs").unwrap(); let mut reader = BufReader::new(f); for line in reader.lines() { let line = line.unwrap(); parse_state.parse_line(&line); } }); } And it currently fails to compile like this: [101] % cargo test threading Compiling syntect v1.0.0 (file:///Users/rstocker/Projects/rust/syntect) error[E0277]: the trait bound `std::rc::Rc<std::cell::RefCell<syntect::parsing::syntax_definition::Context>>: std::marker::Sync` is not satisfied --> tests/threading.rs:14:5 | 14 | thread::spawn(|| { | ^^^^^^^^^^^^^ | = note: `std::rc::Rc<std::cell::RefCell<syntect::parsing::syntax_definition::Context>>` cannot be shared between threads safely = note: required because it appears within the type `std::option::Option<std::rc::Rc<std::cell::RefCell<syntect::parsing::syntax_definition::Context>>>` = note: required because it appears within the type `syntect::parsing::SyntaxDefinition` = note: required because it appears within the type `&syntect::parsing::SyntaxDefinition` = note: required because of the requirements on the impl of `std::marker::Send` for `&&syntect::parsing::SyntaxDefinition` = note: required because it appears within the type `[closure@tests/threading.rs:14:19: 23:6 syntax:&&syntect::parsing::SyntaxDefinition]` = note: required by `std::thread::spawn` I tried to work around this with some unsafe code, but it panics (as expected) when multiple threads use the same SyntaxDefinition. As I understand the code currently, it is structured the way it is to avoid having to compile all the regexes eagerly. I wonder if there's a simple way to preserve that while still keeping that nice property. Alternatively, it would be cool if there was a way to precompile a SyntaxDefinition and get a new type that is immutable and thus Send and Sync. By the way, apart from this the library works beautifully :). opened by robinst 29 onig crate doesn't link properly for examples on Windows How to build the examples? I am on the examples folder doing rustc syncat.rs --extern syntect=.\..\target\debug\libsyntect.rlib and it is resulting: error[E0463]: can't find crate for `yaml_rust` which `syntect` depends on --> syncat.rs:1:1 | 1 | extern crate syntect; | ^^^^^^^^^^^^^^^^^^^^^ can't find crate error: aborting due to previous error It is seems to be asking for all the crates syntect depends on. Thought it looks like a lot of crates, how would you build an example more simply? opened by evandrocoan 21 bug/panic caused by Vue.js syntax definition When trying to highlight Vue.js files using the same syntax file that works well in Sublime, Syntect panics: thread '<unnamed>' panicked at 'begin <= end (13 <= 7) when slicing `<style lang="stylus" scoped>`', libcore/str/mod.rs:2098:5 note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. stack backtrace: 0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49 1: std::sys_common::backtrace::print at libstd/sys_common/backtrace.rs:71 at libstd/sys_common/backtrace.rs:59 2: std::panicking::default_hook::{{closure}} at libstd/panicking.rs:211 3: std::panicking::default_hook at libstd/panicking.rs:227 4: std::panicking::rust_panic_with_hook at libstd/panicking.rs:511 5: std::panicking::continue_panic_fmt at libstd/panicking.rs:426 6: rust_begin_unwind at libstd/panicking.rs:337 7: core::panicking::panic_fmt at libcore/panicking.rs:92 8: core::str::slice_error_fail at libcore/str/mod.rs:0 9: core::str::traits::<impl core::slice::SliceIndex<str> for core::ops::range::Range<usize>>::index::{{closure}} 10: <syntect::highlighting::highlighter::HighlightIterator<'a, 'b> as core::iter::iterator::Iterator>::next 11: <alloc::vec::Vec<T> as alloc::vec::SpecExtend<T, I>>::from_iter 12: syntect::html::highlighted_snippet_for_string 13: <std::thread::local::LocalKey< T>>::with 14: syntect_server::rocket_route_fn_index 15: rocket::rocket::Rocket::dispatch 16: <rocket::rocket::Rocket as hyper::server::Handler>::handle 17: <hyper::server::Worker<H>>::handle_connection 18: hyper::server::listener::spawn_with::{{closure}} (only the bits above syntect::html::highlighted_snippet_for_string are relevant) I believe the above is pointing to https://github.com/trishume/syntect/blob/231eafce087beac80064b902c9eadefb1d078c1f/src/highlighting/highlighter.rs#L119 as the culprit, but I haven't investigated what would cause this. The syntax file I am using is here: vue.sublime-syntax (tmLanguage file here) This file produces the panic (but highlights fine in ST3 with the same syntax file): https://raw.githubusercontent.com/vuejs/vue-cli/0ba111eed859aad02156e807ef71d0052b2e7f17/packages/%40vue/cli-ui-addon-webpack/src/components/ModuleListItem.vue This file highlights fine in both Syntect and ST3 with the same syntax file: https://raw.githubusercontent.com/vuejs/vue-cli/0ba111eed859aad02156e807ef71d0052b2e7f17/packages/%40vue/cli-ui-addon-webpack/src/components/ModuleList.vue opened by slimsag 20 600 line Rust source file takes more than 2 minutes How to reproduce (link to view file): curl -LO 'https://github.com/rust-lang/rust/raw/a6fe6c9be10c79d64c988d490519f5cc07e9060d/src/librustc_borrowck/borrowck/mir/dataflow/impls.rs' time target/release/examples/synstats impls.rs Running with syntect from master (currently 431174d97fe8df2fc798e11c96991e989b23a1e9), this is the output of time: target/release/examples/synstats impls.rs 141.49s user 0.54s system 99% cpu 2:22.71 total Sublime text build 3125 seems to have no problem with the file and loads instantly. opened by robinst 18 Wasm support Hi, I'm trying to build a pure wasm version of UnicornConsole (https://github.com/Gigoteur/UnicornConsole) which use syntect for code editing. But right now I'm getting an error with same-file crate dependency: --> /home/xx/.cargo/registry/src/github.com-1ecc6299db9ec823/same-file-0.1.3/src/lib.rs:90:9 | 90 | imp::Handle::from_path(p).map(Handle) | ^^^ Use of undeclared type or module imp error[E0433]: failed to resolve. Use of undeclared type or module imp --> /home/xx/.cargo/registry/src/github.com-1ecc6299db9ec823/same-file-0.1.3/src/lib.rs:95:9 I'm opening the issue here to understand if you need this dependency or not, because it seems not really needed ? Thank you opened by hallucino 17 Can't build on Windows because onig_sys doesn't build on Windows error: failed to run custom build command for onig_sys v61.1.0 I am compiling the rust program https://github.com/trishume/syntect by running make packs. However it throws the error: $ make packs cargo run --example gendata -- synpack testdata/Packages assets/default_newlines.packdump assets/default_nonewlines.packdump Compiling onig_sys v61.1.0 error: failed to run custom build command for `onig_sys v61.1.0` process didn't exit successfully: `D:\syntect\target\debug\build\onig_sys-523c993a9ee13532\build-script-build` (exit code: 101) ... --- stderr DIST_NAME: onig DIST_VERSION: 6.1.1 DIST_LICENSE: BSD DIST_AUTHOR: K.Kosako DIST_MAINTAINER: K.Kosako DIST_URL: https://github.com/kkos/oniguruma DIST_DESC: Oniguruma is a regular expressions library. DIST_DEPENDS: thread 'main' panicked at ' command did not execute successfully, got: exit code: 1 build script failed, must exit now', C:\Users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\cmake-0.1.21\src\lib.rs:605 note: Run with `RUST_BACKTRACE=1` for a backtrace. make: *** [packs] Error 101 Why the https://github.com/kkos/oniguruma is not compiling? I ran cargo build -vv and got this most all the log: D:\syntect>cargo build -vv Fresh num-traits v0.1.37 Fresh winapi-build v0.1.1 Fresh pkg-config v0.3.9 Fresh gcc v0.3.45 Fresh regex-syntax v0.4.0 Fresh num-integer v0.1.33 Fresh cmake v0.1.21 Fresh num-iter v0.1.33 Fresh lazy_static v0.2.4 Fresh winapi v0.2.8 Fresh yaml-rust v0.3.5 Fresh serde v0.8.23 Fresh bitflags v0.7.0 Fresh num v0.1.37 Fresh xml-rs v0.3.6 Fresh rustc-serialize v0.3.23 Fresh bitflags v0.8.0 Fresh fnv v1.0.5 Fresh serde v0.9.11 Fresh kernel32-sys v0.2.2 Fresh byteorder v1.0.0 Fresh bincode v0.6.1 Compiling onig_sys v61.1.0 Fresh same-file v0.1.3 Fresh libc v0.2.21 Running `D:\syntect\target\debug\build\onig_sys-523c993a9ee13532\build-scrip t-build` Fresh byteorder v0.5.3 Fresh walkdir v1.0.7 Fresh miniz-sys v0.1.9 Fresh time v0.1.36 Fresh flate2 v0.2.17 Fresh chrono v0.2.25 Fresh plist v0.1.2 running: "cmake" "C:\\Users\\Professional\\.cargo\\registry\\src\\github.com-1ecc6299db9ec823\\onig_sys-61.1.0\\oniguruma" "-G" "Visual Studio 14 2015 Win64" "-DBUILD_SHARED_LIBS=OFF" "- DCMAKE_INSTALL_PREFIX=D:\\syntect\\target\\debug\\build\\onig_sys-d633e 4c1e0d45051\\out" "-DCMAKE_C_FLAGS= /nologo /MD" "-DCMAKE_C_FLAGS_RELEASE= /nologo /MD" "-DCMAKE_CXX_FLAGS= /nologo /MD" "-DCMAKE_CXX_FLAGS_RELEASE= /nologo /MD" "-DCMAKE_BUILD_TYPE=Rele ase" DIST_NAME: onig DIST_VERSION: 6.1.1 DIST_LICENSE: BSD DIST_AUTHOR: K.Kosako DIST_MAINTAINER: K.Kosako DIST_URL: https://github.com/kkos/oniguruma DIST_DESC: Oniguruma is a regular expressions library. DIST_DEPENDS: -- Configuring done -- Generating done -- Build files have been written to: D:/syntect/target/debug/build/onig_sys-d633e 4c1e0d45051/out/build running: "cmake" "--build" "." "--target" "install" "--config" "Release" "--" Microsoft (R) Build Engine version 14.0.24723.2 Copyright (C) Microsoft Corporation. All rights reserved. Build started 17/03/2017 15:41:57. The target "BeforeGenerateProjectPriFile" listed in a BeforeTargets attribute at "C:\Program Files (x86)\MSBuild\Microsoft\NuGet\Microsoft.NuGet.targets (186,61)" does not exist in the p roject, and will be ignored. The target "BeforeGenerateProjectPriFile" listed in a BeforeTargets attribute at "C:\Program Files (x86)\MSBuild\Microsoft\NuGet\Microsoft.NuGet.targets (186,61)" does not exist in the p roject, and will be ignored. Project "D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\instal l.vcxproj" on node 1 (default targets). Project "D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\instal l.vcxproj" (1) is building "D:\syntect\target\debug\build\onig_sys-d633e4c1e0d450 51\out\build\ZERO_CHECK.vcxproj" (2) on node 1 (default targets). InitializeBuildStatus: Creating "x64\Release\ZERO_CHECK\ZERO_CHECK.tlog\unsuccessfulbuild" because "AlwaysCreate" was specified. CustomBuild: All outputs are up-to-date. Checking Build System CMake does not need to re-run because D:/syntect/target/debug/build/onig_sys-d6 33e4c1e0d45051/out/build/CMakeFiles/generate.stamp is up-to-date. FinalizeBuildStatus: Deleting file "x64\Release\ZERO_CHECK\ZERO_CHECK.tlog\unsuccessfulbuild". Touching "x64\Release\ZERO_CHECK\ZERO_CHECK.tlog\ZERO_CHECK.lastbuildstate". Done Building Project "D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\ou t\build\ZERO_CHECK.vcxproj" (default targets). The target "BeforeGenerateProjectPriFile" listed in a BeforeTargets attribute at "C:\Program Files (x86)\MSBuild\Microsoft\NuGet\Microsoft.NuGet.targets (186,61)" does not exist in the p roject, and will be ignored. The target "BeforeGenerateProjectPriFile" listed in a BeforeTargets attribute at "C:\Program Files (x86)\MSBuild\Microsoft\NuGet\Microsoft.NuGet.targets (186,61)" does not exist in the p roject, and will be ignored. Project "D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\instal l.vcxproj" (1) is building "D:\syntect\target\debug\build\onig_sys-d633e4c1e0d450 51\out\build\ALL_BUILD.vcxproj" (3) on node 1 (default targets). Project "D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\ALL_BU ILD.vcxproj" (3) is building "D:\syntect\target\debug\build\onig_sys-d633e4c1e0d4 5051\out\build\onig.vcxproj" (4) on node 1 (default targets). InitializeBuildStatus: Touching "onig.dir\Release\onig.tlog\unsuccessfulbuild". CustomBuild: All outputs are up-to-date. Building Custom Rule C:/Users/user/.cargo/registry/src/github.com-1ecc6299db9ec823/onig_sys-61.1.0/oniguruma/CMakeLists.txt CMake does not need to re-run because D:/syntect/target/debug/build/onig_sys-d6 33e4c1e0d45051/out/build/CMakeFiles/generate.stamp is up-to-date. ClCompile: F:\VisualStudio2015\VC\bin\x86_amd64\CL.exe /c /I"D:\syntect\target\debug\build \onig_sys-d633e4c1e0d45051\out\build" /I"C:\Users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma" /nologo /W1 /WX- /O2 /D _CRT_SECURE_NO_WARNINGS /D "CMAKE_INTDIR=\"Release\"" /D _MBCS /Gm- /MD /GS /fp:precise /Zc:wchar_t /Zc:forScope /Zc:inline /Fo"onig.dir\Release\\" /Fd"onig.dir\Release\onig.pdb" /Gd /TC /errorReport:queue "C:\ Users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.c" regenc.c C:\Users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.c(32): warning C4273: 'OnigEncDefaultCharEncoding': inconsistent dll linkage [D :\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\onig.vcxproj] c:\users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.h(227): note: see previous definition of 'OnigEncDefaultCharEncoding' C:\Users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.c(32): error C2099: initializer is not a constant [D:\User\\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\onig.vcxproj] C:\Users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.c(36): warning C4273: 'onigenc_init': inconsistent dll linkage [D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\onig.vcxproj] c:\users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\oniguruma.h(343): note: see previous definition of 'onigenc_init' c:\users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\oniguruma.h(345): note: see previous definition of 'onig_initialize_encoding' C:\Users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.c(53): warning C4273: 'onigenc_get_default_encoding': inconsistent dll linkage [D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\onig.vcxproj] c:\users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\oniguruma.h(349): note: see previous definition of 'onigenc_get_default_encoding' C:\Users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.c(59): warning C4273: 'onigenc_set_default_encoding': inconsistent dll linkage [D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\onig.vcxproj] c:\users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\oniguruma.h(347): note: see previous definition of 'onigenc_set_default_encoding' C:\Users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.c(66): warning C4273: 'onigenc_get_right_adjust_char_head': inconsistent dll li nkage [D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\onig.vcx proj] c:\users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\oniguruma.h(359): note: see previous definition of 'onigenc_get_right_adjust_char_he ad' C:\Users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.c(77): warning C4273: 'onigenc_get_right_adjust_char_head_with_prev': inconsist ent dll linkage [D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\buil d\onig.vcxproj] c:\users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\oniguruma.h(353): note: see previous definition of 'onigenc_get_right_adjust_char_he ad_with_prev' C:\Users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.c(92): warning C4273: 'onigenc_get_prev_char_head': inconsistent dll linkage [D :\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\onig.vcxproj] c:\users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\oniguruma.h(355): note: see previous definition of 'onigenc_get_prev_char_head' C:\Users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.c(101): warning C4273: 'onigenc_step_back': inconsistent dll linkage [D:\User\D ropbox\Applications\SoftwareVersioning\SublimeText\Data\Packages\ObjectBeautifier\source\libraries\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\onig.vcxproj] c:\users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\oniguruma.h(338): note: see previous definition of 'onigenc_step_back' C:\Users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.c(127): warning C4273: 'onigenc_step': inconsistent dll linkage [D:\User\Dropbo x\Applications\SoftwareVersioning\SublimeText\Data\Packages\ObjectBeautifier\source\libraries\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\onig.vcxproj] c:\users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.h(222): note: see previous definition of 'onigenc_step' C:\Users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.c(137): warning C4273: 'onigenc_strlen': inconsistent dll linkage [D:\User\Drop box\Applications\SoftwareVersioning\SublimeText\Data\Packages\ObjectBeautifier\source\libraries\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\onig.vcxproj] c:\users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\oniguruma.h(361): note: see previous definition of 'onigenc_strlen' C:\Users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.c(150): warning C4273: 'onigenc_strlen_null': inconsistent dll linkage [D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\onig.vcxproj] c:\users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\oniguruma.h(363): note: see previous definition of 'onigenc_strlen_null' C:\Users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.c(175): warning C4273: 'onigenc_str_bytelen_null': inconsistent dll linkage [D: \syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\onig.vcxproj] c:\users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\oniguruma.h(365): note: see previous definition of 'onigenc_str_bytelen_null' C:\Users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.c(197): warning C4273: 'OnigEncAsciiToLowerCaseTable': inconsistent dll linkage [D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\onig.vcxproj] c:\users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.h(228): note: see previous definition of 'OnigEncAsciiToLowerCaseTable' C:\Users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.c(269): warning C4273: 'OnigEncAsciiCtypeTable': inconsistent dll linkage [D:\U ser\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\onig.vcxproj] c:\users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.h(230): note: see previous definition of 'OnigEncAsciiCtypeTable' C:\Users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.c(304): warning C4273: 'OnigEncISO_8859_1_ToLowerCaseTable': inconsistent dll l inkage [D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\onig.vc xproj] c:\users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.h(216): note: see previous definition of 'OnigEncISO_8859_1_ToLowerCaseTable' "D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\install.vcxpro j" (default target) (1) -> "D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\ALL_BUILD.vcxp roj" (default target) (3) -> "D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\onig.vcxproj" (default target) (4) -> (ClCompile target) -> C:\Users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.c(32): error C2099: initializer is not a constant [D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\onig.vcxproj] 45 Warning(s) 1 Error(s) Time Elapsed 00:00:03.85 thread 'main' panicked at ' command did not execute successfully, got: exit code: 1 build script failed, must exit now', C:\Users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\cmake-0.1.21\src\lib.rs:605 note: Run with `RUST_BACKTRACE=1` for a backtrace. error: failed to run custom build command for `onig_sys v61.1.0` process didn't exit successfully: `D:\syntect\target\debug\build\onig_sys-523c993 a9ee13532\build-script-build` (exit code: 101) --- stdout running: "cmake" "C:\\Users\\Professional\\.cargo\\registry\\src\\github.com-1ecc6299db9ec823\\onig_sys-61.1.0\\oniguruma" "-G" "Visual Studio 14 2015 Win64" "-DBUILD_SHARED_LIBS=OFF" "- DCMAKE_INSTALL_PREFIX=D:\\User\\syntect\\target\\debug\\build\\onig_sys-d633e 4c1e0d45051\\out" "-DCMAKE_C_FLAGS= /nologo /MD" "-DCMAKE_C_FLAGS_RELEASE= /nologo /MD" "-DCMAKE_CXX_FLAGS= /nologo /MD" "-DCMAKE_CXX_FLAGS_RELEASE= /nologo /MD" "-DCMAKE_BUILD_TYPE=Rele ase" -- Configuring done -- Generating done -- Build files have been written to: D:/syntect/target/debug/build/onig_sys-d633e 4c1e0d45051/out/build running: "cmake" "--build" "." "--target" "install" "--config" "Release" "--" Microsoft (R) Build Engine version 14.0.24723.2 Copyright (C) Microsoft Corporation. All rights reserved. Build started 17/03/2017 15:41:57. The target "BeforeGenerateProjectPriFile" listed in a BeforeTargets attribute at "C:\Program Files (x86)\MSBuild\Microsoft\NuGet\Microsoft.NuGet.targets (186,61)" does not exist in the p roject, and will be ignored. The target "BeforeGenerateProjectPriFile" listed in a BeforeTargets attribute at "C:\Program Files (x86)\MSBuild\Microsoft\NuGet\Microsoft.NuGet.targets (186,61)" does not exist in the p roject, and will be ignored. Project "D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\instal l.vcxproj" on node 1 (default targets). Project "D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\instal l.vcxproj" (1) is building "D:\syntect\target\debug\build\onig_sys-d633e4c1e0d450 51\out\build\ZERO_CHECK.vcxproj" (2) on node 1 (default targets). InitializeBuildStatus: Creating "x64\Release\ZERO_CHECK\ZERO_CHECK.tlog\unsuccessfulbuild" because "AlwaysCreate" was specified. CustomBuild: All outputs are up-to-date. Checking Build System CMake does not need to re-run because D:/syntect/target/debug/build/onig_sys-d6 33e4c1e0d45051/out/build/CMakeFiles/generate.stamp is up-to-date. FinalizeBuildStatus: Deleting file "x64\Release\ZERO_CHECK\ZERO_CHECK.tlog\unsuccessfulbuild". Touching "x64\Release\ZERO_CHECK\ZERO_CHECK.tlog\ZERO_CHECK.lastbuildstate". Done Building Project "D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\ou t\build\ZERO_CHECK.vcxproj" (default targets). The target "BeforeGenerateProjectPriFile" listed in a BeforeTargets attribute at "C:\Program Files (x86)\MSBuild\Microsoft\NuGet\Microsoft.NuGet.targets (186,61)" does not exist in the p roject, and will be ignored. The target "BeforeGenerateProjectPriFile" listed in a BeforeTargets attribute at "C:\Program Files (x86)\MSBuild\Microsoft\NuGet\Microsoft.NuGet.targets (186,61)" does not exist in the p roject, and will be ignored. Project "D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\instal l.vcxproj" (1) is building "D:\syntect\target\debug\build\onig_sys-d633e4c1e0d450 51\out\build\ALL_BUILD.vcxproj" (3) on node 1 (default targets). Project "D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\ALL_BU ILD.vcxproj" (3) is building "D:\syntect\target\debug\build\onig_sys-d633e4c1e0d4 5051\out\build\onig.vcxproj" (4) on node 1 (default targets). InitializeBuildStatus: Touching "onig.dir\Release\onig.tlog\unsuccessfulbuild". CustomBuild: All outputs are up-to-date. Building Custom Rule C:/Users/user/.cargo/registry/src/github.com-1ecc6299db9ec823/onig_sys-61.1.0/oniguruma/CMakeLists.txt CMake does not need to re-run because D:/syntect/target/debug/build/onig_sys-d6 33e4c1e0d45051/out/build/CMakeFiles/generate.stamp is up-to-date. ClCompile: F:\VisualStudio2015\VC\bin\x86_amd64\CL.exe /c /I"D:\syntect\target\debug\build \onig_sys-d633e4c1e0d45051\out\build" /I"C:\Users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma" /nologo /W1 /WX- /O2 /D _CRT_SECURE_NO_WARNINGS /D "CMAKE_INTDIR=\"Release\"" /D _MBCS /Gm- /MD /GS /fp:precise /Zc:wchar_t /Zc:forScope /Zc:inline /Fo"onig.dir\Release\\" /Fd"onig.dir\Release\onig.pdb" /Gd /TC /errorReport:queue "C:\ Users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.c" regenc.c C:\Users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.c(32): warning C4273: 'OnigEncDefaultCharEncoding': inconsistent dll linkage [D :\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\onig.vcxproj] c:\users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.h(227): note: see previous definition of 'OnigEncDefaultCharEncoding' C:\Users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.c(32): error C2099: initializer is not a constant [D:\User\\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\onig.vcxproj] C:\Users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.c(36): warning C4273: 'onigenc_init': inconsistent dll linkage [D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\onig.vcxproj] c:\users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\oniguruma.h(343): note: see previous definition of 'onigenc_init' c:\users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\oniguruma.h(345): note: see previous definition of 'onig_initialize_encoding' C:\Users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.c(53): warning C4273: 'onigenc_get_default_encoding': inconsistent dll linkage [D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\onig.vcxproj] c:\users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\oniguruma.h(349): note: see previous definition of 'onigenc_get_default_encoding' C:\Users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.c(59): warning C4273: 'onigenc_set_default_encoding': inconsistent dll linkage [D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\onig.vcxproj] c:\users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\oniguruma.h(347): note: see previous definition of 'onigenc_set_default_encoding' C:\Users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.c(66): warning C4273: 'onigenc_get_right_adjust_char_head': inconsistent dll li nkage [D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\onig.vcx proj] c:\users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\oniguruma.h(359): note: see previous definition of 'onigenc_get_right_adjust_char_he ad' C:\Users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.c(77): warning C4273: 'onigenc_get_right_adjust_char_head_with_prev': inconsist ent dll linkage [D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\buil d\onig.vcxproj] c:\users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\oniguruma.h(353): note: see previous definition of 'onigenc_get_right_adjust_char_he ad_with_prev' C:\Users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.c(861): warning C4273: 'onigenc_mb4_is_code_ctype': inconsistent dll linkage [D :\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\onig.vcxproj] c:\users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.h(149): note: see previous definition of 'onigenc_mb4_is_code_ctype' C:\Users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.c(876): warning C4273: 'onigenc_with_ascii_strncmp': inconsistent dll linkage [ D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\onig.vcxproj] c:\users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.h(220): note: see previous definition of 'onigenc_with_ascii_strncmp' Done Building Project "D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\ou t\build\onig.vcxproj" (default targets) -- FAILED. Done Building Project "D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\ou t\build\ALL_BUILD.vcxproj" (default targets) -- FAILED. Done Building Project "D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\ou t\build\install.vcxproj" (default targets) -- FAILED. Build FAILED. "D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\install.vcxpro j" (default target) (1) -> "D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\ALL_BUILD.vcxp roj" (default target) (3) -> "D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\onig.vcxproj" (default target) (4) -> (ClCompile target) -> C:\Users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.c(32): warning C4273: 'OnigEncDefaultCharEncoding': inconsistent dll linkage [D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\onig.vcxproj] C:\Users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.c(861): warning C4273: 'onigenc_mb4_is_code_ctype': inconsistent dll linkage [D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\onig.vcxproj] C:\Users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.c(876): warning C4273: 'onigenc_with_ascii_strncmp': inconsistent dll linkage [D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\onig.vcxproj] "D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\install.vcxpro j" (default target) (1) -> "D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\ALL_BUILD.vcxp roj" (default target) (3) -> "D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\onig.vcxproj" (default target) (4) -> (ClCompile target) -> C:\Users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\onig_sys-61.1.0\oniguruma\src\regenc.c(32): error C2099: initializer is not a constant [D:\syntect\target\debug\build\onig_sys-d633e4c1e0d45051\out\build\onig.vcxproj] 45 Warning(s) 1 Error(s) Time Elapsed 00:00:03.85 --- stderr DIST_NAME: onig DIST_VERSION: 6.1.1 DIST_LICENSE: BSD DIST_AUTHOR: K.Kosako DIST_MAINTAINER: K.Kosako DIST_URL: https://github.com/kkos/oniguruma DIST_DESC: Oniguruma is a regular expressions library. DIST_DEPENDS: thread 'main' panicked at ' command did not execute successfully, got: exit code: 1 build script failed, must exit now', C:\Users\User\.cargo\registry\src\github.com-1ecc6299db9ec823\cmake-0.1.21\src\lib.rs:605 note: Run with `RUST_BACKTRACE=1` for a backtrace. D:\syntect> opened by evandrocoan 16 added syntest example to run ST syntax tests This PR adds a new example called "syntest", which will parse and execute ST's "syntax_test_" files. Currently, there is no way to reference a syntax definition in a SyntaxSet from it's original file path, which is how tests specify which syntax definition to use, so it just lets syntect choose based on the extension of the test file at the moment. I used the regular "regex" crate to achieve the test line parsing - feel free to replace this with "onig" if you don't want the extra reference - I don't quite have enough experience yet to know how to use it effectively. Also, if the syntax test files use Windows line endings, it has to replace "\r" with nothing, because otherwise the regular expressions don't match expressions like "$\n?" properly, but perhaps it would be better to update the main parsing module to fix this? I've checked and it shows success on the XML, HTML and JSON syntax tests. I also tried modifying them to cause a test to fail, and syntest correctly picked that up. However, it seems to detect a failure in the Haskell syntax test file, and it looks like some meta scopes are being applied multiple times - so I'd appreciate if you could take a look and see if it's a flaw in my syntest logic somewhere or a bug in the parser. Thanks! cargo run --example syntest testdata/Packages/Haskell/ Finished debug [unoptimized + debuginfo] target(s) in 0.0 secs Running `target/debug/examples/syntest testdata/Packages/Haskell/` Testing file testdata/Packages/Haskell/syntax_test_haskell.hs The test file references syntax definition file: Packages/Haskell/Haskell.sublime-syntax Assertion selector "- comment.line.double-dash.haskell" from line 6 failed on line 5, column range 0-1 (with text ['2']) has scope [<source.haskell>, <comment.line.double-dash.haskell>, <comment.line.double-dash.haskell>, <comment.line.double-dash.haskell>, <comment.line.double-dash.haskell>, <constant.numeric.haskell>] Ok(FailedAssertions(1, 422)) opened by keith-hall 15 Move all regex usage to separate module to add support for fancy-regex This has the same goal as #34 but with a different approach. Note that the fancy-regex implementation doesn't compile yet, but I thought it would be useful to get this reviewed earlier rather than later. I haven't ported over the regex rewriting changes yet, I'm hoping that we can generate regexes that work on both onig and fancy-regex. [x] Add std::error::Error impl for fancy-regex, see https://github.com/fancy-regex/fancy-regex/pull/35 [x] Release https://github.com/fancy-regex/fancy-regex/pull/33 [x] Port regex rewriting changes opened by robinst 14 Hang while highlighting JavaScript using Packages#v4050 When attempting to highlight some JS code we noticed that it would never return and just spin consuming a full core. We're using syntect with a custom pack built from mostly https://github.com/sublimehq/Packages/releases/tag/v4050. The code: { // this is an inline comment "foo": "bar" // another inline comment /* This is a block comment that continues on another line */ } This hangs while parsing the /* line. I've pushed a complete testcase, but since this requires a newer version of sublimehq/Packages than currently tested against I expect that this would need a smaller syntax file to be synthesized for a proper testcase. opened by Nemo157 3 mlir: mlir is currently not syntax highlighted It seems that sublimehq/packages doesn't include syntax highlighting for MLIR files. Worth noting that they're syntax-highlighted on the GitHub interface, as well as on VSCode, with the MLIR package. opened by artagnon 0 Fix `split_at` (decrease index) Fixed #453 See also another PR #455 . This PR fixes split_at, which can panic when splitting occurs inside the middle of a wide character, to decrese index to avoid panic (In this version, split_at returns the same type as before). Also includes rust-analyzer's formatting. Test results: commit 8678b7c68a7fbd195733aa23dcaa3dbaedaceb10 failures: highlighting::highlighter::tests::can_parse highlighting::highlighter::tests::can_parse_with_highlight_state_from_cache highlighting::highlighter::tests::test_ranges highlighting::theme_set::tests::can_parse_common_themes html::tests::strings html::tests::tricky_test_syntax parsing::parser::tests::can_compare_parse_states parsing::parser::tests::can_parse_backrefs parsing::parser::tests::can_parse_includes parsing::parser::tests::can_parse_issue25 parsing::parser::tests::can_parse_preprocessor_rules parsing::parser::tests::can_parse_simple parsing::parser::tests::can_parse_yaml parsing::syntax_set::tests::can_load test result: FAILED. 86 passed; 14 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.11s fix-split_at-change-index failures: highlighting::highlighter::tests::can_parse highlighting::highlighter::tests::can_parse_with_highlight_state_from_cache highlighting::highlighter::tests::test_ranges highlighting::theme_set::tests::can_parse_common_themes html::tests::strings html::tests::tricky_test_syntax parsing::parser::tests::can_compare_parse_states parsing::parser::tests::can_parse_backrefs parsing::parser::tests::can_parse_includes parsing::parser::tests::can_parse_issue25 parsing::parser::tests::can_parse_preprocessor_rules parsing::parser::tests::can_parse_simple parsing::parser::tests::can_parse_yaml parsing::syntax_set::tests::can_load test result: FAILED. 86 passed; 14 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.00s Nothing changed. opened by kyoheiu 0 Fix `split_at` (return Result ver.) Fixed #453 See also another PR #456 . This PR fixes split_at, which can panic when splitting occurs inside the middle of a wide character, to return Result to avoid panic. Also includes some of rust-analyzer's formatting. Test results: commit 8678b7c68a7fbd195733aa23dcaa3dbaedaceb10 failures: highlighting::highlighter::tests::can_parse highlighting::highlighter::tests::can_parse_with_highlight_state_from_cache highlighting::highlighter::tests::test_ranges highlighting::theme_set::tests::can_parse_common_themes html::tests::strings html::tests::tricky_test_syntax parsing::parser::tests::can_compare_parse_states parsing::parser::tests::can_parse_backrefs parsing::parser::tests::can_parse_includes parsing::parser::tests::can_parse_issue25 parsing::parser::tests::can_parse_preprocessor_rules parsing::parser::tests::can_parse_simple parsing::parser::tests::can_parse_yaml parsing::syntax_set::tests::can_load test result: FAILED. 86 passed; 14 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.11s fix-split_at-result failures: highlighting::highlighter::tests::can_parse highlighting::highlighter::tests::can_parse_with_highlight_state_from_cache highlighting::highlighter::tests::test_ranges highlighting::theme_set::tests::can_parse_common_themes html::tests::strings html::tests::tricky_test_syntax parsing::parser::tests::can_compare_parse_states parsing::parser::tests::can_parse_backrefs parsing::parser::tests::can_parse_includes parsing::parser::tests::can_parse_issue25 parsing::parser::tests::can_parse_preprocessor_rules parsing::parser::tests::can_parse_simple parsing::parser::tests::can_parse_yaml parsing::syntax_set::tests::can_load test result: FAILED. 86 passed; 14 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.00s which means nothing is changed by this PR except split_at and its test. opened by kyoheiu 2 util::split_at panics with multibyte characters Sorry if duplicated. To reproduce panic, run the following code: use syntect::easy::HighlightLines; use syntect::highlighting::{Style, ThemeSet}; use syntect::parsing::SyntaxSet; use syntect::util::{as_24_bit_terminal_escaped, split_at, LinesWithEndings}; fn main() { let ps = SyntaxSet::load_defaults_newlines(); let ts = ThemeSet::load_defaults(); let syntax = ps.find_syntax_by_extension("rs").unwrap(); let mut h = HighlightLines::new(syntax, &ts.themes["base16-ocean.dark"]); let s = "日本の首都は東京です"; for line in LinesWithEndings::from(s) { let ranges: Vec<(Style, &str)> = h.highlight_line(line, &ps).unwrap(); //thread 'main' panicked at 'byte index 4 is not a char boundary; it is inside '本' (bytes 3..6) of `日本の首都は東京です`', library/core/src/str/mod.rs:127:5 let ranges = split_at(&ranges, 4); let escaped = as_24_bit_terminal_escaped(&ranges.0, false); print!("{}", escaped); println!(); } } I think it would be better to return Result<(Vec..., Vec...), Error> instead of (Vec... , Vec...) to avoid this type of panic. bug opened by kyoheiu 3 Releases(v5.0.0) v5.0.0(May 4, 2022) Thanks to @Enselic for basically all the work to make this release happen! Breaking changes Lazy-load syntaxes to significantly improve startup time. This changes the binary format of syntax dump files. Remove ContextId::new() from public API to support lazy-loading of syntaxes Rename HighlightLines::highlight() to HighlightLines::highlight_line() to make it clear that the function takes one line at a time Make plist dependency (used for loading themes) optional via new plist-load feature Remove obsolete dump-load-rs and dump-create-rs features that has been identical to dump-load and dump-create for two years Remove deprecated items ThemeSettings::highlight_foreground, ThemeSettings::selection_background, ClassedHTMLGenerator::new, ClassedHTMLGenerator::parse_html_for_line, html::css_for_theme, html::tokens_to_classed_html and html::tokens_to_classed_spans Mark all error enums as #[non_exhaustive] These functions have been changed to return a Result to allow propagation of errors: html::ClassedHTMLGenerator::parse_html_for_line_which_includes_newline html::append_highlighted_html_for_styled_line html::css_for_theme_with_class_style html::highlighted_html_for_string html::line_tokens_to_classed_spans html::styled_line_to_highlighted_html parsing::ParseState::parse_line parsing::ScopeStack::apply parsing::ScopeStack::apply_with_hook parsing::syntax_definition::Context::match_at parsing::syntax_definition::ContextReference::id parsing::syntax_definition::ContextReference::resolve Other changes Fall back to Plain Text if a referenced syntax is missing Add support for hidden_file_extensions key in syntaxes. Implement Error and Display for all error enums by using thiserror Replace lazycell with once_cell to fix crash on lazy initialization Add ScopeRangeIterator Add CI check for Minimum Supported Rust Version. This is currently Rust 1.53. Make looking up a syntax by extension use case-insensitive comparison Make from_dump_file() ~15% faster Blend alpha value on converting colors to ANSI color sequences Fix sample code in documentation to avoid double newlines Fix lots of build warnings and lints Add Criterion benchmarks for a whole syntect pipeline and for from_dump_file() Source code(tar.gz) Source code(zip) v4.7.1(Jan 2, 2022) See the release notes for v4.7.0, this release removes a new Cargo feature which constituted a semver violation: Remove 'plist-load' feature again due to semver violation. #403 Source code(tar.gz) Source code(zip) v4.7.0(Dec 26, 2021) Big release this time thanks to tons of fantastic contributions from @Enselic, this release was basically all him! The headline feature is much faster startup time due to lazy-loading at the syntax level. Lazy-load syntaxes to significantly improve startup time Remove ContextId::new() from public API to support lazy-loading of syntaxes. This is technically a breaking change but I have no idea why I made this API public and I'm pretty sure zero people have used it. Add ScopeRangeIterator Add CI check for Minimum Supported Rust Version. This is currently Rust 1.51. Make 'plist' dependency (used for loading themes) optional via new 'plist-load' feature Make looking up a syntax by extension use case-insensitive comparison Make from_dump_file() ~15% faster Blend alpha value on converting colors to ANSI color sequences Fix sample code in documentation to avoid double newlines Fix lots of build warnings and lints Add Criterion benchmarks for a whole syntect pipeline and for from_dump_file() Replace lazycell with once_cell to fix crash on lazy initialization Source code(tar.gz) Source code(zip) v4.6.0(Aug 1, 2021) Add html::line_tokens_to_classed_spans to also take a mutable ScopeStack, deprecate tokens_to_classed_spans, to avoid panics and incorrect highlighting. Derive Hash for Color and Style Add find_unlinked_contexts to SyntaxSet Add syntaxes method to SyntaxSetBuilder Bump fancy-regex to v0.7 and yaml-rust to v0.4.5 Source code(tar.gz) Source code(zip) v4.5.0(Dec 9, 2020) Added a new function for producing classed HTML which handles newlines correctly and deprecated old one. #307 Source code(tar.gz) Source code(zip) v4.4.0(Aug 20, 2020) Errors are now Send + Sync + 'static #304 I also forgot to make a Github release for v4.3.0 but released it on crates.io so belatedly in v4.3.0: Fixes unnecesary dependency of the html feature on the assets feature. #300 Adds ability to add prefixes to html module CSS class names. #296 Source code(tar.gz) Source code(zip) v4.2.0(May 23, 2020) Updates to new versions of onig and plist. The new onig version doesn't require bindgen thus making compilation easier. #293 Source code(tar.gz) Source code(zip) v4.1.1(Apr 21, 2020) Properly handle backreferences in included contexts #288 Source code(tar.gz) Source code(zip) v4.1.0(Mar 31, 2020) Make sure errors implement Send #285 Fix errors to not use the deprecated description() #286 Thanks @sharkdp for the bug fixes! Bumping second part of semver since Send is adding functionality (back). Source code(tar.gz) Source code(zip) v4.0.0(Mar 29, 2020) Headline feature: pure-Rust fancy-regex engine option Users can now opt in to a pure-Rust regex engine using Cargo features, making compilation easier in general. People experiencing difficulty compiling for Windows and Wasm should try switching to fancy-regex. Note this currently approximately halves highlighting speed. See the Readme and #270 for details. Thanks to @robinst for implementing this! Other changes Ability to generate CSS for a theme for use with classed HTML generation (won't always be correct) #274 Don't generate empty spans in classed HTML #276 Miscellaneous dependency bumps and cleanup Breaking changes and upgrading Upgrading should cause no errors for nearly all users. Users using more unusual APIs may have a small amount of tweaking to do. Edit March 30: If you generate custom pack files and want to use fancy-regex you need to regenerate them. The binary format is the same but at YAML loading time regex rewrites get applied that make fancy-regex work properly. If you use default-features = false you may need to update your features to choose a regex engine A bunch of technically public APIs that I don't know if anyone uses changed due to the regex engine refactor, common uses shouldn't break Source code(tar.gz) Source code(zip) v3.3.0(Sep 22, 2019) Fixes multiple bugs Add RangedHighlightIterator Add as_latex_escaped util Source code(tar.gz) Source code(zip) v3.2.1(Aug 10, 2019) Hopefully no behavior changes. Source code(tar.gz) Source code(zip) v3.2.0(Mar 9, 2019) Really a bug fix but the bug was not having an API be public so hence SemVer demands a minor version bump. Source code(tar.gz) Source code(zip) v3.1.0(Feb 24, 2019) Add support for loading metadata (#223 #225 #230) Improve support for generating classed HTML and fix a bug, old function is deprecated because it's impossible to use correctly (#235) Update plist to v0.4 and pretty_assertions to v0.6 (#232 #236) Source code(tar.gz) Source code(zip) v3.0.2(Nov 11, 2018) Fix application of multiple with_prototypes (#220, fixes #160, #178, ASP highlighting) Fix prototype marking logic (#221, fixes #219) Source code(tar.gz) Source code(zip) v3.0.1(Oct 16, 2018) Fix a bug with syntaxes that used captures in lookarounds (#176 #215) Fix the precedence order of syntaxes to match Sublime (#217 #216) See previous release for major breaking changes and new things. Source code(tar.gz) Source code(zip) v3.0.0(Oct 9, 2018) This is a major release with multiple breaking API changes, although upgrading shouldn't be too difficult. It fixes bugs and comes with some nice new features. Breaking changes and upgrading The SyntaxSet API has been revamped to use a builder and an arena of contexts. See example usage. Many functions now need to be passed the SyntaxSet that goes with the rest of their arguments because of this new arena. Filename added to LoadingError::ParseSyntax Many functions in the html module now take the newlines version of syntaxes. These methods have also been renamed, partially so that code that needs updating doesn't break without a compile error. The HTML they output also treats newlines slightly differently and I think more correctly but uglier when you look at the HTML. Breaking rename upgrade guide SyntaxSet::add_syntax -> SyntaxSetBuilder::add SyntaxSet::load_syntaxes -> SyntaxSetBuilder::add_from_folder SyntaxSet::load_plain_text_syntax -> SyntaxSetBuilder::add_plain_text_syntax html::highlighted_snippet_for_string -> html::highlighted_html_for_string: also change to newlines SyntaxSet html::highlighted_snippet_for_file -> html::highlighted_html_for_file: also change to newlines SyntaxSet html::styles_to_coloured_html -> html::styled_line_to_highlighted_html: also change to newlines SyntaxSet html::start_coloured_html_snippet -> html::start_highlighted_html_snippet: return type also changed Major changes and new features Use arena for contexts (#182 #186 #187 #190 #195): This makes the code cleaner, enables use of syntaxes from multiple threads, and prevents accidental misuse. This involves a new SyntaxSetBuilder API for constructing new SyntaxSets See the revamped parsyncat example. Encourage use of newlines (#197 #207 #196): The nonewlines mode is often buggy so we made it easier to use the newlines mode. Added a LinesWithEndings utility for iterating over the lines of a string with \n characters. Reengineer the html module to use newlines syntaxes. Add helpers for modifying highlighted lines (#198): For use cases like highlighting a piece of text in a blog code snippet or debugger. This allows you to reach into the highlighted spans and add styles. Check out split_at and modify_range in the util module. New ThemeSet::add_from_folder function (#200): For modifying existing theme sets. Bug Fixes Improve nonewlines regex rewriting: #212 #211 Reengineer theme application to match Sublime: #209 Also mark contexts referenced by name as "no prototype" (same as ST): #180 keep with_prototype when switching contexts with set: #177 #166 Fix unused import warning: #174 Ignore trailing dots in selectors: #173 Fix embed to not include prototypes: #172 #160 Upgraded dependencies plist: 0.2 -> 0.3 regex: 0.2 -> 1.0 onig: 3.2.1 -> 4.1 Source code(tar.gz) Source code(zip) v2.1.0(May 31, 2018) Check regexes compile upon loading from YAML (There's technically a small breaking change here if you match on the previously unused regex error, but I don't think anyone does) Can detect the correct syntax on full file names like CMakeLists.txt Make nonewlines mode marginally less buggy (still prefer using newlines mode) Better error types Better examples and tests Source code(tar.gz) Source code(zip) v2.0.1(Apr 28, 2018) Parsing now abandons a regex after reaching a recursion depth limit instead of taking forever Loop detection better matches Sublime Text Parsing is faster! Dependency upgrades Other minor tweaks Thanks to @robinst for the headline features of this release! Source code(tar.gz) Source code(zip) v2.0.0(Jan 2, 2018) Breaking changes: The static-onig feature was removed, static linking is now the default Font styles and color constants now use associated consts because of bitflags upgrade SyntaxDefinition::load_from_str now has an extra parameter Other notable changes: Support for new embed syntax #124 Updates to many dependencies Updated dumps More compact HTML output Source code(tar.gz) Source code(zip) v1.8.2(Nov 11, 2017) Source code(tar.gz) Source code(zip) v1.8.0(Oct 13, 2017) This release changes how the constants for FontStyle and Color, relying on the new associated consts feature in Rust 1.20. The old constants are still available but are deprecated and will be removed in v2.0. Packages were also updated to newer versions. Source code(tar.gz) Source code(zip) v1.7.3(Sep 14, 2017) Fixes comparisons of parse states so they are fast and don't recurse infinitely. Thanks @raphlinus Source code(tar.gz) Source code(zip) v1.7.2(Sep 5, 2017) Fixes #101, which caused some syntaxes like PHP to behave incorrectly. Updates Packages with new syntax versions Adds new handy flags to the syncat example Source code(tar.gz) Source code(zip) v1.7.0(Jun 30, 2017) Source code(tar.gz) Source code(zip) v1.6.0(Jun 21, 2017) Source code(tar.gz) Source code(zip) v1.5.0(May 31, 2017) Small release, adds a convenience method for highlighting an entire stack, and derives some more things on Scope. Source code(tar.gz) Source code(zip) v1.4.0(May 25, 2017) This release switches the dump format from rustc-serialize to Serde, anyone using custom dumps will have to update them. It also makes the parsing part of the library optional behind a feature flag, anyone not using the default feature flags probably will want to add the parsing flag. Source code(tar.gz) Source code(zip) v1.3.0(Apr 5, 2017) This is one of the largest releases yet thanks to new contributors and me getting around to doing more work! Things in this version: Syntax tests: there is a new syntest example for running Sublime Text syntax tests Bug fixes: there's a ton of bugs fixed in this release, mostly found via the syntax tests. These mostly affected certain syntaxes which pushed/set multiple contexts at once. Updated packages: The Sublime packages have been updated to the latest version Feature flags: there's now Cargo feature flags for disabling some parts of syntect if you don't want unnecessary binary and dependency bloat. Source code(tar.gz) Source code(zip)