There are some core properties that are commonly used in Unicode algorithsm, as well as in applications directly, and are not specific to any single area. Examples of these properties are Alphabetic and White_Space.

Also, there are some resolved properties that are used commonly, like Numeric and Alphanumeric, which are commonly defined as based on General_Category and Alphabetic properties. Since they are common in applications, it makes sense to provided optimized implementations.

This new componet, unic-ucd-common, hosts these properties.

Right now, we have Age as this:

https://github.com/behnam/rust-unic/blob/72ca9a893373f1857bc6ab6440389dfdceea6f13/unic/ucd/age/src/age.rs#L36-L42

1. It's nice to have this CompleteCharProperty, as it gives meaningful API like is_assigned() and is_unassigned().

2. Another option is to convert this to a PartialCharProperty, which returns None if the char is Unassigned, and Some(UnicodeVersion) otherwise.

3. One more option would be to have it as PartialCharProperty, but the return type will be more following the UAX#44 spec: only having major and minor numbers. Basically, return value would be Some(Age), with Age being similar to UnicodeVersion, but without minor number. Then, we can have transformations between Age and UnicodeVersion.

Option (3) makes it much more like other types in UCD, option (2) is a bit farther, and option (1), which is the current one, is the farthest from general approach here.

BUT, there's also the fact that option (1) looks nicer and more organic comparing to the other ones.

I'm filing this mostly because I need to figure out a way to provide a property-range contract for Age, and would be great if it's not just an empty contract.

Any ideas?

This is a work in progress for implementing Unicode Name - NR2 (#171).

• [x] NR2: CJK Unified Ideograph / Tangut Ideograph / Nushu Character / CJK Compatibility Ideograph
• [x] Add tests

Feedback is welcome!

The former implementation adds an unnecessary allocation to the Display impl. Because implementing Display gives the to_string method for free, there shouldn't be any Display methods.

If there are any methods which offer strings that differ from the Display implementation, those should have different names. I deprecated the methods instead of removing them, just to be safe.

Add UCD Segmentation source data to /data/, implement conversion from new files to property map data tables.

Add unic-ucd-segment component with initial implementation of three main segmentation-related properties:

• Grapheme_Cluster_Break,
• Word_Break, and
• Sentence_Break.

Current implementation uses char_property!() macro for EnumeratedCharProperty implementation, which only supports TotalCharProperty.

Since the Other (abbr: XX) value in all these properties are notions of non-existance of breaking property, we want to switch to PartialCharProperty domain type and use Option<enum>. This is left as a separate step because it needs changes to the macro.

Tracker: https://github.com/behnam/rust-unic/issues/135

Do we want to add display() as an instance method to CharProperty API? We already have impl fmt::Display for it, but returning &str would be useful in cases that the display string is not expected to be matched with formatting.

I have seen both cases in third-party libraries.

What do you think?

Replaces #41. See #41 for earlier discussion.

An example will show better than I can tell:

char_property! {
    /// Represents the Unicode character
    /// [*Bidi_Class*](http://www.unicode.org/reports/tr44/#Bidi_Class) property,
    /// also known as the *bidirectional character type*.
    ///
    /// * <http://www.unicode.org/reports/tr9/#Bidirectional_Character_Types>
    /// * <http://www.unicode.org/reports/tr44/#Bidi_Class_Values>
    pub enum BidiClass {
        /// Any strong left-to-right character
        ///
        /// ***General Scope***
        ///
        /// LRM, most alphabetic, syllabic, Han ideographs,
        /// non-European or non-Arabic digits, ...
        LeftToRight {
            abbr => L,
            long => Left_To_Right,
            display => "Left-to-Right",
        }

        /// Any strong right-to-left (non-Arabic-type) character
        ///
        /// ***General Scope***
        ///
        /// RLM, Hebrew alphabet, and related punctuation
        RightToLeft {
            abbr => R,
            long => Right_To_Left,
            display => "Right-to-Left",
        }

        /// Any strong right-to-left (Arabic-type) character
        ///
        /// ***General Scope***
        ///
        /// ALM, Arabic, Thaana, and Syriac alphabets,
        /// most punctuation specific to those scripts, ...
        ArabicLetter {
            abbr => AL,
            long => Arabic_Letter,
            display => "Right-to-Left Arabic",
        }
    }
}

/// Abbreviated name bindings for the `BidiClass` property
pub mod abbr_names for abbr;
/// Name bindings for the `BidiClass` property as they appear in Unicode documentation
pub mod long_names for long;

expands to:

/// Represents the Unicode character
/// [*Bidi_Class*](http://www.unicode.org/reports/tr44/#Bidi_Class) property,
/// also known as the *bidirectional character type*.
///
/// * <http://www.unicode.org/reports/tr9/#Bidirectional_Character_Types>
/// * <http://www.unicode.org/reports/tr44/#Bidi_Class_Values>
#[allow(bad_style)]
#[derive(Copy, Clone, Debug, Eq, PartialEq, Hash)]
pub enum BidiClass {
    /// Any strong left-to-right character
    LeftToRight,
    /// Any strong right-to-left (non-Arabic-type) character
    RightToLeft,
    /// Any strong right-to-left (Arabic-type) character
    ArabicLetter,
}
/// Abbreviated name bindings for the `BidiClass` property
#[allow(bad_style)]
pub mod abbr_names {
    pub use super::BidiClass::LeftToRight as L;
    pub use super::BidiClass::RightToLeft as R;
    pub use super::BidiClass::ArabicLetter as AL;
}
/// Name bindings for the `BidiClass` property as they appear in Unicode documentation
#[allow(bad_style)]
pub mod long_names {
    pub use super::BidiClass::LeftToRight as Left_To_Right;
    pub use super::BidiClass::RightToLeft as Right_To_Left;
    pub use super::BidiClass::ArabicLetter as Arabic_Letter;
}
#[allow(bad_style)]
#[allow(unreachable_patterns)]
impl ::std::str::FromStr for BidiClass {
    type Err = ();
    fn from_str(s: &str) -> Result<Self, Self::Err> {
        match s {
            "LeftToRight" => Ok(BidiClass::LeftToRight),
            "RightToLeft" => Ok(BidiClass::RightToLeft),
            "ArabicLetter" => Ok(BidiClass::ArabicLetter),
            "L" => Ok(BidiClass::LeftToRight),
            "R" => Ok(BidiClass::RightToLeft),
            "AL" => Ok(BidiClass::ArabicLetter),
            "Left_To_Right" => Ok(BidiClass::LeftToRight),
            "Right_To_Left" => Ok(BidiClass::RightToLeft),
            "Arabic_Letter" => Ok(BidiClass::ArabicLetter),
            _ => Err(()),
        }
    }
}
#[allow(bad_style)]
#[allow(unreachable_patterns)]
impl ::std::fmt::Display for BidiClass {
    fn fmt(&self, f: &mut ::std::fmt::Formatter) -> ::std::fmt::Result {
        match *self {
            BidiClass::LeftToRight => write!(f, "{}", "Left-to-Right"),
            BidiClass::RightToLeft => write!(f, "{}", "Right-to-Left"),
            BidiClass::ArabicLetter => write!(f, "{}", "Right-to-Left Arabic"),
            BidiClass::LeftToRight => write!(f, "{}", "Left_To_Right".replace('_', " ")),
            BidiClass::RightToLeft => write!(f, "{}", "Right_To_Left".replace('_', " ")),
            BidiClass::ArabicLetter => write!(f, "{}", "Arabic_Letter".replace('_', " ")),
            _ => {
                write!(
                    f,
                    "{}",
                    match *self {
                        BidiClass::LeftToRight => "L",
                        BidiClass::RightToLeft => "R",
                        BidiClass::ArabicLetter => "AL",
                        BidiClass::LeftToRight => "LeftToRight",
                        BidiClass::RightToLeft => "RightToLeft",
                        BidiClass::ArabicLetter => "ArabicLetter",
                    }
                )
            }
        }
    }
}
#[allow(bad_style)]
impl ::char_property::EnumeratedCharProperty for BidiClass {
    fn abbr_name(&self) -> &'static str {
        match *self {
            BidiClass::LeftToRight => "L",
            BidiClass::RightToLeft => "R",
            BidiClass::ArabicLetter => "AL",
        }
    }
    fn all_values() -> &'static [BidiClass] {
        const VALUES: &[BidiClass] = &[
            BidiClass::LeftToRight,
            BidiClass::RightToLeft,
            BidiClass::ArabicLetter,
        ];
        VALUES
    }
}

All three of the abbr, long, and display properties of the enum are optional, and have sane fallbacks: abbr_name and long_name return None if unspecified, and fmt::Display will check, in order, for display, long_name, abbr_name, and the variant name until it finds one to use (stringified, of course).

FromStr is defined, matching against any of the provided abbr, long, and variant name.

Important notes:

• The current format uses associated consts, so it works on beta but won't work on stable until 1.20 is stable.
  • Consts have a slightly different meaning than pub use -- pub use aliases the type where const is a new object and if used in pattern matching is a == call and not a pattern match.
  • For this reason I'm actually slightly leaning towards using pub use even once associated consts land; they're compartmentalized (so use Property::* doesn't pull in 3x as many symbols as there are variants). After using the const based aliasing for a little bit, I'm inclined to like the current solution of unic::ucd::bidi::BidiClass::* + unic::ucd::bidi::bidi_class::abbr_names::*. These really should be a pub use and not a const.
  • Note that I still think const are the way to go for cases like Canonical_Combining_Class, though.
• The current syntax could easily be adapted to use modules instead of associated consts, but was written with the associated consts so we could get a feel of how it would look with them.
• The zero-or-more meta match before a enum variant conflicts with the ident match before 1.20. See rust-lang/rust#42913, rust-lang/rust#24189
• There only tests of the macro are rather thin and could be expanded.
• It's a macro, so the response when you stick stuff not matching the expected pattern is cryptic at best.
• The CharProperty trait is pretty much the lowest common denominator. It's a starting point, and we can iterate from there.
• How and where do we want to make CharProperty a externally visible trait? Currently having it in namespace is the only way to access abbr_name and long_name.
• Earlier discussion suggested putting these into unic::utils::char_property. Moving it would be simple, but for now it's living in the root of unic-utils
• The crate unic-utils is currently in the workspace by virtue of being a dependency of unic, but is not in any way visible a crate depending on unic.
• Documentation doesn't exist.

I can submit a separate PR after 31e6582 if we want to pull the edited macro and the application separately.

The diff is kind of opaque because this touches so much. You're probably better off just comparing before/after the PR without looking at the diff output.

This PR is submitted in a ready-to-merge (after review) state rather than my more usual track-final-polish.

This will require a rebase of #118 as it touches many of the same parts of the code.

The first half of adressing #91. Closes #111, this manner of attack is better than it.

This PR only has one type, CharRange. It is effectively std::ops::RangeInclusive translated to characters. The matter of construction is handled by both half-open and closed constructors offered and a macro to allow for 'a'..='z' syntax.

It's a WIP. One problem with it right now is that it increases the compile time by dozens of seconds, because of the optimization being done with the &[&'static str] table. Because of that, we may be better off with a manually implemented mapping, meaning that table generation would take some time, but component compilation stays fast.

We have a cross-component test in components/ucd/tests/category_tests.rs that checks values of the Bidi_Class property against General_Category property, based on UAX#9's Table 4. Bidirectional Character Types.

Now that we have component/ucd/category, we can expand the test to also cover General_Category values.

Hello there! I was trying to create a package for Fedora when I realized that the hangul crate does not contain any license files. Both the MIT and Apache-2.0 license require that redistributed sources contain a copy of the license text.

Upon calling unic::emoji::char::is_emoji('0'), this library returns true. I'm not aware of the specifics of the unicode standard, but I believe that '0' is not an emoji.

This test may be useful to introduce:

#[test]
fn are_nums_emojis() {
    use unic::emoji::char::is_emoji;
    assert_eq!(is_emoji('0'), false);
    assert_eq!(is_emoji('1'), false);
    assert_eq!(is_emoji('2'), false);
    assert_eq!(is_emoji('3'), false);
    assert_eq!(is_emoji('4'), false);
    assert_eq!(is_emoji('5'), false);
    assert_eq!(is_emoji('6'), false);
    assert_eq!(is_emoji('7'), false);
    assert_eq!(is_emoji('8'), false);
    assert_eq!(is_emoji('9'), false);
}

I'm working on a font editor, MFEK. I also contribute to Unicode when I can. One of my fonts requires characters in Unicode 14.0.

For those reasons, I had to fork the project. I only need blocks, categories, and names, so I called my version QD-UNIC—“quick and dirty UNIC”. https://github.com/MFEK/qd-unic.rlib

I think that, perhaps, this project was too ambitious, in the sense that all the modules must match each other in Unicode version. That's what's caused a single PR, #226, to stall development of everything because of issues with unic-ucd-segment.

Obviously some of these modules are very easy to keep updated, and unic-gen works phenomenally well. Those implementing things like text segmentation and BIDI are going to be more difficult, and certainly subject to the needs of the community…which more often match mine than not. Basically, in short, users who only care about getting character names shouldn't suffer because no one has yet contributed a fix to a text segmentation problem.

Anyway, I doubt y'all will agree, which is why I forked, but I thought I'd let you know why I forked.

This fixes a bug where indices within a line are incorrectly used to index within the paragraph. Closes #272.

This is a port of servo/unicode-bidi#55 by @laurmaedje. Please see that PR for more details.

This change is 

I'm looking at using unic-bidi for a contemplated text layout project. For this discussion, consider the primary goal to be adding BiDi support to a (hypothetical) in-Rust implementation of text layout conforming to Piet's TextLayout trait. I see a number of concerns.

As a higher level process issue, one way to address those concerns is to fork unic-bidi. But I see value in trying to maintain a single source of truth for BiDi in the Rust ecosystem, so it feels like working with upstream is a good way to do that. That said, #272 is pretty good evidence to me, in addition to reverse dependencies (note that the RustPython reverse dep has actually recently been removed) that nobody is actually using unic-bidi, especially within the context of paragraph level text formatting.

Now for my concerns.

A big one is the &'text lifetime requirement on BidiInfo. That precludes retaining BidiInfo as part of our text layout object. One pattern for fixing this is to change the type parameter from <'text> to AsRef<str>, which would let the text be borrowed or owned, at the caller's request. Another way to fix it is to simply clone the string (it's not as if the implementation is particularly clone-averse). But I'm not convinced we actually need to retain the string. If you look at the methods on BidiInfo, I'm not sure reorder_line() is actually still valid in a modern environment; if my understanding is correct, HarfBuzz always wants its input in logical order. The string is also used in reorder_levels_per_char() to find the codepoint boundaries (I'm not sure this function is useful), and in visual_runs(), again mostly to find codepoint boundaries, but I think the "reset" logic can be run entirely from original_classes without making reference to the text.

Other concerns center around performance. I'm not advocating doing a lot of performance work now, but I also believe that the current API locks in certain decisions that make such performance work difficult in the future. That's mostly around the per-byte (technically, per UTF-8 code unit) arrays to represent things like levels. I think these don't represent the way people typically work with text in Rust, and in any case there are about twice as many UTF-8 code units as UTF-16 for most BiDi work.

So what I'd like better is for visual_runs to return an array of runs, where each run is a range in the text and a level (the latter would be after the reset logic is applied). I think the various "reorder*" functions should go, as I don't see them as being particularly useful, and if they are needed, I think their function can be written quite reasonably in terms of this proposed visual_runs.

Lastly, the stuff around ParagraphInfo seems crufty to me. Why is this passed in? It's possible to infer from the text position. Taking a step back, it's not clear to me that segmenting into paragraphs is a useful function for unic-bidi in the first place. It seems extremely likely that its client will have already done segmentation into paragraphs by the time it gets to BiDi analysis. If the goal of the crate is to cover all of UAX#9, then perhaps paragraph segmentation can be provided through a separate API. Then the scope of BidiInfo could be narrowed to a single paragraph, which I think would be a desirable simplification.

I post this in part to provoke discussion from potential other clients of the crate, and also to test whether unic is a good container for the work.