We should try to make fluent-bundle release size as small as possible.

@cmyr pointed out in https://github.com/zbraniecki/unic-locale/issues/10 that at the moment the size increase is ~300kb.

I'd like to look for ways to make it smaller.

According to the analysis the bulk of the size increase goes into rental and syn (used for langid macro).

I think that first low hanging fruit would be to stop using the macro in fluent-bundle, and investigate the rental.

The fluent API is currently very allocation-intensive. It could be possible to reduce allocations by e.g. changing the arguments of format from HashMap<&'a str, FluentValue> to something like &'a [&'a str, FluentValue<'a>], and letting FluentValue store references instead of owned strings.

This updates @Michael-F-Bryan's work from #184 to work with the current master branch. It started out as more of a quick and ugly fix, so please let me know if there's anything you want me to smooth out.

The current management of FluentResource and FluentBundle is likely optimally fast and memory safe, but hard to deal with in reality.

While working on an experiment to bring FluentBundle and FluentResource to Gecko via WebIDL, I once again ended up with an expectation to use Rc.

Sooo, how about if we switch to Rc? Suddenly, we don't need a fancy resource manager to store FluentResource objects, and we can freely feed resources to bundle. And if we need two bundles to use the same resource, we just feed them to Rcs!

Question 1: Is Rc limiting us in anything major? Since we're targeting fluent-rs to be a general use crate for localization, is there any common use case which would make fluent-rs unusable if we switched to Rc?

The challenge I encountered is that I'm not sure how can I hash references to particular ast::Message for quick retrieval in the Rc scenario.

In the current code I have (simplified):

struct FluentBundle<'bundle> {
    messages: HashMap<String, &'bundle ast::Message>,
}

impl FluentBundle {
    fn add_resource(&mut self, res: &FluentResource) {
        for msg in res.body {
            self.messages.insert(msg.id.name.to_string(), msg);
        }  
    }
}

now, in the Rc version I'd have sth like:

struct FluentBundle {
    resources: Vec<Rc<FluentResource>>,
    messages: HashMap<String, &'? ast::Message>,
}

impl FluentBundle {
    fn add_resource(&mut self, res: Rc<FluentResource>) {
        self.resources.push(res);
        ?? // how to hash references to ast::Message ?
    }
}

@Manishearth - can you advise on those two issues?

AST in FluentSyntax currently uses &str which means that its basically read-only. As we mature the fluent-syntax crate we'll likely want to add a serializer and ability for users to operate on the AST.

There are two ways we could approach it:

1. We could add OwnedMessage which would be added to the Resource::body like any other node and a way to copy Message to a new OwnedMessage. This would be quite limiting, but likely sufficient for most use cases.

2. We could switch the AST to use Cow<&str> which means it'll still handle &str for all basic parsing/runtime work, but allow it to be replaced with owned String when modified/added.

(2) sounds tempting, but I'm not sure what are performance/memory consequences.

Jared Wein@LAPTOP-BRR0AH2E ~/Documents/GitHub/fluent-rs/fluent-syntax $ cargo build Updating crates.io index Compiling fluent-syntax v0.1.1 (C:\Users\Jared Wein\Documents\GitHub\fluent-rs\fluent-syntax) Finished dev [unoptimized + debuginfo] target(s) in 17.25s

Jared Wein@LAPTOP-BRR0AH2E ~/Documents/GitHub/fluent-rs/fluent-syntax $ cargo test Downloaded glob v0.2.11 Downloaded criterion v0.2.9 Downloaded serde_json v1.0.37 Downloaded assert-json-diff v0.2.1 Downloaded rand_xoshiro v0.1.0 Downloaded atty v0.2.11 Downloaded csv v1.0.5 Downloaded itertools v0.8.0 Downloaded rayon v1.0.3 Downloaded serde_derive v1.0.85 Downloaded serde v1.0.85 Downloaded rand_core v0.3.1 Downloaded walkdir v2.2.7 Downloaded cast v0.2.2 Downloaded criterion-plot v0.3.0 Downloaded clap v2.32.0 Downloaded lazy_static v1.2.0 Downloaded tinytemplate v1.0.1 Downloaded byteorder v1.3.1 Downloaded rand_core v0.4.0 Downloaded rayon-core v1.4.1 Downloaded rand_os v0.1.1 Downloaded bitflags v1.0.4 Downloaded winapi-util v0.1.1 Downloaded unicode-width v0.1.5 Downloaded quote v0.6.11 Downloaded either v1.5.0 Downloaded num-traits v0.2.6 Downloaded winapi v0.3.6 Downloaded textwrap v0.10.0 Downloaded same-file v1.0.4 Downloaded itoa v0.4.3 Downloaded crossbeam-deque v0.2.0 Downloaded proc-macro2 v0.4.26 Downloaded ryu v0.2.7 Downloaded csv-core v0.1.5 Downloaded libc v0.2.48 Downloaded crossbeam-utils v0.2.2 Downloaded crossbeam-epoch v0.3.1 Downloaded unicode-xid v0.1.0 Downloaded memchr v2.1.3 Downloaded cfg-if v0.1.6 Downloaded scopeguard v0.3.3 Downloaded memoffset v0.2.1 Downloaded nodrop v0.1.13 Downloaded arrayvec v0.4.10 Downloaded syn v0.15.26 Downloaded num_cpus v1.9.0 Compiling arrayvec v0.4.10 Compiling winapi v0.3.6 Compiling nodrop v0.1.13 Compiling libc v0.2.48 Compiling proc-macro2 v0.4.26 Compiling cfg-if v0.1.6 Compiling memoffset v0.2.1 Compiling lazy_static v1.2.0 Compiling memchr v2.1.3 Compiling ryu v0.2.7 Compiling unicode-xid v0.1.0 Compiling serde v1.0.85 Compiling scopeguard v0.3.3 Compiling rayon-core v1.4.1 Compiling byteorder v1.3.1 Compiling unicode-width v0.1.5 Compiling num-traits v0.2.6 Compiling rayon v1.0.3 Compiling itoa v0.4.3 Compiling either v1.5.0 Compiling rand_core v0.4.0 Compiling cast v0.2.2 Compiling bitflags v1.0.4 Compiling glob v0.2.11 Compiling crossbeam-utils v0.2.2 Compiling textwrap v0.10.0 Compiling itertools v0.8.0 Compiling rand_core v0.3.1 Compiling num_cpus v1.9.0 Compiling clap v2.32.0 Compiling crossbeam-epoch v0.3.1 Compiling quote v0.6.11 Compiling rand_xoshiro v0.1.0 Compiling criterion-plot v0.3.0 Compiling csv-core v0.1.5 Compiling syn v0.15.26 Compiling crossbeam-deque v0.2.0 Compiling winapi-util v0.1.1 Compiling atty v0.2.11 Compiling rand_os v0.1.1 Compiling same-file v1.0.4 Compiling serde_derive v1.0.85 Compiling walkdir v2.2.7 Compiling serde_json v1.0.37 Compiling csv v1.0.5 Compiling tinytemplate v1.0.1 Compiling assert-json-diff v0.2.1 Compiling criterion v0.2.9 Compiling fluent-syntax v0.1.1 (C:\Users\Jared Wein\Documents\GitHub\fluent-rs\fluent-syntax) Finished dev [unoptimized + debuginfo] target(s) in 46.62s Running c:\Users\Jared Wein\Documents\GitHub\fluent-rs\target\debug\deps\fluent_syntax-50e75f5cdb1124cd.exe

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

 Running c:\Users\Jared Wein\Documents\GitHub\fluent-rs\target\debug\deps\parser_fixtures-cf1c70fdc28839b7.exe

running 2 tests test parse_fixtures_compare ... FAILED test parse_fixtures ... ok

failures:

---- parse_fixtures_compare stdout ---- Parsing: "tests\fixtures\any_char.ftl" Parsing: "tests\fixtures\astral.ftl" thread 'parse_fixtures_compare' panicked at '

json atoms at path ".body[8].content" are not equal: expected: "err-😂 = Value\n\n" actual: "err-😂 = Value\r\n\r\n"

json atoms at path ".body[10].content" are not equal: expected: "err-invalid-expression = { 😂 }\n\n" actual: "err-invalid-expression = { 😂 }\r\n\r\n"

json atoms at path ".body[12].content" are not equal: expected: "err-invalid-variant-key = { $sel ->\n *[😂] Value\n}\n" actual: "err-invalid-variant-key = { $sel ->\r\n *[😂] Value\r\n}\r\n"

', fluent-syntax\tests\parser_fixtures.rs:17:5 note: Run with RUST_BACKTRACE=1 for a backtrace.

failures: parse_fixtures_compare

test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out

error: test failed, to rerun pass '--test parser_fixtures'

Jared Wein@LAPTOP-BRR0AH2E ~/Documents/GitHub/fluent-rs/fluent-syntax $

This is a major rewrite, to the point where I'm not even sure how to exactly plan reviewing it and landing, but I think it is getting close to being done.

There's one open question about FluentBundle::add_messages which I posted in https://github.com/projectfluent/fluent.js/issues/328

This, unfortunately, break a lot of examples/tests in fluent-bundle as before they freely passed &str and constructed FluentResource on fly. We'll need to answer that question, and then clean up the code, make sure that such zero-copy abstraction works well for Amethyst, and begin reviewing it to land.

I believe that the proposed structure and memory allocation model should bring us close to the model we'll use for fluent-rs 1.0.

Currently FluentArgs is a HashMap over &str (and I'm going to change it to over String).

It would be nice to have it over Cow maybe and write some helper/macro to construct args.

I'm having trouble updating to fluent 0.13.x, and it seems part of the issue is a circle of dependencies that pulls in older versions. In particular fluent-fallback seems to depend on fluent-bundle 0.12, dragging both 0.12 and 0.13 into my app and fouling up the works.

   --> src/i18n.rs:134:19
    |
134 |         let loc = Localization::new(
    |                   ^^^^^^^^^^^^^^^^^ expected struct `fluent::bundle::FluentBundleBase`, found struct `fluent_bundle::bundle::FluentBundleBase`
    |
   ::: /home/caleb/.cargo/registry/src/github.com-1ecc6299db9ec823/fluent-fallback-0.0.4/src/lib.rs:28:17
    |
28  |     I: Iterator<Item = FluentBundle<R>> + 'loc,
    |                 ---------------------- required by this bound in `fluent_fallback::Localization`
    |
    = note: expected enum `std::option::Option<fluent::bundle::FluentBundleBase<fluent::FluentResource, intl_memoizer::IntlLangMemoizer>>`
               found enum `std::option::Option<fluent_bundle::bundle::FluentBundleBase<_, intl_memoizer::IntlLangMemoizer>>`
    = note: perhaps two different versions of crate `fluent_bundle` are being used?
    = note: required because of the requirements on the impl of `std::iter::Iterator` for `std::iter::FromFn<[closure@src/i18n.rs:111:27: 130:14 resolved_locales:_, res_path_scheme:_, res_ids:_]>`

I looked at the current sources in this repo and they look okay, I'm guessing the crate just needs a new release to reflect the current state of the sources.

I encountered this issue while writing tests for my Java bindings. I noticed that the return value of format_pattern adds additional characters before and after each argument. At first I thought this may be caused by the conversions between Java and Rust strings, but upon further investigation, this also happens in the simplest and purest possible Rust scenario.

This is the entire code:

use fluent::concurrent::FluentBundle;
use fluent::{FluentResource, FluentValue, FluentArgs};
use unic_langid::LanguageIdentifier;

fn main() {
    let resource = FluentResource::try_new("key = { $one } and { $two }!".to_owned()).unwrap();
    let id: LanguageIdentifier = "en-US".parse().unwrap();
    let mut bundle = FluentBundle::new(&[id]);
    bundle.add_resource(resource).unwrap();
    let msg = bundle.get_message("key").unwrap();
    let mut errors = vec![];
    let mut args = FluentArgs::new();
    args.insert("one", FluentValue::from("test"));
    args.insert("two", FluentValue::from(3.141));
    let value = bundle.format_pattern(&msg.value.unwrap(), Some(&args), &mut errors);
    println!("{}", value);
    println!("{}", value.to_owned() == "test and 3.141!".to_owned());
}

Expected output:

test and 3.141!
true

Actual output:

⁨test⁩ and ⁨3.141⁩!
false

The values look the same, but there are some zero-width characters before and after "test" and 3.141 respectively. You can see for yourself by pasting them into some string comparison tool.

The characters change the output on devices that are able to display them, for example here in a bash terminal:

For this pure test, I used fluent version 0.11.0, but it also happens directly with fluent-bundle. I'm running on Windows 10 64 Bit.

I am currently working on Java bindings for this implementation of fluent and essentially mirroring the fluent-bundle API, as requested here.

I have encountered a rather fatal issue, though: many, if not most of the API structs do not implement Send, which effectively prevents any binding to those structs. The reason for that is that in order to move values between Java and Rust, the values need to be Send. There is no way around it that I am aware of.

The structs/types that currently lack a Send implementation and that I know are critical for JNI so far, are the following:

• FluentValue (derives from the Custom variant that has a Box<dyn FluentType + 'static>)
• FluentArgs (I presume it derives from FluentValue)
• FluentBundle (though there is a concurrent version of it, as far as I'm aware? Maybe I could use that, still have to look into it)

So, it would make my life much easier if you added an unsafe Send implementation where required. If you can't do this because people might get the idea that it's thread safe then, I'm all open for other ideas or solutions.

As per the comment in #299.

I've created this alias in the concurrent module. That seemed like the best place to put it imo, but in order to do that I had to make the module public. I hope that's not a problem.

Cheers.

This is a refactor with the goal to have only 1 implementation of Read/Resolve for ast expressions. In #305 I had to duplicate some Write implementations to support Resolve as well. That's not necessary anymore after this PR.

The last commit reduces the number of lifetimes on the Scope struct so that the resolve code becomes a little nicer to work with. They didn't need to be separate as it doesn't affect public API.

I'm not sure what to name this, so suggestions to improve "WriteOrResolve" are welcome.

Shares the implementation of unescape_unicode and unescape_unicode_to_string. I've added a check to the corresponding test to make sure values that don't need unescaping are returned as Cow::Borrowed.

The most important change is in fluent-bundle/src/resolver/pattern.rs, the rest is adding missing codepaths that were previously only done through WriteValue and are now also needed in ResolveValue.

There are more cases where we can resolve to a borrowed str than just the TextElement case. For example there might be a select that resolves into a single TextElement, or a TermReference which is a simple string. With these changes, ResolveValue methods will be called until a pattern with more than 1 element is found, which needs string concatenation and is handled by the WriteValue methods as before.

When resolving into a FluentValue::Error, the WriteValue is called to get "{error}" formatting instead of an empty string.

The main value add for a custom argument resolver is not needing to put arguments into a FluentArgs, which has no type safety. See the typesafe_messages example that demonstrates this use case. The first commit optimizes FluentArg by not storing keys as Cows, but as regular &str instead. This reduces a small overhead in data size and arg lookup performance.

Adds basic decimal formatting for FluentNumber.as_string(). Many features are still missing like currency symbols and percentage formatting. #269

Since data providers need to be managed explicitly, it needs to be handed to the memoizer to be able to construct a formatter. This required a new argument + associated type on Memoizable.

Something to figure out is where the data comes from. I've went with a simple FluentBundle.set_number_format_provider, which seems most flexible to me. I've added a test that uses icu_testdata for this, but other providers should work as well. https://icu4x.unicode.org/doc/icu/#data-management