Snips NLU rust implementation

Snips

Last update: Dec 26, 2022

Related tags

Overview

Snips NLU Rust

https://travis-ci.org/snipsco/snips-nlu-rs.svg?branch=master

https://ci.appveyor.com/api/projects/status/rsf27a9txeomic8o/branch/master?svg=true

Installation

Add it to your Cargo.toml:

[dependencies]
snips-nlu-lib = { git = "https://github.com/snipsco/snips-nlu-rs", branch = "master" }

Add extern crate snips_nlu_lib to your crate root and you are good to go!

Intent Parsing with Snips NLU

The purpose of the main crate of this repository, snips-nlu-lib, is to perform an information extraction task called intent parsing.

Let’s take an example to illustrate the main purpose of this lib, and consider the following sentence:

"What will be the weather in paris at 9pm?"

Properly trained, the Snips NLU engine will be able to extract structured data such as:

{
   "intent": {
      "intentName": "searchWeatherForecast",
      "confidenceScore": 0.95
   },
   "slots": [
      {
         "value": "paris",
         "entity": "locality",
         "slotName": "forecast_locality"
      },
      {
         "value": {
            "kind": "InstantTime",
            "value": "2018-02-08 20:00:00 +00:00"
         },
         "entity": "snips/datetime",
         "slotName": "forecast_start_datetime"
      }
   ]
}

In order to achieve such a result, the NLU engine needs to be fed with a trained model (json file). This repository only contains the inference part, in order to produce trained models please check the Snips NLU python library.

Example and API Usage

The interactive parsing CLI is a good example of to how to use snips-nlu-rs.

Here is how you can run the CLI example:

$ git clone https://github.com/snipsco/snips-nlu-rs
$ cd snips-nlu-rs
$ cargo run --example interactive_parsing_cli data/tests/models/nlu_engine

Here we used a sample trained engine, which consists in two intents: MakeCoffee and MakeTea. Thus, it will be able to parse queries like "Make me two cups of coffee please" or "I'd like a hot tea".

As mentioned in the previous section, you can train your own nlu engine with the Snips NLU python library.

License

Licensed under either of

Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
MIT license (LICENSE-MIT) or http://opensource.org/licenses/MIT)

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Comments

undefined reference to `crfsuite_instance_init_n'

I am trying to compile on an AWS EC2 Instance, and the code fails to compile with the following error:

/home/ec2-user/snips-nlu-rs/snips-nlu-lib/target/debug/deps/libsnips_nlu_lib-81442700a8c7b248.rlib(snips_nlu_lib-81442700a8c7b248.g4ocvkiolag6no4.rcgu.o): In function crfsuite::Tagger::set::h2f0968d9babb3c13': /home/ec2-user/.cargo/git/checkouts/crfsuite-rs-88ab7cb8bdb2b2f2/b18d95c/src/lib.rs:145: undefined reference tocrfsuite_instance_init_n' /home/ec2-user/snips-nlu-rs/snips-nlu-lib/target/debug/deps/libsnips_nlu_lib-81442700a8c7b248.rlib(snips_nlu_lib-81442700a8c7b248.g4ocvkiolag6no4.rcgu.o): In function crfsuite::Tagger::set::h2f0968d9babb3c13': g4ocvkiolag6no4-c308c9baf1e779a490d159873856ad7a.rs:(.text._ZN8crfsuite6Tagger3set17h2f0968d9babb3c13E+0x320): undefined reference tocrfsuite_item_init' g4ocvkiolag6no4-c308c9baf1e779a490d159873856ad7a.rs:(.text._ZN8crfsuite6Tagger3set17h2f0968d9babb3c13E+0x647): undefined reference to crfsuite_attribute_set' g4ocvkiolag6no4-c308c9baf1e779a490d159873856ad7a.rs:(.text._ZN8crfsuite6Tagger3set17h2f0968d9babb3c13E+0x659): undefined reference tocrfsuite_item_append_attribute' g4ocvkiolag6no4-c308c9baf1e779a490d159873856ad7a.rs:(.text._ZN8crfsuite6Tagger3set17h2f0968d9babb3c13E+0x6c0): undefined reference to crfsuite_instance_finish' g4ocvkiolag6no4-c308c9baf1e779a490d159873856ad7a.rs:(.text._ZN8crfsuite6Tagger3set17h2f0968d9babb3c13E+0x6ce): undefined reference tocrfsuite_instance_finish' /home/ec2-user/snips-nlu-rs/snips-nlu-lib/target/debug/deps/libcrfsuite-ad8b4b3fb39b3a5b.rlib(crfsuite-ad8b4b3fb39b3a5b.crfsuite0.rcgu.o): In function crfsuite::Tagger::create_from_memory::hf43200afd689451e': /home/ec2-user/.cargo/git/checkouts/crfsuite-rs-88ab7cb8bdb2b2f2/b18d95c/src/lib.rs:76: undefined reference tocrfsuite_create_instance_from_memory' collect2: error: ld returned 1 exit status

error: aborting due to previous error

error: Could not compile snips-nlu-lib.

opened by walpoletim-bjss 10

Adding snips-nlu-lib causes errror

Hey guys, I would like to compile an android lib with rust, jni and your snips-nlu-lib. Without the snips-nlu-lib my android lib compiles as expected. My cargo config looks like:

[target.armv7-linux-androideabi]
ar = "/Users/marcelkaufmann/Library/Android/sdk/ndk-bundle/toolchains/llvm/prebuilt/darwin-x86_64/bin/arm-linux-androideabi-ar"
linker = "/Users/marcelkaufmann/Library/Android/sdk/ndk-bundle/toolchains/llvm/prebuilt/darwin-x86_64/bin/armv7a-linux-androideabi21-clang++"

I am not using the standalone android toolchain but the standard ndk as mentioned in https://developer.android.com/ndk/guides/other_build_systems because of my ndk version r19. The lib as well as the JNI connection works:

Finished release [optimized] target(s) in 0.14s

After adding

snips-nlu-lib = { git = "https://github.com/snipsco/snips-nlu-rs", branch = "master" }

I find

Compiling backtrace-sys v0.1.28
   Compiling bzip2-sys v0.1.7
   Compiling clap v2.32.0
   Compiling rmp v0.8.7
   Compiling clang-sys v0.23.0
   Compiling enum_primitive v0.1.1
   Compiling chrono v0.4.6
error: failed to run custom build command for `backtrace-sys v0.1.28`
process didn't exit successfully: `/Users/marcelkaufmann/Projekte/MeineProjekte/rust-nlp/cargo/target/release/build/backtrace-sys-5917014bf132b9c5/build-script-build` (exit code: 101)
--- stdout
cargo:rustc-cfg=rbt
TARGET = Some("armv7-linux-androideabi")
OPT_LEVEL = Some("3")
HOST = Some("x86_64-apple-darwin")
CC_armv7-linux-androideabi = None
CC_armv7_linux_androideabi = None
TARGET_CC = None
CC = None
CFLAGS_armv7-linux-androideabi = None
CFLAGS_armv7_linux_androideabi = None
TARGET_CFLAGS = None
CFLAGS = None
running: "arm-linux-androideabi-clang" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "--target=armv7-linux-androideabi" "-I" "src/libbacktrace" "-I" "/Users/marcelkaufmann/Projekte/MeineProjekte/rust-nlp/cargo/target/armv7-linux-androideabi/release/build/backtrace-sys-27dbfe46cbd61a97/out" "-fvisibility=hidden" "-DBACKTRACE_ELF_SIZE=32" "-DBACKTRACE_SUPPORTED=1" "-DBACKTRACE_USES_MALLOC=1" "-DBACKTRACE_SUPPORTS_THREADS=0" "-DBACKTRACE_SUPPORTS_DATA=0" "-D_GNU_SOURCE=1" "-D_LARGE_FILES=1" "-Dbacktrace_full=__rbt_backtrace_full" "-Dbacktrace_dwarf_add=__rbt_backtrace_dwarf_add" "-Dbacktrace_initialize=__rbt_backtrace_initialize" "-Dbacktrace_pcinfo=__rbt_backtrace_pcinfo" "-Dbacktrace_syminfo=__rbt_backtrace_syminfo" "-Dbacktrace_get_view=__rbt_backtrace_get_view" "-Dbacktrace_release_view=__rbt_backtrace_release_view" "-Dbacktrace_alloc=__rbt_backtrace_alloc" "-Dbacktrace_free=__rbt_backtrace_free" "-Dbacktrace_vector_finish=__rbt_backtrace_vector_finish" "-Dbacktrace_vector_grow=__rbt_backtrace_vector_grow" "-Dbacktrace_vector_release=__rbt_backtrace_vector_release" "-Dbacktrace_close=__rbt_backtrace_close" "-Dbacktrace_open=__rbt_backtrace_open" "-Dbacktrace_print=__rbt_backtrace_print" "-Dbacktrace_simple=__rbt_backtrace_simple" "-Dbacktrace_qsort=__rbt_backtrace_qsort" "-Dbacktrace_create_state=__rbt_backtrace_create_state" "-Dbacktrace_uncompress_zdebug=__rbt_backtrace_uncompress_zdebug" "-o" "/Users/marcelkaufmann/Projekte/MeineProjekte/rust-nlp/cargo/target/armv7-linux-androideabi/release/build/backtrace-sys-27dbfe46cbd61a97/out/src/libbacktrace/alloc.o" "-c" "src/libbacktrace/alloc.c"

--- stderr
thread 'main' panicked at '

Internal error occurred: Failed to find tool. Is `arm-linux-androideabi-clang` installed?

', /Users/marcelkaufmann/.cargo/registry/src/github.com-1ecc6299db9ec823/cc-1.0.29/src/lib.rs:2314:5
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
   1: std::sys_common::backtrace::_print
   2: std::panicking::default_hook::{{closure}}
   3: std::panicking::default_hook
   4: std::panicking::rust_panic_with_hook
   5: std::panicking::continue_panic_fmt
   6: std::panicking::begin_panic_fmt
   7: cc::fail
   8: cc::Build::compile
   9: build_script_build::main
  10: std::rt::lang_start::{{closure}}
  11: std::panicking::try::do_call
  12: __rust_maybe_catch_panic
  13: std::rt::lang_start_internal
  14: main

warning: build failed, waiting for other jobs to finish...
error: failed to run custom build command for `bzip2-sys v0.1.7`
process didn't exit successfully: `/Users/marcelkaufmann/Projekte/MeineProjekte/rust-nlp/cargo/target/release/build/bzip2-sys-338438d94e50ff86/build-script-build` (exit code: 101)
--- stdout
TARGET = Some("armv7-linux-androideabi")
OPT_LEVEL = Some("3")
HOST = Some("x86_64-apple-darwin")
CC_armv7-linux-androideabi = None
CC_armv7_linux_androideabi = None
TARGET_CC = None
CC = None
CFLAGS_armv7-linux-androideabi = None
CFLAGS_armv7_linux_androideabi = None
TARGET_CFLAGS = None
CFLAGS = None
DEBUG = Some("false")
running: "arm-linux-androideabi-clang" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "--target=armv7-linux-androideabi" "-I" "bzip2-1.0.6" "-D_FILE_OFFSET_BITS=64" "-DBZ_NO_STDIO" "-o" "/Users/marcelkaufmann/Projekte/MeineProjekte/rust-nlp/cargo/target/armv7-linux-androideabi/release/build/bzip2-sys-4a4802aaf443cba8/out/lib/bzip2-1.0.6/blocksort.o" "-c" "bzip2-1.0.6/blocksort.c"

--- stderr
thread 'main' panicked at '

Internal error occurred: Failed to find tool. Is `arm-linux-androideabi-clang` installed?

', /Users/marcelkaufmann/.cargo/registry/src/github.com-1ecc6299db9ec823/cc-1.0.29/src/lib.rs:2314:5
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
   1: std::sys_common::backtrace::_print
   2: std::panicking::default_hook::{{closure}}
   3: std::panicking::default_hook
   4: std::panicking::rust_panic_with_hook
   5: std::panicking::continue_panic_fmt
   6: std::panicking::begin_panic_fmt
   7: cc::fail
   8: cc::Build::compile
   9: build_script_build::main
  10: std::rt::lang_start::{{closure}}
  11: std::panicking::try::do_call
  12: __rust_maybe_catch_panic
  13: std::rt::lang_start_internal
  14: main

warning: build failed, waiting for other jobs to finish...
error: build failed
------------------------------------------------------------

Do you have any suggestion what the problem is? Why does the compiler search for arm-linux-androideabi-clang and where? And why after adding your library? Thanks :)

opened by Flocksserver 7

error: Could not compile `snips-nlu-resources-packed`

I'm not terribly experienced with Rust, so I apologize if the information here is lacking.

I have a project with the following Cargo.toml

[dependencies]
snips-nlu-lib = { git = "https://github.com/talkative-ai/snips-nlu-rs", branch = "master" }
redis = "0.8.0"
hyper = "0.11"
futures = "0.1.14"

When I execute cargo run --verbose eventually the compilation fails:

error: Could not compile `snips-nlu-resources-packed`

The information on the error is as follows:

Caused by: process didn't exit successfully: `rustc --crate-name snips_nlu_resources_packed /usr/local/cargo/git/checkouts/snips-nlu-rs-c3854c7c83110319/85787f9/snips-nlu-resources-packed/src/lib.rs --crate-type lib --emit=dep-info,link -C debuginfo=2 -C metadata=cdca8d510a223171 -C extra-filename=-cdca8d510a223171 --out-dir /tmp/project/target/debug/deps -L dependency=/tmp/project/target/debug/deps --extern snips_nlu_ontology=/tmp/project/target/debug/deps/libsnips_nlu_ontology-64e186b4f4ebb163.rlib --extern phf=/tmp/project/target/debug/deps/libphf-f0c00edcacda1116.rlib --extern failure=/tmp/project/target/debug/deps/libfailure-4e47e324962fe3f2.rlib --cap-lints allow -L native=/tmp/project/target/debug/build/backtrace-sys-7833d5b313c46d32/out/.libs`

rustc and cargo info: rustc 1.23.0 (766bd11c8 2018-01-01) cargo 0.24.0 (45043115c 2017-12-05)

Any ideas why this is happening?

opened by warent 7

Many intents

Hello,

I'm looking for an NLU engine for a chatbot project. Firstly, thanks for opensourcing this rust lib !

One sentence seems to be linked to only one intent, is it a kind of limitation, isn't it ?

As an example : "Can you open the door and put light on ?". Here I've two intents : DoorOpening and PutLightsOn.

Any way to achieve this one ?

Maybe it's easy for Deterministic intent parser but not for probabilistic one, maybe returning all intents with probability and let the app pick one or many (as an example, a strategy may be to take all intents with probability over 90% ?)

Thanks, Gaël

opened by Nainterceptor 5
S3 Access Denied Error
Hi, on trying to build the Swift library in snips-nlu-ffi/swift the build script tries to download a file via curl -s https://s3.amazonaws.com/snips/snips-nlu-dev/snips-nlu-ios.0.58.0-SNAPSHOT.tgz. This then fails on trying to extract the archive.

When trying to manually debug this and download the archive, I get the following response.

<?xml version="1.0" encoding="UTF-8"?> <Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>B0F1298F8121AAEE</RequestId><HostId>g899i0XBMKQWq3iEp0nS9zyiTgpc6gv8fPMnPsnczAzAqbCbIUmYFTXOfrMOMoQQHFxlm3ORMSA=</HostId></Error>%
bug
opened by kiliankoe 4

Unable to run rust example on ubuntu

I am running cargo on Ubuntu 16.04 LTS. Running the command

cargo run --verbose --example weather examples "What will be the weather in London tomorrow at 8am?"

results in the following error:

note: /home/raghu/snips-nlu-rs/snips-nlu-lib/target/debug/deps/libsnips_nlu_lib-db0d3b8725811e50.rlib(snips_nlu_lib-db0d3b8725811e50.g4ocvkiolag6no4.rcgu.o): In function `crfsuite::Tagger::set::hf3ed06d04cff7be0':
          /home/raghu/.cargo/git/checkouts/crfsuite-rs-88ab7cb8bdb2b2f2/b18d95c/src/lib.rs:145: undefined reference to `crfsuite_instance_init_n'
          /home/raghu/.cargo/git/checkouts/crfsuite-rs-88ab7cb8bdb2b2f2/b18d95c/src/lib.rs:156: undefined reference to `crfsuite_item_init'
          /home/raghu/.cargo/git/checkouts/crfsuite-rs-88ab7cb8bdb2b2f2/b18d95c/src/lib.rs:164: undefined reference to `crfsuite_attribute_set'
          /home/raghu/.cargo/git/checkouts/crfsuite-rs-88ab7cb8bdb2b2f2/b18d95c/src/lib.rs:165: undefined reference to `crfsuite_item_append_attribute'
          /home/raghu/.cargo/git/checkouts/crfsuite-rs-88ab7cb8bdb2b2f2/b18d95c/src/lib.rs:176: undefined reference to `crfsuite_instance_finish'
          /home/raghu/.cargo/git/checkouts/crfsuite-rs-88ab7cb8bdb2b2f2/b18d95c/src/lib.rs:180: undefined reference to `crfsuite_instance_finish'
          /home/raghu/snips-nlu-rs/snips-nlu-lib/target/debug/deps/libcrfsuite-7428f11a847b8a45.rlib(crfsuite-7428f11a847b8a45.crfsuite4.rcgu.o): In function `crfsuite::Tagger::create_from_memory::h1ceffb08ed3d9cb3':
          /home/raghu/.cargo/git/checkouts/crfsuite-rs-88ab7cb8bdb2b2f2/b18d95c/src/lib.rs:76: undefined reference to `crfsuite_create_instance_from_memory'
          collect2: error: ld returned 1 exit status

error: aborting due to previous error

error: Could not compile `snips-nlu-lib`.

I have also included the weather_example_run.log, which contains the complete output from the terminal window.

opened by kraghavk 4

Can one engine instance be used across mnany threads?

I want to use the native snips_nlu_ffi library binary in a multi-threaded app. Is it safe to share one engine pointer across many threads and call functions like snips_nlu_engine_run_get_intents concurrently?

opened by allisterb 3

build failures due to snips-nlu-ontology-parsers

I'm trying to run the example as per the readme (with an engine already built) and getting this failure:

> cargo run -p snips-nlu-lib --example interactive_parsing_cli ../snlu/engine
error: no matching package named `snips-nlu-ontology-parsers` found
location searched: https://github.com/snipsco/snips-nlu-ontology?branch=develop
required by package `snips-nlu-lib v0.63.0-SNAPSHOT (/Users/naitik/workspace/snips-nlu-rs/snips-nlu-lib)`

It seemed like this was the fix:

-snips-nlu-ontology-parsers = { git = "https://github.com/snipsco/snips-nlu-ontology", branch = "develop" }
+snips-nlu-parsers = { git = "https://github.com/snipsco/snips-nlu-parsers", branch = "develop" }

But still fails:

error: no matching package named `snips-nlu-ontology-parsers` found
location searched: https://github.com/snipsco/snips-nlu-ontology?branch=develop
required by package `snips-nlu-lib v0.63.0-SNAPSHOT (https://github.com/snipsco/snips-nlu-rs?branch=develop#2d1f58b6)`
    ... which is depended on by `snips-nlu-ffi v0.63.0-SNAPSHOT (https://github.com/snipsco/snips-nlu-rs?branch=develop#2d1f58b6)`
    ... which is depended on by `snips-nlu-python-ffi v0.63.0-SNAPSHOT (/Users/naitik/workspace/snips-nlu-rs/snips-nlu-ffi/python/snips-nlu-python-ffi)`

I think it's because the dependency is pulling in the upstream snips-nlu-rs without my local fix? Any help would be appreciated!

opened by daaku 2

trained_assistant.json file downloaded from the website (in zip) seems to break things

Hey!

I've been hacking around for a while now using the example in this repo to get up and running and had an issue earlier where it would break when I tried to parse a query. More specifically thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: InvalidLength', libcore/result.rs:945:5

I then ran a backtrace and tracked it down to lib/src/intent_parser/probabilistic_intent_parser.rs:50. Afterwards, I decided to try the trained_assistant.json file in the examples directory and everything worked fine which leads me to believe the trained_assistant.json file in the zip file, which I've downloaded from snips.ai, was faulty.

I also spotted these older version numbers in the zip file version which are different to the example versions. "training_package_version": "0.11.2", "model_version": "0.12.0"

See this gist which shows my code Not sure if that helps at all?

Like I mentioned, I'm up and running now using the example file, just thought I'd mention it here.

I already bombarded your Discord NLU channel too 🎉

opened by fjlaubscher 2
Add new APIs in ffi and bindings
Description This PR completes the existing ffi with the following missing APIs:

additional intents_whitelist and intents_blacklist parameters in the parse API

get_slots

get_intents

Additionally, the wrappers built on top of this FFI (python, kotlin, swift) are updated to integrate these changes.
opened by adrienball 1
[Build error] Could not find auto-linked library 'snips_nlu_ffi'
Basically the same as https://github.com/snipsco/snips-nlu-rs/issues/53, but when trying to build for macos it gives the following error: Could not find auto-linked library 'snips_nlu_ffi'. This is because the build script tries to copy libsnips_nlu_ffi.a in install_local_core:

cp: /Users/christian/workspace/soundreply/snips/snips-nlu-rs/snips-nlu-ffi/swift/SnipsNlu/../../..//target//debug/libsnips_nlu_ffi.a

which gives No such file or directory.
opened by hffmnn 1
Bump wheel from 0.30.0 to 0.38.1 in /platforms/python
Bumps wheel from 0.30.0 to 0.38.1.

Changelog

Sourced from wheel's changelog.

Release Notes

UNRELEASED

Updated vendored packaging to 22.0

0.38.4 (2022-11-09)

Fixed PKG-INFO conversion in bdist_wheel mangling UTF-8 header values in METADATA (PR by Anderson Bravalheri)

0.38.3 (2022-11-08)

Fixed install failure when used with --no-binary, reported on Ubuntu 20.04, by removing setup_requires from setup.cfg

0.38.2 (2022-11-05)

Fixed regression introduced in v0.38.1 which broke parsing of wheel file names with multiple platform tags

0.38.1 (2022-11-04)

Removed install dependency on setuptools

The future-proof fix in 0.36.0 for converting PyPy's SOABI into a abi tag was faulty. Fixed so that future changes in the SOABI will not change the tag.

0.38.0 (2022-10-21)

Dropped support for Python < 3.7

Updated vendored packaging to 21.3

Replaced all uses of distutils with setuptools

The handling of license_files (including glob patterns and default values) is now delegated to setuptools>=57.0.0 (#466). The package dependencies were updated to reflect this change.

Fixed potential DoS attack via the WHEEL_INFO_RE regular expression

Fixed ValueError: ZIP does not support timestamps before 1980 when using SOURCE_DATE_EPOCH=0 or when on-disk timestamps are earlier than 1980-01-01. Such timestamps are now changed to the minimum value before packaging.

0.37.1 (2021-12-22)

Fixed wheel pack duplicating the WHEEL contents when the build number has changed (#415)

Fixed parsing of file names containing commas in RECORD (PR by Hood Chatham)

0.37.0 (2021-08-09)

Added official Python 3.10 support

Updated vendored packaging library to v20.9

... (truncated)

Commits

6f1608d Created a new release

cf8f5ef Moved news item from PR #484 to its proper place

9ec2016 Removed install dependency on setuptools (#483)

747e1f6 Fixed PyPy SOABI parsing (#484)

7627548 [pre-commit.ci] pre-commit autoupdate (#480)

7b9e8e1 Test on Python 3.11 final

a04dfef Updated the pypi-publish action

94bb62c Fixed docs not building due to code style changes

d635664 Updated the codecov action to the latest version

fcb94cd Updated version to match the release

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies
opened by dependabot[bot] 0
0.65.0 build failing for android

0.65.0 build is failing for android when compiling the crfsuite-rs NDK Version = android-ndk-r20

running: "aarch64-linux-android-gcc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-I" "c/include" "-Wall" "-Wextra" "-o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/crfsuite.o" "-c" "c/crf/crfsuite.c" cargo:warning=c/crf/crfsuite.c:322:14: warning: unused function 'safe_strncpy' [-Wunused-function] cargo:warning=static char *safe_strncpy(char *dst, const char *src, size_t n) cargo:warning= ^ cargo:warning=1 warning generated. exit code: 0 AR_aarch64-linux-android = None AR_aarch64_linux_android = None TARGET_AR = None AR = None running: "aarch64-linux-android-ar" "crs" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/libcrfsuite.a" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/lbfgs/lbfgs.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/cqdb/lookup3.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/cqdb/cqdb.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/dictionary.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/logging.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/params.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/quark.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/rumavl.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/dataset.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/holdout.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/train_arow.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/train_averaged_perceptron.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/train_l2sgd.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/train_lbfgs.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/train_passive_aggressive.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/crf1d_context.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/crf1d_model.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/crf1d_feature.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/crf1d_encode.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/crf1d_tag.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/crfsuite_train.o" "/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out/c/crf/crfsuite.o" exit code: 0 cargo:rustc-link-lib=static=crfsuite cargo:rustc-link-search=native=/home/cahuja/snips/snips-nlu-rs-0.65.0/target/aarch64-linux-android/debug/build/crfsuite-sys-68380cbf5474de65/out

--- stderr clang80: error: no input files clang80: error: no input files clang80: error: no input files clang80: error: no input files clang80: error: no input files clang80: error: no input files clang80: error: no input files clang80: error: no input files clang80: error: no input files clang80: error: no input files clang80: error: no input files clang80: error: no input files clang80: error: no input files clang80: error: no input files clang80: error: no input files clang80: error: no input files clang80: error: no input files clang80: error: no input files clang80: error: no input files clang80: error: no input files clang80: error: no input files clang80: error: no input files clang80: error: no input files thread 'main' panicked at 'called Result::unwrap() on an Err value: Error(Msg("You must either define a TARGET_SYSROOT or use Dinghy to build your project."), State { next_error: None })', src/libcore/result.rs:997:5 stack backtrace: 0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:39 1: std::sys_common::backtrace::_print at src/libstd/sys_common/backtrace.rs:71 2: std::panicking::default_hook::{{closure}} at src/libstd/sys_common/backtrace.rs:59 at src/libstd/panicking.rs:197 3: std::panicking::default_hook at src/libstd/panicking.rs:211 4: std::panicking::rust_panic_with_hook at src/libstd/panicking.rs:474 5: std::panicking::continue_panic_fmt at src/libstd/panicking.rs:381 6: rust_begin_unwind at src/libstd/panicking.rs:308 7: core::panicking::panic_fmt at src/libcore/panicking.rs:85 8: core::result::unwrap_failed at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libcore/macros.rs:18 9: core::result::Result<T,E>::unwrap at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libcore/result.rs:798 10: build_script_build::main at ./build.rs:55 11: std::rt::lang_start::{{closure}} at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libstd/rt.rs:64 12: std::panicking::try::do_call at src/libstd/rt.rs:49 at src/libstd/panicking.rs:293 13: __rust_maybe_catch_panic at src/libpanic_unwind/lib.rs:87 14: std::rt::lang_start_internal at src/libstd/panicking.rs:272 at src/libstd/panic.rs:388 at src/libstd/rt.rs:48 15: std::rt::lang_start at /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libstd/rt.rs:64 16: main 17: __libc_start_main 18: _start

opened by cahuja1992 0

Failed building wheel for snips-nlu-rust

On raspbian stretch (RPi 3 B), trying to install snips-nlu-rust without success not familiar with rust, don't know what I'm doing wrong

sudo apt-get install python3-venv 
mkdir snips-nlu && cd snips-nlu
python3 -m venv .env 
source .env/bin/activate 
sudo apt install rustc cargo
pip install setuptools_rust==0.8.4
pip install wheel
pip install snips-nlu-rust

getting:

error: native library `clang` is being linked to by more than one version of the same package, but it can only be linked once; try updating or pinning your dependencies to ensure that this package only shows up once
  
    clang-sys v0.26.4
    clang-sys v0.22.0
  error: cargo failed with code: 101
    
  ----------------------------------------
  Failed building wheel for snips-nlu-rust

Complete output:

Installing collected packages: snips-nlu-rust
  Running setup.py install for snips-nlu-rust ... error
    Complete output from command /home/admin/snips-nlu/.env/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-w7nn52sk/snips-nlu-rust/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-mundmd59-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/admin/snips-nlu/.env/include/site/python3.5/snips-nlu-rust:
    /home/admin/snips-nlu/.env/lib/python3.5/site-packages/setuptools/dist.py:333: UserWarning: Normalizing '0.57.0
    ' to '0.57.0'
      normalized_version,
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-armv7l-3.5
    creating build/lib.linux-armv7l-3.5/snips_nlu_rust
    copying snips_nlu_rust/nlu_engine.py -> build/lib.linux-armv7l-3.5/snips_nlu_rust
    copying snips_nlu_rust/utils.py -> build/lib.linux-armv7l-3.5/snips_nlu_rust
    copying snips_nlu_rust/__init__.py -> build/lib.linux-armv7l-3.5/snips_nlu_rust
    running egg_info
    writing top-level names to snips_nlu_rust.egg-info/top_level.txt
    writing dependency_links to snips_nlu_rust.egg-info/dependency_links.txt
    writing snips_nlu_rust.egg-info/PKG-INFO
    writing requirements to snips_nlu_rust.egg-info/requires.txt
    reading manifest file 'snips_nlu_rust.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    warning: no previously-included files matching '*' found under directory 'snips-nlu-python-ffi/target/'
    warning: no previously-included files matching '__pycache__' found anywhere in distribution
    warning: no previously-included files matching '*.py[cod]' found anywhere in distribution
    writing manifest file 'snips_nlu_rust.egg-info/SOURCES.txt'
    copying snips_nlu_rust/__version__ -> build/lib.linux-armv7l-3.5/snips_nlu_rust
    creating build/lib.linux-armv7l-3.5/snips_nlu_rust/dylib
    copying snips_nlu_rust/dylib/.gitignore -> build/lib.linux-armv7l-3.5/snips_nlu_rust/dylib
    running build_ext
    running build_rust
    cargo rustc --lib --manifest-path /tmp/pip-build-w7nn52sk/snips-nlu-rust/snips-nlu-python-ffi/Cargo.toml --release -- --crate-type cdylib
    error: native library `clang` is being linked to by more than one version of the same package, but it can only be linked once; try updating or pinning your dependencies to ensure that this package only shows up once
    
      clang-sys v0.26.4
      clang-sys v0.22.0
    error: cargo failed with code: 101
    
    
    ----------------------------------------
Command "/home/admin/snips-nlu/.env/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-w7nn52sk/snips-nlu-rust/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-mundmd59-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/admin/snips-nlu/.env/include/site/python3.5/snips-nlu-rust" failed with error code 1 in /tmp/pip-build-w7nn52sk/snips-nlu-rust/

Any help would be appreciate; really appreciate snips-nlu on my laptop by the way Thanks

opened by laboitemail 1

Support for wasm target?

Is there any intention to support wasm32-unknown-unknown target? I've tried to target web assembly but there are current old or deprecated dependencies that don't support wasm32 looking at the cargo tree. How hard would it be updating or replacing some of the dependencies that use libraries like time rustc-serialize memchr and support wasm?
enhancement

opened by rodrigopivi 1

Releases(0.65.6)

0.65.6(Jan 28, 2020)
Changed

Bumped crfsuite to 0.3.3 #158

Source code(tar.gz)
Source code(zip)
0.65.5(Jan 27, 2020)
Changed

Bumped snips-nlu-parsers to 0.4.3 #157

Source code(tar.gz)
Source code(zip)
0.65.4(Jan 15, 2020)
Changed

Bump crfsuite to 0.3.2 #155

Source code(tar.gz)
Source code(zip)
0.65.3(Sep 12, 2019)
Fixed

Optimize memory footprint of resources #151

Added

Add support for datetime subtypes in French #152

Source code(tar.gz)
Source code(zip)
0.65.2(Sep 6, 2019)
Fixed

Freeze chrono to 0.4.8 to fix issue with rustling-ontology #149

Source code(tar.gz)
Source code(zip)
0.65.1(Sep 4, 2019)
Added

Add new parse_with_alternatives API to SnipsNluEngine to get alternative intents and slot values #148

Source code(tar.gz)
Source code(zip)
0.65.0(Jul 12, 2019)
Changed

Bumped ontology to 0.65.0 #143

Bumped parsers to 0.3.0 #143

Added

Added a lookup intent parser #121

Introduce new builtin entities:

snips/time

snips/timePeriod

snips/date

snips/datePeriod

snips/city

snips/country

snips/region

Source code(tar.gz)
Source code(zip)
0.64.4(Jun 20, 2019)
Changed

Accept ambiguous results from DeterministicIntentParser when confidence score is above 0.5 #141

Bumped ontology to 0.64.7 #142

Bumped parsers to 0.2.2 #142

Source code(tar.gz)
Source code(zip)
0.64.3(Apr 29, 2019)
Fixed

Make the WrongModelVersion error message intelligible #133

Fix error handling in Python wrapper #134

Return an error when using unknown intents in whitelist or blacklist #136

Fix issue with stop words in DeterministicIntentParser #137

Fix caching issue in CustomEntityParser #138

Source code(tar.gz)
Source code(zip)
0.64.2(Apr 9, 2019)
Fixed

Fix handling of ambiguous utterances in DeterministicIntentParser #129

Stop normalizing confidence scores when there is an intents filter #130

Added

Add new APIs in ffi and bindings (python, kotlin, swift) #131

Changed

Leverage entity scopes of each intent in deterministic intent parser #126

Source code(tar.gz)
Source code(zip)
0.64.1(Mar 1, 2019)
Fixed

Fix bug with regex patterns containing duplicated slot names

Source code(tar.gz)
Source code(zip)
0.64.0(Mar 1, 2019)
Changed

Bumped snips-nlu-ontology to 0.64.4

Bumped crf-suite to 0.3.1

Source code(tar.gz)
Source code(zip)
0.63.1(Feb 11, 2019)
Fixed

Fix an issue regarding the way builtin entities were handled by the CRFSlotFiller

Source code(tar.gz)
Source code(zip)
0.63.0(Feb 4, 2019)
Added

get_intents API: get the probabilities of all intents (including the null intent) with respect to an input text

Pass --top_intents to the parsing CLI to use the get_intents API instead of parse

get_slots API: extract slots by providing a text along with its corresponding intent

Added a an optional CooccurrenceVectorizer to the Featurizer that extracts co-occurrence features

Changed

A probability is now returned when no intent is found

The parse API now takes a new optional parameter intents_blacklist which allows to filter out specific intents

Slot object now contains an optional confidence_score attribute

intent value of IntentParserResult is no longer optional: the optionality is moved to intent_name in the IntentClassificationResult object

slots value of IntentParserResult is no longer optional (None is replaced by empty Vec)

Update to Rust 2018

Refactored the Featurizer and moved its attributes to an underlying TfidfVectorizer

Source code(tar.gz)
Source code(zip)
0.61.2(Jan 17, 2019)
Changed

Bump snips-nlu-ontology to 0.61.3

Source code(tar.gz)
Source code(zip)
0.61.1(Dec 14, 2018)
Changed

Bump snips-nlu-ontology to 0.61.2

Fixed

Issue when resolving custom entities

Source code(tar.gz)
Source code(zip)
0.62.0(Nov 26, 2018)
Changed

Bumped snips-nlu-ontology to 0.62.0

Fixed

Made CI faster by running the full test suite only when merging on master

Added

Added a script to update snips-nlu-ontology everywhere in the codebase

Added SNIPS_NLU_VERSION in libsnips_nlu.h and a cbindgen.toml to help us generating libsnips_nlu.h automatically

Added logs in the build script

Source code(tar.gz)
Source code(zip)
0.61.0(Oct 16, 2018)
Changed

Entity injection API is now handled by an NLUInjector object

Added

Support for builtin music entities in english

Fixed

Handle stemming properly in entity injection

Source code(tar.gz)
Source code(zip)
0.60.1(Oct 9, 2018)
Added

Entity injection API for both custom entities and builtin gazetteer entities

Fixed

Swift wrapper

DeterministicIntentParser now relies on the custom entity parser

Source code(tar.gz)
Source code(zip)
0.60.0(Oct 5, 2018)
Added

Support for 3 new builtin entities in French: snips/musicAlbum, snips/musicArtist and snips/musicTrack

Changed

model version 0.16.0 => 0.17.0

Replace snips-nlu-cli crate with Rust example

Fixed

Bug with entity feature name in intent classification

Source code(tar.gz)
Source code(zip)
0.59.0(Oct 5, 2018)
Added

Limited support for Italian by bumping the snips-nlu-ontology to 0.58.0 and snips-nlu-utils to 0.7.0

Source code(tar.gz)
Source code(zip)
0.58.3(Aug 23, 2018)
Fixed

Fix mapping issue when multiple synonyms have same normalization

Source code(tar.gz)
Source code(zip)
0.58.2(Aug 21, 2018)
Changed

Bump snips-nlu-ontology to 0.57.3

Source code(tar.gz)
Source code(zip)
0.58.1(Jul 24, 2018)
Fixed

Error when loading a SnipsNluEngine from zip data

Source code(tar.gz)
Source code(zip)
0.58.0(Jul 17, 2018)
Added

Interactive parsing CLI

Changed

The SnipsNluEngine object is now loaded from a directory instead of a single json file (see https://github.com/snipsco/snips-nlu/releases/tag/0.16.0).

Language resources are now loaded dynamically from the trained engine directory instead of being statically hardcoded, reducing the binary size by 31Mb.

Removed

snips-nlu-resources and snips-nlu-resources-packed crates no longer exists.

FileBasedConfiguration, ZipBasedConfiguration and `NluEngineConfigurationConvertible

Rust examples (replaced by interactive CLI).

Source code(tar.gz)
Source code(zip)
0.57.2(Jul 12, 2018)
Fixed

Conflict with bindgen dependency

Source code(tar.gz)
Source code(zip)
0.57.1(Jul 9, 2018)
Changed

Bump snips-nlu-ontology to 0.57.1

Fixed

Crash when parsing implicit years before 1970

Source code(tar.gz)
Source code(zip)
0.57.0(Jun 8, 2018)
Changed

Improve matching of synonyms

Improve caching strategy for builtin entity parsing

Improve intent classification

Bump model version to 0.15.0

Bump snips-nlu-ontology to 0.57.0

Source code(tar.gz)
Source code(zip)
0.56.1(May 18, 2018)
Changed

Improve calibration of intent classification probabilities

Update the IntentParser API and keep only parse method, while removing get_intent and get_slots

DeterministicIntentParser: Replace tokenized out characters with whitespaces to improve matching

Fixed

DeterministicIntentParser: Fix issue with ranges of custom slots appearing after builtin slots

Source code(tar.gz)
Source code(zip)
0.56.0(May 3, 2018)
Changed

Change ffi signatures

Update swift project to Xcode 9.3

Bump snips-nlu-ontology to 0.55.0

Source code(tar.gz)
Source code(zip)