basic-parse.conf doesn't seem to be up to date with the parser anymore. A second config has been added to account for this. Alternatively, basic-parse.conf could also be replaced by extended-parse.conf.

The feature definition file is restricted to ASCII_ALPHANUMERIC feature names. I don't know if that's wanted, but there are probably cases where people (me) have features with underscores or some other non-ascii-alphanumeric chars.

Quick fix for this would be replace ASCII_ALPHANUMERIC in the grammar definition file by a newly defined char rule:

feature_name = ${ char+ }
char = _{ !(WHITESPACE | "|" | ":" ) ~ ANY }

or a more defensive change to feature_name by whitelisting non-ascii-alphanumeric letters explicitly

feature_name = ${ (ASCI_ALPHANUMERIC | "_" | "-" )+ }

This removes an external dependency (ragel) for building the feature parser.

@sebpuetz: I thought I'd add you as the reviewer for this one, since we talked about Pest last week, so I though you'd have an above average interest ;).

Transitions is a wrapper of Numberer that has several additional properties that are useful for transition tables.

• It insures that the identifier 0 for unknown transitions.
• It returns the correct length of the transition table, that includes the special identifier 0.
• The table can be both fresh and frozen. A fresh table automatically adds transitions that are not known. A frozen table returns the special identifier 0 when a transition is now known.

For future consideration: provide a thaw method as well?

Currently, dpar assumes that all dependencies are projective, even though they are read from the non-projective column.

Support for pseudo-projective parsing should be added to deal with non-projective structures.

We vectorize all the data before optimizing the graph. This worked fine when we were just storing indices, but now that we store embeddings for embedding layers, memory use is getting out of hand (~30GB on TüBa-D/Z).

I guess we we should generate the batches on the fly instead.

So far, input layers only stored the i32 indices that were lookups into an embedding matrix or a lookup table. A new kind of input vector is now added which makes it possible to use f32 values directly as an input vector. This is convenient for e.g. including association measures like PMI into the training process of the parser. PMIs are retrieved for the parser states that are involved in an attachment.

Note that the lookup of the PMIs is only a dummy HashMap right now. In the next step, the PMIs will be looked up from a file.

This PR adds functionality to retrieve the addresses of the two tokens between which a dependency relation is established. Depending on the transition system, these are tokens from the stack or buffer.

This change introduces a single lookup table that replaces:

• TransitionLookup
• LookupTable
• MutableLookupTable

StoredLookupTable is simplified, but still exists as a wrapper around FreezableLookup to simplify serialization.

FreezableLookup implements three traits:

• LookupValue: defines the methods to look up an identifier for a value and vice versa. In contrast to the lookups that are replaced, values are borrowed during lookup and only cloned upon insertion.
• LookupLen: defines the len method.
• Lookup: inherits LookupValue and LookupLen and is implemented for any type that implements both traits.

In my first iteration, I had a single lookup trait. However, this sometimes led to type inference problems when the len method was used. Since len does not use the borrowed type in its signature, the type parameter for Lookup<B> could not be inferred. So, instead, we make the len method part of a trait without type parameters.

Fixes #19.

Big boring change --- replaces the error-chain crate by failure, which is now the preferred error handling crate in the Rust community. This also allows us to upgrade to conllx 0.10 without too much wrapping.

Ragel dropped support for all languages outside C/C++/ASM. To change/regenerate the addressed value parser, one has to manually compile an old Ragel version.

It would be nicer to just switch to nom, which is Rust-native and well-maintained.

There is a lot of overlap between the transition-specific lookup and feature table lookup. This can be factored out to one class that replaces Numberer.

Maybe this should be a separate crate, because it is generally useful.