Entails changes to finalfusion-rust for serialization/usage.

• [x] Add config
• [x] Restructure vocab into module (vocab.rs for all variants is quite unwieldy)
• [x] Extend vocab module
  • Use indexer approach from finalfusion-rust, parameterize SubwordVocab with Indexer, separate ngram and word indices in the index through enum (might get ugly, considering how many type parameter we already have for the trainer structs)
  • Implement independent vocab type
• [x] Add support in binaries (depends on support in finalfusion-rust)
• [x] Update finalfusion-rust dependency once it supports NGramVocabs
• [x] Replace finalfusion dependency with release.
• [x] fix #72

Hello. I'm a genomics researcher interested in using finalfrontier to create embeddings based on DNA and protein sequences. Unfortunately, I'm a bit new to rust and very new to the finalfrontier codebase. I've got a few issues right away I need some help with (but likely to have many more).

For DNA we use kmers (like ngrams, as DNA is essentially very large continuous strings) with a sliding window approach, and I've code to count a large corpus in about 3 hours (300 million unique kmers, 50Gb compressed -- almost as fast as FastText, without the extra processing on the front-end nor the extra few hundred Gb of data). I'd like to get this into a SubwordVocab.

1. It would be great to get a function to supplement count (vocab/mod.rs) and pass a known value? As the corpus is already processed this would speed things up rather than calling count() multiple times. I can create a PR if that would help.

2. Is it possible to create a way to skip bracketing for ngram creation? Happy to create a PR as well.

3. Is it possible to create a set of specific ngram values instead of a range: 9 and 11, rather than 9, 10, and 11?

4. I am storing everything at Vec but it seems like everything is String in finalfusion. This is more of a performance question, will it hurt anything if I switch all of my kmers over to String?

Or -- Should I focus instead on creating a different vocab implementation instead so as not to mess up anything you have already?

Any and all help is greatly appreciated!

Cheers, --Joseph

I think the norms storage change is pretty important. The earlier we push 0.6.0 out, the better, since it reduces the number of embeddings in the wild that do not have norms.

That said, I think it would be nice to have Nicole's directional skipgram implementation in as well, since then we also have a nice user-visible feature.

Is there anything else that we want to add before branching 0.6?

Adds option to projectivize dependency graphs before training.

This is all a bit messier than I hoped. With how the binaries are structured at this point, I couldn't come up with a nice abstraction. For now we could stick with some wrapper struct holding an optional projectivizer:

struct SentenceIter<P, R> {
    inner: Reader<R>,
    projectivizer: Option<P>,
}

and implement Iterator<Item = Sentence> for it.

There's some ugliness in the do_work method because HeadProjectivizer doesn't derive Copy + Clone.

Mostly moving files around. I put most of the finalfrontier-utils library stuff into the io module. The remainder is now in the app module.

We have a lot of crate-level symbols that should probably be removed over time (instead exposing just the submodules).

This commit makes SkipGramTrainer generic over the vocabulary.

Mostly replacing SubwordVocab with V and adding trait bounds for V.

Content wise, biggest changes are made in vocab:

The AssociatedIndices trait defines a method to return a slice of indices associated with some lookup idx. SimpleVocab returns Some(&[]), regardless of the input, the implementation on SubwordVocab is just a copy of the subword_indices_idx method.

Trainer delegates the n_input_types method to Vocab, since we can't calculate this number in Trainer without knowing the vocabulary type. Also adds to more flexibility if there'd be an additional input vocabulary involved.

The next PR should be restricted to adding a flag to ff-train and making some of the methods generic.

This PR looks bigger than it is. Mostly hyperparameters are moved around. Since changes to the config mean changes throughout the entire crate, this commit changes quite a number of files.

To allow more flexibility with different configurations, the Config is split up into several specialized structs holding only the hyperparameters relevant to each component.

The CommonConfig struct holds those hyperparameters that (at this point?) are shared by all training variants, including e.g. dims, negative_samples and loss_type.

Additionally, there are two config structs for vocabularies: SimpleVocabConfig and SubwordVocabConfig, holding the relevant hyperparameters.

Finally, there is the SkipgramConfig which holds settings relevant to skipgram-like training routines, i.e. the context_window and model_type determining whether vanilla skipgram is used or structgram.

Since different Trainer instantiations can and will have different sets of configurations, changes were necessary to the WriteModelBinary impl on Trainmodel, too. Instead of converting the Config of the Trainmodel to a toml::Value, Trainer now defines a metadata method

/// Get this Trainer's hyperparameters as `toml::Value`.
fn to_metadata(&self) -> Result<Value, Error>;

leaving it to the Trainer implementations to take care of the conversion.

The SkipgramTrainer's takes care of this conversion through an additional struct:

#[derive(Clone, Copy, Debug, Serialize)]
struct SkipgramMetadata<V>
where
    V: Serialize,
{
    common_config: CommonConfig,
    #[serde(rename = "model_config")]
    skipgram_config: SkipGramConfig,
    vocab_config: V,
}

which is converted to a toml::Value.

This framework is compatible with training dependency embeddings without requiring a lot of additional changes. The next steps in my mind are to make the SkipgramTrainer's input vocabulary generic, then adjust ff-train accordingly and finally introduce the DepembedsTrainer in a third PR. After the DepembedsTrainer is there, only minor changes would be left to ff-train (or a decision to split things into different crates/binaries).

I have implemented support for training floret embeddings, but the command-line gets a bit unwieldy. Floret is quite a bit different than what we have so far:

• We need an option to set the number of hashes.
• We need an option to set the seed for murmur3.
• Upstream floret doesn't use a matrix size that is a power of 2, it would be nice to provide the same freedom in finalfrontier.
• Most output formats do not really make sense for floret, e.g. the word2vec and text formats are useless, since floret does not use word embeddings.

I see two ways forward:

1. We add the necessary options and validations to ensure that no incompatible set of options is used.
2. We add another level of subcommands, with only the relevant set of options, e.g.: finalfrontier skipgram floret, finalfrontier skipgram fasttext, finalfrontier skipgram buckets, finalfrontier skipgram explicit and the same for deps.

For (2), I am not sure if this is the best partitioning.

1. Add an option vocab_size allowing people to set a target vocabulary size, due to which "mincount" is sized so that the vocabulary size is less than the target vocab size.
2. Add enum VocabCutoff with two variants: TargetVocabSize and MinCount
3. Refactor From<VocabBuilder<SimpleVocabConfig, T>> for SimpleVocab and From<VocabBuilder<SubwordVocabConfig, T>> for SubwordVocab, so that Vocabs can be built from corresponding VocabConfig with TargetVocabSize or MinCount

Based on: #58

I see two ways:

1. Extend finalfusion to handle files with both input and output matrices
2. Dump the output matrix + vocab in a seperate finalfusion model.

Re 1.: More work and might make APIs (more) complex. Re 2.: Hackier solution, output types need to implement to_string(). Lookup consequently also done through stringly keys.

Rather light weight changes with lots of lines.

While implementing the ff-deps binary I realized I didn't really like the same AppBuilder for both training types. I defined SkipGramApp and DepembedsApp structs that handle all clap-related business and construct all the required configs. The required configs and values can be accessed through getter methods.

Next step after this PR is the actual binary, then clean up (there were some incorrect docs somewhere in the dep module) and we should be ready to release.

Bumps regex from 1.5.4 to 1.5.6.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps crossbeam-utils from 0.8.5 to 0.8.8.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Hi @danieldk @sebpuetz ,

I just want to say thanks. You saved me from a text classifier deadline where flair/bert embedding would be too slow, and I was unable to do any magical invocation(tried version 3.8, version 4, other parameters) to force gensim to train an working word2vec, the model would simplify not converge and got way worse on each epoch. You rock and finalfrontier from my POV look like the only game in town ( spaCy was lack-lusting even with floret)!!!

As you are using Hogwild! for the multicore SGD implementation, it would perhaps be interesting to investigate whether you can speed up the optimization with

• AsySVRG (https://arxiv.org/abs/1508.05711)
• ASAG (https://arxiv.org/abs/1606.04809)

ps nice project you have there :+1:

The vast majority of time during training is spent in the dot product and scaled additions. We have been doing unaligned loads so far. I have made a quick modification that ensures that every embedding is aligned on a 16-byte boundary and changed the SSE code to do aligned loads, the compiled machine code seems ok and the compiler even performs some loop unrolling.

Unfortunately, using aligned data/loads does not seem to have a measurable impact on running time. This is probably caused by those functions being constrained by memory bandwidth.

I just wanted to jot down two possible opportunities for reducing cache missed that might have an impact on performance.

1. Some papers that replace core word2vec computations using by kernels sample a set of negatives for a sentence, rather than for each token. In the base case the number of cache misses due to negatives is reduced by a factor corresponding to the sentence length. Of course, this modification may have an impact on the quality of the embeddings.

2. The embeddings in the output matrix and the vocab part of the input matrix are ordered by the frequencies of the corresponding tokens. This might improve locality (due to zipf's law). However, the lookups for subword units are randomized by the hash function. Maybe something can be gained by ordering the embeddings in the subword matrix by hash code frequency. However, in the most obvious implementation it would add an indirection (hash index code -> index).

@sebpuetz