Recent versions of Tensorflow Keras will automatically switch between cuDNN and Tensorflow implementations. The trained parameters work regardless of the selected implementation.

The conditions for using the cuDNN implementation are documented at:

https://www.tensorflow.org/api_docs/python/tf/keras/layers/LSTM

They boil down to: 1. a NVIDIA GPU is available, 2. certain hyper parameters (e.g. activations) are set to specific values. If the cuDNN implementation is selected, this results in a nice speedup.

The Tensorflow requirements are bumped to 1.15.0. This setup fails with 1.14.0 with a constant folding error in Grappler:

https://github.com/tensorflow/tensorflow/issues/29525

Adds linear learning-rate warmup to sticker train.

The scheduler now needs to be passed into run_epoch since the warmup is per-batch and not per-epoch.

Not the scope of this PR: it may be useful to change ExponentialDecay to decay per-batch instead of per-epoch.

This PR consists of three commits:

• Update protocol buffer definitions to Tensorflow 1.14.0.
• Regenerate Rust Tensorflow protocol buffer files.
• The actual implementation in the graph write scripts and Rust tagger/trainer.

I did not have the opportunity to test this PR yet. Both GPUs on hopper are used. I am currently compiling Tensorflow on tesniere with the right compute capabilities.

For pretraining it's nice to do periodical evaluation and save on improvements. It's even nicer to plot this periodical evaluation in tensorboard since it nicely illustrates how the accuracy fluctuates between pretrain-batches.

For my (non-sticker) experiments, dev-accuracy was still moving up and down by ~0.1% after ~300k train steps, although the model had reached the final level (0.02% below the highscore) after less than 200k train steps.

sticker uses the average of all embeddings for the unknown embedding. However, this has a bad side-effect for quantized embeddings. Since all the embeddings in the embedding matrix are reconstructed, loading models takes quite a while.

Possible workarounds:

• Precompute the unknown embedding and add it to the embedding matrix. I dislike this somewhat, since sticker would need specially prepared matrices again.
• Precompute the unknown embedding and store it somewhere outside the embedding matrix. Breaks existing models.
• Precompute the unknown embedding over a sample of the embeddings.
• Maybe it's enough to average all subquantizers, since they are the centroids of the embeddings? This would be very cheap. Problem: the original embeddings may not be distributed evenly over the clusters.

@twuebi @sebpuetz any opinions?

We currently have sticker and sticker-python. I have also been pondering whether I should move the models from blob.danieldk.eu to a sticker-models repo (where the repo would contain metadata and the associated releases would store the models).

In favor:

• All sticker-related projects are grouped together.
• Hosting models on GitHub may prove to be more reliable.

Against:

• Even stronger coupling to Microsoft GitHub ;). (sticker started as a project that was purely on https://sourcehut.org/ , which was kinda nice, because sr.ht is open source + has far nice/better CI.)
• GitHub releases has size limitations (2GB per file). We do not hit them anymore since I have started quantizing embeddings, but it may prove to be annoying in the future.
• GitHub is been finicky with downloads before. Before Releases they had a download feature, which they suddenly cancelled and there was no download option between this cancellation and when they introduced Releases.
• sticker is already used. So we'd need an organization with a different name (I already reserver glumarko, which is Esperanto for 'sticker').

A good chunk of these lines are docs ;)

This PR adds the transformer model.

In detail:

• the feed-forward residual layers
• the encoder assembly
• the write script including commandline args

I did some preliminary experiments for topo-field tagging and dependency parsing, both perform reasonably well.

While training, the batch loss/accuracy of the last trained batch is shown. However, these numbers tend to jump around quite a bit. Maybe we should show the average so far in the current epoch? This would help observing the trends a bit better.

With the addition of the transformer architecture, we will have three model types, each with their own set of hyperparameters.

Currently, arguments are disambiguated by their names, (--rnn_layers) or by the help string (number of dilated convolution levels). With the new model, this is becoming increasingly confusing, it also raises the question for which model type the defaults apply.

I see a few options to make things clearer:

1. subcommands to determine the model type:

+ all model specific arguments that don't belong to the selected one are inactive + model specific --help + easy to access via a shell script to run some experiments

- only write-graph --help + write-graph <MODEL_TYPE> --help will give the full information - arguments are not persistent

2. read the hyperparameters from a toml file:

+ structure unambiguous + parsing handled by a library + adds persistent metadata to pre-trained models + manipulation for experiments through e.g. python script

- slightly more difficult to automate experiments - need to handle many config files

3. read all hyperparameters from config.py

+ no parsing + persistent storage

- config.py is loaded via import, no simple switching between configs - more difficult to automate experiments - need to handle many config files

4. specific write-graph scripts for each model type

+ clear separation + viable for both reading configs from a file or using argparse

If we stick with a single write-graph, I'd prefer a config.toml (2.). If we switch to multiple write-scripts, sticking to command line arguments should be fine, although, saving the training parameters would be a nice feature. What do you think?

This change restores compatibility with models that are trained using sticker 0.4. The only graph incompatibility is the addition of training and validation summaries for TensorBoard. Make these ops optional, so that prediction (and training) using old graphs continues to work.

The ops could be made mandatory in some future version when the sticker 0.4 models have all been replaced.

https://github.com/tensorflow/tensorflow/commit/826027dbd4277a2636fc2935ed245700fd01e7cd#diff-335a6620279c6fa5d7f70c37c91265a3

this broke our _create_file_writer_generic_type in vendored.py, it was introduced with Tensorflow 1.14. I'm looking into it.

Hello, Thank you for this awesome library. I´m very impressed by the quality.

My task is to extract/segment information from a semi structured text. But not only the text is important some "external features" are also important.

For example, imagine that I want to segment this text about GitHub projects in Category, Project name, URl and description.

I utilize an BIO scheme to tag each html token as a category.

Where token=NLP start_of_p=True bold=True center=True B-Category token=Projects start_of_p=False bold=True center=True I-Category token=Project start_of_p=True bold=True center=False B-Project-name token=Name start_of_p=False bold=True center=False I-Project-name token=: start_of_p=False bold=False center=False I-Project-name

The final result is something like:

Pay attention that some features are important such as: Text formatting (italic, bold, centered), position in text and more...

There is anyway of using those custom external features for training Sticker?

Thank you!

Right now, the exponential decay lr schedule is not available for sticker trainand sticker pretrain. Once #145 is merged, it would make sense to have it available for both subcommands.

This may clutter the command line arguments a bit since we then have:

• Plateau decay

  • lr_scale
  • lr_patience

• Exponential decay

  • decay_rate
  • decay_exponent

Maybe it would make sense to move the learning-rate schedule related things to the config file.

Once an epoch is done, we start a new line. The batch accuracy/loss that is displayed is then that of the last batch of the epoch. These numbers are confusing when inspecting the outputs for accuracies/losses.

Add epoch-averaged (train and dev) values of the summarized metrics to tensorboard. After a few epochs it's hard to tell anything from the per-batch graphs.