HAL : Hyper Adaptive Learning

Rust based Cross-GPU Machine Learning.

Why Rust?

This project is for those that miss strongly typed compiled languages. Rust was chosen specifically because of this Furthermore, we can offer fine grained control of your operations. This means being able to grab dimensions of tensors at any stage, none of that unknown shape nonsense. We can also micro-control steps. An example of this working with each individual forward timesteps on say an LSTM. Usually these are controlled by inner loops [Theano/Tensorflow, etc].

Features

Multi GPU [model based] support
OpenCL + CUDA + Parallel CPU support
LSTM's with internal RTRL [Work in Progress]
RNN's [Work in Progress]
Perceptrons, AutoEncoders, ConvNets**[TODO]**
Optimizers: [SGD, Adam, AdaGrad**[TODO]**]
Activations: [Linear, Sigmoid, Tanh, ReLU, LReLU, Softmax]
Initializations: [Lecun Uniform, Glorot Normal, Glorot Uniform, Normal, Uniform]
Data Gatherers: [SinSource, MNIST**[In Progress], CIFAR10[TODO]**]
Loss Functions: [MSE, L2, Cross-Entropy]
OpenGL based plotting and image loading, see here for more info
Multi GPU [horizontal] support [TODO]

Requirements

Rust 1.8 +
Download and install ArrayFire binaries based on your operating system.

Use from Crates.io

To use the rust bindings for ArrayFire from crates.io, the following requirements are to be met first.

Download and install ArrayFire binaries based on your operating system.
Set the evironment variable AF_PATH to point to ArrayFire installation root folder.
Make sure you add the path to library files to your path environment variables.
- On Linux & OSX: do export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$AF_PATH/lib
- On Windows: Add %AF_PATH%\lib to your PATH environment variable.
Add hal-ml as a dependency in your Cargo.toml

Build from Source

Edit build.conf to modify the build flags. The structure is a simple JSON blob. Currently Rust does not allow key:value pairs to be passed from the CLI. To use an existing ArrayFire installation modify the first three JSON values. You can install ArrayFire using one of the following two ways.

To build arrayfire submodule available in the rust wrapper, you have to do the following.

git submodule update --init --recursive
cargo build

This is recommended way to build Rust wrapper since the submodule points to the most compatible version of ArrayFire the Rust wrapper has been tested with. You can find the ArrayFire dependencies below.

Examples

cargo run --example autoencoder
cargo run --example xor_rnn

Testing

HAL utilizes RUST's test framework to extensively test all of our modules.
We employ gradient checking on individual functions as well as layers.
Testing needs to run on one thread due to our device probing methods.
Furthermore, graphics needs to be disabled for testing [glfw issue].

AF_DISABLE_GRAPHICS=1 RUST_TEST_THREADS=1 cargo test

If you would like to see the results of the test (as well as benchmarks) run:

AF_DISABLE_GRAPHICS=1 RUST_TEST_THREADS=1 cargo test -- --nocapture

Credits

Thanks to the arrayfire team for working with me to get the rust bindings up.
Keras for inspiration as a lot of functions are similar to their implementation (minus the theano nonsense).
Dr. Felix Gers for his insight into internal RTRL
Dr. Sepp Hochreiter for advise on LSTM's

Comments

Feature/unitary
Files :

src/data/unit_src.rs : generating real and comlex number for unitary layer

example/test.rs : temporary file to test unitary module

Functions :

src/layer/unitaryr.rs::forward : forward propagation of unitary layer

src/params.rs::addUnitary : initialization of parameters for unitary layer
opened by dducry 3
same build error Win10

Only change I made was to Csrgo.toml to update spmc as the version in there is gone. ArrayFire-rust alone compiles fine

error[E0277]: (dyn std::ops::Fn(defines::AfError) + 'static) cannot be shared between threads safely --> arrayfire-rust\src\error.rs:25:1 | 25 | pub static DEFAULT_HANDLE_ERROR: &'static ErrorCallback = &handle_error_general; | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (dyn std::ops::Fn(defines::AfError) + 'static) cannot be shared between threads safely | = help: within &'static (dyn std::ops::Fn(defines::AfError) + 'static), the trait std::marker::Sync is not implemented for (dyn std::ops::Fn(defines::AfError) + 'static) = note: required because it appears within the type &'static (dyn std::ops::Fn(defines::AfError) + 'static) = note: shared static variables must have a type that implements Sync

opened by clive-g-brown 5
Add thread based data prefetcher

Data loading is slow. To resolve this we need a separate thread(s) that prefetch data into a queue which is used then in the actual model to execute (get_batch_*) calls.

opened by jramapuram 0