Tests fail to finish on M1 Mac

$ cargo test device_new --features device_tests
running 1 test
test device::tests::device_new has been running for over 60 seconds
error: test failed, to rerun pass '--lib'
Caused by:
  process didn't exit successfully: `/Users/rjzak/Downloads/autograph/target/debug/deps/autograph-868587c6365604da device_new` (signal: 9, SIGKILL: kill)

$ cargo test --features "full device_tests"
test device::buffer::tests::device_buffer_copy_from_slice has been running for over 60 seconds
test device::buffer::tests::device_buffer_serde has been running for over 60 seconds
test device::buffer::tests::fill_bf16 has been running for over 60 seconds
test device::buffer::tests::fill_f16 has been running for over 60 seconds
test device::buffer::tests::fill_f32 has been running for over 60 seconds
test device::buffer::tests::fill_f64 has been running for over 60 seconds
test device::buffer::tests::fill_i16 has been running for over 60 seconds
test device::buffer::tests::fill_i32 has been running for over 60 seconds
error: test failed, to rerun pass '--lib'
Caused by:
  process didn't exit successfully: `/Users/rjzak/Downloads/autograph/target/debug/deps/autograph-aa9dbc5e89ab94bc` (signal: 9, SIGKILL: kill)

$ uname -a
Darwin macmini.local 21.1.0 Darwin Kernel Version 21.1.0: Wed Oct 13 17:33:24 PDT 2021; root:xnu-8019.41.5~1/RELEASE_ARM64_T8101 arm64
$ rustc --version
rustc 1.57.0 (f1edd0429 2021-11-29)

Downloaded as zip file, ran cargo run --example mnist_dense --features "datasets" --release:

CMake Error: The source directory "autograph-master/oneDNN" does not appear to contain CMakeLists.txt.

This was just a quick test, maybe I’m missing some obvious step.

From your examples (https://github.com/charles-r-earp/autograph/blob/master/examples/mnist_lenet5.rs) it seems that one has to write a lof of boilerplate code to do some actual learning. Is it possible to provide some kind of default Model struct with builder that would remove the need to write all this? I think it could look the following way:

struct Model {
    layers: Vec<Layer>
}

impl Model {
    fn build() -> Self
    fn fit(&mut self, ...)
    ...
}

I was checking out cargo test on metal and all non-f64 tests pass 🎉

Going forward, I think tests parametrized on dtype and Device (include/exclude via test params?) would make for a DRY yet granular foundation. I'm no expert in rust fixtures, but saw rstest and macro approaches.

Do you have any opinion on that?

Fixes #46

Reworked this to be more robust and deterministic. The basic idea is to select all of the heaps that are DEVICE_LOCAL and are of sufficient size, and use those for allocating "storage" (ie device) memory. This is used for Buffer's, compute operations. Then "mapping" memory is selected in a similar way, for heaps that are not DEVICE_LOCAL, sorted such that we prefer CPU_CACHED and require CPU_VISIBLE and COHERENT. The trick is that some drivers, at least Microsoft Basic Render Driver (used for testing on github actions) has just one heap that is DEVICE_LOCAL. So in the case that there isn't a non DEVICE_LOCAL heap, we don't require mapping memory to be not DEVICE_LOCAL. Mapping memory is used for staging buffers for both writes and reads.

In theory, for configs where you have DEVICE_LOCAL and CPU_VISIBLE memory, the host can write and read directly from this memory (so long as the gpu isn't using it), which saves a copy. However, this is very complex and in fact the storage allocator aliases Buffer's to allow for temporaries to be reused, which isn't the case for writes and reads which have to be fully allocated for a given frame. The allocation scheme is different too, because of the different usage pattern, so it would be difficult to take advantage of being able to say write directly into a buffer from the host and use it in a compute shader, and read back the results without staging buffers. In general the assumption is that this overhead is small when the device has to do a bunch of work with the data but for inference on mobile this may be a consideration.

I added tests to verify expected behavior on each of my dev platforms, and I think this should work on anything that is in fact supported. Unfortunately I can't find the relevant info to create additional tests without extracting the memory config manually via cargo test allocator_config_diagnostic -- --ignored --nocapture, which will print out the MemoryProperties and it's then simple to construct a unit test for those properties.

Currently 64 bit operations are not fully supported on windows and macos. Additionally, 8 and 16 bit ops require extensions, and may not be fully supported as well.

For images, it is very beneficial to load data as u8 to the device and then convert it to floating point, as this increases bandwidth by a factor 4. What I did before was just pack the u8's into a u32, and then do bitwise operations to extract those into 4 u32's on the device. Not sure on the performance, but at least it's the most portable.

I would like to support bf16 eventually, even if 16 bit values are simply converted to f32s for operations. Potentially this may be faster due to 2x the bandwidth and memory.

Addresses #21. Along with branches flatten_layer and forward_requires_layer, you could then do something like this:

fn lenet5(device: &Device) -> impl Forward<Ix4, OutputDim=Ix2> {
    Sequential::builder()
        .layer(
            Conv2d::builder()
                .device(&device)
                .inputs(1)
                .outputs(6)
                .kernel(5)
                .build();
        )
        .layer(Relu::default())
        .layer(
            MaxPool2d::builder()
                .args(
                    Pool2dArgs::default()
                        .kernel(2)
                        .strides(2)
                )
                .build()
        )
        .layer(
            Conv2d::builder()
                .device(&device)
                .inputs(6)
                .outputs(16)
                .kernel(5)
                .build()
            )
        )
        .layer(Relu::default())
        .layer(
            MaxPool2d::builder()
                .args(
                    Pool2dArgs::default()
                        .kernel(2)
                        .strides(2)
                )
                .build()
        )
        .layer(Flatten::default())
        .layer(
            Dense::builder()
                .device(&device)
                .inputs(256)
                .outputs(120)
                .build()
        )
        .layer(Relu::default())
        .layer(
            Dense::builder()
            .device(&device)
            .inputs(120)
            .outputs(84)
            .build()
        )
        .layer(Relu::default())
        .layer(
            Dense::builder()
                .device(&device)
                .inputs(84)
                .outputs(10)
                .bias()
                .build()
        )
        .build()
}

Fixes #26 Moving oneDNN to it's own crate might make it easier to manually download and build. This could even be a sub crate within this repo. That should trigger cargo to do the download for you.

Hi, I use the network defined in https://github.com/charles-r-earp/autograph/blob/main/examples/neural-network-mnist/src/main.rs The output of the model in inference as_raw_slice() is F32(BufferBase { device: Device(0), len: 14190, elem: "f32" }). Now I try to get a Vec<32> of it to interpret the results. Can you help me to understand how to get the data?

Here is a peace of code:

autograph=v0.1.1

let prediction = self
        .net
        .clone()
        .into_device(device)
        .await
        .unwrap()
        .infer(&x)
        .unwrap();
let prediction: FloatBuffer = prediction
        .as_raw_slice()
        .into_device(Device::host())
        .await
        .unwrap();
println!("{:?}", prediction);
// F32(BufferBase { device: Host, len: 14190, elem: "f32" })

Some examples show to "read" data back: let output = y.read().await?; But FloatBuffer does not have the read() function.