Summary

Now that GlobalAlloc is in stable Rust, we should cfg-off the Alloc implementation such that wee_alloc can be used as a global allocator in stable rust.

https://blog.rust-lang.org/2018/08/02/Rust-1.28.html

Motivation

Stabilize all the things!

Details

Will need to introduce a nightly cargo feature to turn on the Alloc implementation. Since the GlobalAlloc implementation proxies to the Alloc implementation, we will need to do some light refactoring to move those proxied methods into normal, non-trait methods on WeeAlloc.

This could be a good issue for someone who wants to dive into the code base for the first time!

Maybe a weird idea, but what do you think about

Summary

Move units.rs to separate crate and publish it on crates.io?

Motivation

I came up with that idea when I was working on wasmi. I noticed that Pages→Bytes and Bytes→Pages are quite common thing that people want to do, when working with parity-wasm/wasmi. Example here and there.

Details

I think just taking units.rs and moving it into another crate will be enough. I'm ready to take this if we agree if this idea is worth it.

Summary

Include a sentence or two summarizing the bug.

Does not build with rust nightly

Steps to Reproduce

$ cargo build

error[E0152]: duplicate lang item found: `panic_fmt`.
  --> example/src/lib.rs:22:1
   |
22 | / extern "C" fn panic_fmt(_args: ::core::fmt::Arguments, _file: &'static str, _line: u32) -> ! {
23 | |     unsafe {
24 | |         ::core::intrinsics::abort();
25 | |     }
26 | | }
   | |_^
   |
   = note: first defined in crate `std`.

error[E0152]: duplicate lang item found: `oom`.
  --> example/src/lib.rs:31:1
   |
31 | / extern "C" fn oom() -> ! {
32 | |     unsafe {
33 | |         ::core::intrinsics::abort();
34 | |     }
35 | | }
   | |_^
   |
   = note: first defined in crate `std`.

error: aborting due to 2 previous errors

For more information about this error, try `rustc --explain E0152`.
error: Could not compile `wee_alloc_example`.


$ /usr/bin/ld --version
GNU ld (GNU Binutils for Ubuntu) 2.26.1
Copyright (C) 2015 Free Software Foundation, Inc.
...

Same with rustc nightly 2018-04-27 and 2018-05-09

Summary

When I create a Vec of Enums that have a u64 field on them I get the following error: RuntimeError: unreachable

Steps to Reproduce

• git clone https://github.com/masonforest/wee_alloc_vec_of_enums_bug
• cd wee_alloc_vec_of_enums_bug
• cargo rustc --target wasm32-unknown-unknown --lib -- -O && node run.js

Actual Results

    Finished dev [unoptimized + debuginfo] target(s) in 0.0 secs
(node:6108) UnhandledPromiseRejectionWarning: RuntimeError: unreachable
    at run (wasm-function[7]:1)
    at run (/Users/masonf/src/wee_alloc_vec_of_enums_bug/run.js:20:20)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:160:7)
    at Function.Module.runMain (module.js:703:11)
    at startup (bootstrap_node.js:193:16)
    at bootstrap_node.js:617:3
(node:6108) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:6108) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Expected Results

Finished dev [unoptimized + debuginfo] target(s) in 0.0 secs

This introduces a script that uses valgrind to trace some program's mallocs and frees, and then it turns those into something that can be used as test/benchmark inputs for wee_alloc.

cc @alexcrichton; you may also find this useful for the other rust allocators you maintain ;)

Problem statement

The wee_alloc implementation of the static_array_backend feature erroneously causes an out-of-memory error. The context where the problem arises is in compilation for a bare-metal embedded system (OS independent, target is something like armv7-unknown-linux-gnueabihf), using the static_array_backend feature and assigning wee_alloc as the global allocator.

The root cause of the resulting error is the implementation of imp_static_array::alloc_pages(). The function creates a pointer to a slice of the SCRATCH_HEAP of the requested size, which is used to create the FreeCell objects that compose the free list. The function does not enforce that the slice pointer maintain any byte boundary alignment, resulting in unaligned FreeCell objects. This ultimately creates an edge case error where allocation fails despite adequate memory left in the SCRATCH_HEAP. The error occurs regardless of the configured static backend array size exported at compile time. Additionally, the failure arises regardless of size policy (e.g. whether or not the size_classes feature is enabled).

Detailed description of the control flow in which the problem arises

The problem is immediately detectable by enabling the extra_assertions feature. With this feature enabled, the first call to allocate memory from the free list results in an alignment assertion failure, because as described above, the FreeCell objects created from the SCRATCH_HEAP object do not have the proper alignment. However, when extra_assertions feature is not enabled, the edge case in the example system occurs when the first 256 Kb of the free list are exhausted a.k.a. a full walk of the initial free list results in alloc_with_refill() executing the code branch to create and insert a new FreeCell.

Upon an allocation request, the allocator attempts to allocate from the SCRATCH_HEAP. This is done via call to the GlobalAlloc trait’s implementation of alloc(), which calls alloc_impl(), which then calls alloc_with_refill() within a function closure passed to self.with_free_list_and_policy_for_size. The alloc_with_refill function first calls alloc_first_fit() which walks the free list to find a FreeCell of adequate size. After the 256 Kb exhaustion point, alloc_first_fit fails to find a FreeCell of adequate size. At this point, the branch for creating and inserting a new FreeCell is executed via calls to new_cell_for_free_list() and insert_into_free_list(). Control flow then returns to alloc_with_refill, which calls alloc_first_fit again, which in turn calls try_alloc() on the newly inserted FreeCell object, which is the point of failure. The checks in try_alloc() assert that the newly inserted FreeCell is large enough to hold the requested allocation size, but that it is not big enough to be split in two. This leads to the final check in try_alloc(), which determines if the FreeCell is appropriately aligned, which, because of how alloc_pages() is implemented for the impl_static_array mod, results in a failure to allocate.

Steps to reproduce problem

1. Use the static array backend feature, compiled for a no_std target (e.g. `armv7-unknown-linux-gnueabihf`) and use the default backend array bytes size
   

2. Allocate via `alloc::Vec::Vec()` an empty vector of u32s, and then push 65 data to it.
   

3. wee_alloc will fail to allocate more heap memory at the attempt to push the 65th u32 to the vector, triggering an oom() error (if the system has one defined) via the method described above
   

Proposed problem fix

To address this error, the imp_static_array::alloc_pages() function should enforce alignment requirements such that pointers to slices of the SCRATCH_HEAP are always aligned to the needed byte boundary. The resulting FreeCell objects will therefore always meet alignment checks, whether or not the extra_assertions feature is enabled.

Updates the requirements on rand to permit the latest version.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Note: This repo was added to Dependabot recently, so you'll receive a maximum of 5 PRs for your first few update runs. Once an update run creates fewer than 5 PRs we'll remove that limit.

You can always request more updates by clicking Bump now in your Dependabot dashboard.

Hello, wee_alloc team! @fitzgen @pepyakin @DrGoldfire

Summary

This PR expands the usability of wee_alloc for arbitrary #[no_std] targets with a bit of memory lying around.

What

The main change is the introduction of a static_array_backend feature that causes wee_alloc to use an implementation that works largely independent of the operating environment capabilities, at the cost of having a completely fixed maximum size, currently 32 megabytes.

This feature is entirely conditionally compiled, and not present by default, and thus should have no noteworthy burden on extant code capabilities or size.

Per the CONTRIBUTING guidelines, cargo fmt --all was run on this branch, which caused some nonessential changes to unrelated files.

Why

This contribution rose out of my desire to apply wee_alloc, a well-tested and actively developed allocator in the context of a non-Unix/WASM/Windows environment.

I hope that an OS-independent option will also provide other developers a quick way to get wee_alloc up and running for use cases associated with less common operating systems or environments.

Future Directions

I could imagine further refinement by using either feature flags, tricky macros, or generics to allow control of the size of the presently-fixed static array backend. Said refinements have been intentionally excluded from this PR in order to keep the complexity down and, more importantly, avoid any impact on the primary use cases. If there is interest, we could expand the backend flexibility in future efforts.

This makes it so that the neighbors list, and all of its pointers and their low bits, are abstracted out a little bit. This should help pave the way for using https://github.com/fitzgen/intrusive_splay_tree soon.

Surprisingly, this is also results in something like a 10x speed up on the #[bench]s and a small code size bump. Happy to take the speed up, and unconcerned about the size bump, because this is a stepping stone towards larger, more fundamental refactorings in the future.

Recent changes in nightly broke wee_alloc build. In an attempt to solve the issues I prepared the following PR.

Unfortunately, not all changes are resolved at the moment, so I would really appreciate any suggestions and comments on the remaining issues.

Currently there are three of them:

• [x] definition of unknown language item panic_fmt at example/src/lib.rs:21:1
• [ ] ~~duplicate lang item found: oom at example/src/lib.rs:31:1~~
• [ ] ~~duplicate lang item found: eh_personality at example/src/lib.rs:39:1~~

Aside of that, the rest of the code compiles successfully.

Summary

Allow the static_array_backend to be optionally sized at compile time by the end user

What

A build script has been added to wee_alloc that reads the WEE_ALLOC_STATIC_ARRAY_BACKEND_BYTES environment variable, does some sanitization to be sure it fits in u32, and records the numeric value as UTF8 in a "wee_alloc_static_array_backend_size_bytes.txt" file in the build output directory. This file is directly included in wee_alloc/src/imp_static_array.rs as the value for SCRATCH_LEN_BYTES.

This environment variable usage is entirely optional. The default remains the same, at 32 megabytes.

Why

To allow users with direct knowledge of the memory available on their systems to better fit their use of the static_array_backend feature to the resources at hand.

Summary

There's a pretty big bug #106 that sees a lot of projects switch away from using this crate. It hasn't received any comment or apparent attempt at fixing from anyone in the rust wasm org/team in 2 months.

Also, there has only been 1 commit in the last 3 years. Is this crate still maintained? It looks like @fitzgen is the original author and the other team members are @ZackPierce and @pepyakin.

Maybe the bug could be mentioned in the readme to raise awareness, so that people don't spend a lot of time debugging something that is not solvable.

What are alternatives for this crate?

This crate is quite popular at 2k daily downloads.

Describe the Bug

Making two large allocations, then dropping them in the order they were allocated leaks memory in wee_alloc, but not in default allocator.

Steps to Reproduce

Native code

1. Clone this branch of my app: https://github.com/CraigMacomber/wee_alloc_leak_png
2. cargo run
3. GB of memory is consumed per second
4. Comment out use of wee_alloc global_allocator.
5. cargo run
6. Memory use does not increase.

Wasm

1. Clone this branch of my app: https://github.com/noencke/uuid-cluster/tree/wee_alloc_leak
2. npm run serve (runs wasm-pack, npm install and webpack dev server)
3. go to localhost:8080 in browser
4. click the button (it's the only thing on the page) to increase heap by 131 MB each time

Expected Behavior

Second time allocations are made uses free list.

Actual Behavior

Repeated allocations grow heap infinitely.

Additional Context

This seems similar to an issue mentioned in #105 Heap size does not increase if using default allocator.

Rust source for example is just:

extern crate wee_alloc;

// Use `wee_alloc` as the global allocator.
#[global_allocator]
static ALLOC: wee_alloc::WeeAlloc = wee_alloc::WeeAlloc::INIT;

pub fn leak_test() {
    // This leaks when using wee_alloc
    let a = Box::new([0; 85196]);
    let b = Box::new([0; 80000]);
    drop(a);
    drop(b);
}

fn main() {
    loop {
        leak_test();
    }
}

Describe the Bug

Unfortunately, I'm not sure. Using wee_alloc causes bugs in Bevy, suggesting a bug somewhere in the allocator.

Steps to Reproduce

See https://github.com/bevyengine/bevy/issues/3763

Expected Behavior

No memory corruption bugs.

Actual Behavior

Memory corruption.

The README currently specifies to use extern crate wee_alloc;, but this is only needed for Rust 2015 and not for Rust 2018. This is distracting for new Rust users who wonder why this is needed.

Either remove this usage of extern crate or clearly mark it as only needed for Rust 2015.

Describe the Bug

Allocating usize::MAX - 8 bytes fails but allocates new memory pages every time.

Steps to Reproduce

#[test]
fn cannot_alloc_max_usize_m8() {
    let a = &wee_alloc::WeeAlloc::INIT;
    let layout = Layout::from_size_align(std::usize::MAX - 8, 1)
        .expect("should be able to create a `Layout` with size = std::usize::MAX - 8");
    for _ in 0..10000000 {
        let result = unsafe { a.alloc(layout) };
        assert!(result.is_err());
    }
}

Expected Behavior

The test should complete without causing OOM.

Actual Behavior

With debug assertions: thread 'cannot_alloc_max_usize_m8' panicked at 'attempt to add with overflow', .../.cargo/registry/src/github.com-1ecc6299db9ec823/memory_units-0.4.0/src/lib.rs:166:1

Without debug assertions: The test allocates tens of gigabytes of memory and eventually gets killed by the kernel.