A memory profiler for Linux.

Overview

Bytehound - a memory profiler for Linux

Features

  • Can be used to analyze memory leaks, see where exactly the memory is being consumed, identify temporary allocations and investigate excessive memory fragmentation
  • Gathers every allocation and deallocation, along with full stack traces
  • Can dynamically cull temporary allocations allowing you to profile over a long period of time
  • Uses a custom, tailor-made stack unwinding implementation which makes it a lot cheaper than other similar tools, potentially up to orders of magnitude faster in some cases
  • Can export the data it gathered into various different formats; it can export the data as JSON (so you can analyze it yourself if you want), as Heaptrack (so you can use the excellent Heaptrack GUI for analysis) and as a flamegraph
  • Has its own Web-based GUI which can be used for analysis
  • Can dynamically stream the profiling data to another machine instead of saving it locally, which is useful for profiling on memory-constrained systems
  • Supports AMD64, ARM, AArch64 and MIPS64 architectures (where MIPS64 requires a tiny out-of-tree kernel patch for perf_event_open)
  • Supports profiling of applications which use jemalloc as their allocator (only works on AMD64 with the jemallocator crate)
  • Supports an embedded DSL based on Rhai to allow for programmatic and/or automated data analysis

Screenshots

Building

  1. Install GCC, Rust nightly and the Yarn package manager (for building the GUI)

  2. Build it:

     $ cargo build --release -p bytehound-preload
     $ cargo build --release -p bytehound-cli
    
  3. Grab the binaries from target/release/libbytehound.so and target/release/bytehound

Usage

Basic usage

$ export MEMORY_PROFILER_LOG=warn
$ LD_PRELOAD=./libbytehound.so ./your_application
$ ./bytehound server memory-profiling_*.dat

Then open your Web browser and point it at http://localhost:8080 to access the GUI.

Documentation

You can find the full documentation for the profiler in our Memory profiling for fun and profit book.

Enabling full debug logs

By default the profiler is compiled with most of its debug logs disabled for performance reasons. To reenable them be sure to recompile it with the debug-logs feature, e.g. like this:

$ cd preload
$ cargo build --release --features debug-logs

License

Licensed under either of

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Comments
  • 'memalign' is unimplemented! panic

    'memalign' is unimplemented! panic

    The following problem occurred today, it didn't exist before:

    thread '' panicked at 'not implemented: 'memalign' is unimplemented!', preload/src/api.rs:831:5

    opened by harlanc 10
  • SIGSEGV when loading app with libmemory_profiler.so

    SIGSEGV when loading app with libmemory_profiler.so

    When attempting to load my application with libmemory_profiler.so in LD_PRELOAD, it immediately segfaults.

    Built with:

    rustc 1.36.0-nightly (6afcb5628 2019-05-19)
    

    Captured from GDB:

    gdb --args env LD_PRELOAD=/usr/local/lib/libmemory_profiler.so ./build/myapp
    
    Program received signal SIGSEGV, Segmentation fault.
    0x00007ffff7881e75 in memory_profiler::initialize () at preload/src/lib.rs:1239
    1239	fn initialize() {
    (gdb) bt
    #0  0x00007ffff7881e75 in memory_profiler::initialize () at preload/src/lib.rs:1239
    #1  0x00007ffff7883cc9 in memory_profiler::allocate (size=32, is_calloc=<optimized out>) at preload/src/lib.rs:1417
    #2  malloc (size=32) at preload/src/lib.rs:1459
    #3  0x00007ffff7883df6 in memory_profiler::allocate (size=32, is_calloc=<optimized out>) at preload/src/lib.rs:1429
    #4  malloc (size=32) at preload/src/lib.rs:1459
    ...
    
    opened by NuSkooler 10
  • thread 'main' panicked at 'called `Option::unwrap()` on a `None` value'

    thread 'main' panicked at 'called `Option::unwrap()` on a `None` value'

    Hi

    We have troubles when analysing gathered data. Gathered data is about 9GB. We tried to squeeze the data but it failed. Then we tried to load big data and it failed as well

    ./bytehound strip --output X Y.dat
    thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', /tmp/koute/memory-profiler/cli-core/src/squeeze.rs:217:100
    none: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
    Aborted
    

    when loading big data:

    ./bytehound server Y.dat
    [2022-12-01T15:11:46Z INFO  server_core] Trying to load "Y.dat"...
    thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', cli-core/src/loader.rs:864:68
    stack backtrace:
    note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
    Aborted
    

    Did you encounter such problem? Do you know how we should proceed with that?

    opened by krzysiek6d 7
  • A kingdom for a flamegraph of live allocations!

    A kingdom for a flamegraph of live allocations!

    I am profiling an application which uses obscene amounts of memory, but is not, strictly speaking, leaking memory: if I kill the app, everything is cleanly deallocated.

    Still, I'd love to know what takes all the memory. I think what I need is to take a look at the live allocations at some specific point in time, and get a famegraph. This'll give me essentially a profile for a heap snapshot at a point in time.

    Can I already get this with bytehound? I've seen flamegraph for leaked memory, and the graph of the total live allocations, but neither is quite what I am looking for.

    opened by matklad 6
  • thread '<unnamed>' panicked at 'cannot access a TLS value during or after it is destroyed: AccessError'

    thread '' panicked at 'cannot access a TLS value during or after it is destroyed: AccessError'

    On debian10 with nigthly rust

    thread '<unnamed>' panicked at 'cannot access a TLS value during or after it is destroyed: AccessError', src/libcore/result.rs:999:5
    stack backtrace:
       0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
                 at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:39
       1: std::panicking::default_hook::{{closure}}
                 at src/libstd/sys_common/backtrace.rs:71
                 at src/libstd/sys_common/backtrace.rs:59
                 at src/libstd/panicking.rs:197
       2: std::panicking::rust_panic_with_hook
                 at src/libstd/panicking.rs:211
                 at src/libstd/panicking.rs:474
       3: std::panicking::continue_panic_fmt
                 at src/libstd/panicking.rs:381
       4: rust_begin_unwind
                 at src/libstd/panicking.rs:308
       5: core::panicking::panic_fmt
                 at src/libcore/panicking.rs:85
       6: core::result::unwrap_failed
                 at /rustc/37ff5d388f8c004ca248adb635f1cc84d347eda0/src/libcore/macros.rs:18
       7: memory_profiler::unwind::grab
                 at /root/.cargo/git/checkouts/not-perf-e01bfa01482c86ed/9739e8b/nwind/src/local_unwinding.rs:0
                 at preload/src/unwind.rs:166
       8: calloc
                 at preload/src/lib.rs:1423
                 at preload/src/lib.rs:1470
       9: g_malloc0
      10: g_slice_free_chain_with_offset
      11: g_queue_free
      12: __nptl_deallocate_tsd.part.8
      13: start_thread
      14: clone
    Aborted (core dumped)
    
    opened by rmanus 6
  • Failed to profile rust application

    Failed to profile rust application

    Failed to profile rust application due to error

    thread '<unnamed>' panicked at 'not implemented: 'aligned_alloc' is unimplemented!', preload/src/api.rs:907:5
    stack backtrace:
       0:     0x7f9661783cea - std::backtrace_rs::backtrace::libunwind::trace::h972caad916e73545
                                   at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
       1:     0x7f9661783cea - std::backtrace_rs::backtrace::trace_unsynchronized::he59049878fe5a05d
                                   at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
       2:     0x7f9661783cea - std::sys_common::backtrace::_print_fmt::he4a91f9bcfad9b40
                                   at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/sys_common/backtrace.rs:66:5
       3:     0x7f9661783cea - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h51433dc001920472
                                   at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/sys_common/backtrace.rs:45:22
       4:     0x7f966172d10c - core::fmt::write::hc9dbd37d69b2c204
                                   at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/core/src/fmt/mod.rs:1198:17
       5:     0x7f9661762b14 - std::io::Write::write_fmt::h6b2550ce8adb9e04
                                   at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/io/mod.rs:1672:15
       6:     0x7f9661784af5 - std::sys_common::backtrace::_print::h006829bd22a5a4ee
                                   at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/sys_common/backtrace.rs:48:5
       7:     0x7f9661784af5 - std::sys_common::backtrace::print::h0f4d319136ab4456
                                   at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/sys_common/backtrace.rs:35:9
       8:     0x7f9661784af5 - std::panicking::default_hook::{{closure}}::h5b3cdff51fbe7401
                                   at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/panicking.rs:295:22
       9:     0x7f9661785075 - std::panicking::default_hook::hdc1d8baf28b4ffd7
                                   at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/panicking.rs:314:9
      10:     0x7f9661785075 - std::panicking::rust_panic_with_hook::h80e138cc00203db9
                                   at /rustc/d68e7ebc38cb42b8b237392b28045edeec761503/library/std/src/panicking.rs:698:17
      11:     0x7f96617f9221 - nwind_ret_trampoline_start
                                   at /home/user/.cargo/git/checkouts/not-perf-af1a46759dd83df9/18bd8d3/nwind/src/arch/amd64_trampoline.s:17
      12:                0x0 - <unknown>
    

    used command

    LD_PRELOAD=./libbytehound.so <rust_application> <args>
    

    Tried with local build from the latest master, and with pre-built versions Target OS: Ubuntu 20.04

    Please let me know if any option is missing to run tool properly, or rust applications are not support for now. Thanks in advance.

    opened by aregng 5
  • Build memory-profiler-cli failed

    Build memory-profiler-cli failed

    1. My environment is Ubuntu 18.04
    2. Yarn has been installed already
    3. Rust has been changed to nightly version
    4. I run "cargo build --release -p memory-profiler" suceessfully and libmemory_profiler.so has been generate already

    When I run "cargo build --release -p memory-profiler-cli", it failed and the error log as below:

    [email protected]:~/Tools/memory-profiler$ cargo build --release -p memory-profiler-cli Compiling server-core v0.1.0 (/home/davidwang/Tools/memory-profiler/server-core) 00h00m00s 0/0: : **error: failed to run custom build command for server-core v0.1.0** (/home/davidwang/Tools/memory-profiler/server-core)

    Caused by: process didn't exit successfully: /home/davidwang/Tools/memory-profiler/target/release/build/server-core-5dfb1577abf53aa3/build-script-build (exit code: 101) --- stderr ERROR: [Errno 2] No such file or directory: 'install' thread 'main' panicked at 'Failed to install the dependencies for the WebUI; child process exited with error code Some(1)! You might want to try to run 'rm -Rf ~/.cache/yarn' and try again.', server-core/build.rs:61:21 stack backtrace: 0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace at src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:39 1: std::sys_common::backtrace::_print at src/libstd/sys_common/backtrace.rs:71 2: std::panicking::default_hook::{{closure}} at src/libstd/sys_common/backtrace.rs:59 at src/libstd/panicking.rs:197 3: std::panicking::default_hook at src/libstd/panicking.rs:211 4: std::panicking::rust_panic_with_hook at src/libstd/panicking.rs:474 5: std::panicking::continue_panic_fmt at src/libstd/panicking.rs:381 6: std::panicking::begin_panic_fmt at src/libstd/panicking.rs:336 7: semalock::Semalock::with at server-core/build.rs:61 at /home/davidwang/.cargo/registry/src/github.com-1ecc6299db9ec823/semalock-0.2.0/src/lib.rs:99 at /rustc/8869ee03d7f258e1b76a11c6fbb01b5708a9f504/src/libcore/result.rs:639 at /home/davidwang/.cargo/registry/src/github.com-1ecc6299db9ec823/semalock-0.2.0/src/lib.rs:96 8: build_script_build::main at server-core/build.rs:46 9: std::rt::lang_start::{{closure}} at /rustc/8869ee03d7f258e1b76a11c6fbb01b5708a9f504/src/libstd/rt.rs:64 10: std::panicking::try::do_call at src/libstd/rt.rs:49 at src/libstd/panicking.rs:293 11: __rust_maybe_catch_panic at src/libpanic_unwind/lib.rs:85 12: std::rt::lang_start_internal at src/libstd/panicking.rs:272 at src/libstd/panic.rs:388 at src/libstd/rt.rs:48 13: main 14: __libc_start_main 15: _start

    It seems to find install dir failed, but where should the dir be?

    opened by aguludunu 5
  • Idea: Generate memory flamegraph with inferno

    Idea: Generate memory flamegraph with inferno

    Hello, maybe the most things for this are already in place and it's easy to do. Currently manual usage of the heaptrack GUI is needed to see the memory flamegraph. Inferno could be used to directly generate an interactive SVG by exporting a perf-compatible text format (the text format can even be without collapsed stacks, as there is the inferno-collapse-perf command for that).

    opened by pothos 4
  • thread_local_const_init is being stabilized

    thread_local_const_init is being stabilized

    Hi,

    Currently (on 1.59-nightly), bytehound doesn't compile because of the stabilization of thread_local_const_init feature:

    error[E0635]: unknown feature `thread_local_const_init`
     --> preload/src/lib.rs:1:12
      |
    1 | #![feature(thread_local_const_init)]
      |            ^^^^^^^^^^^^^^^^^^^^^^^
    
    For more information about this error, try `rustc --explain E0635`.
    error: could not compile `bytehound-preload` due to previous error
    

    The feature line should be removed from the code from now on I think.

    Cheers, Gerry

    opened by gagbo 3
  • Plotting memory usage of allocations from given backtrace

    Plotting memory usage of allocations from given backtrace

    Adds plots to allocations so that it's possible to tell if given backtrace is the reason for potentially visible memory leak in overview page. Improves greatly feedback regarding if backtrace is a leak or not, as one sees the trend of allocations from this backtrace - are they increasing? or maybe it was just one spike at the end of trace?

    The plot will be visible when "Group by backtraces" is selected.

    PR code is a little wonky though... but it seems to work well.

    opened by stoperro 3
  • Profiling memory usage of Go application

    Profiling memory usage of Go application

    I was thinking about using this project to compare the memory usage of a Go application that I have rewritten in Rust, but when testing the memory-profiler with a simple go example it generated no result.

    Heres the go example I've tested:

    package main
    
    import "fmt"
    import "os"
    import "strconv"
    
    func fib(n uint64) uint64 {
    	switch n {
    	case 0:
    		return 0
    	case 1:
    		return 1
    	default:
    		return fib(n-1) + fib(n-2)
    	}
    }
    
    func main() {
    	input, _ := strconv.Atoi(os.Args[1])
    	n := uint64(input)
    	fmt.Println(fib(n))
    }
    

    And running:

    go build test.go
    LD_PRELOAD=/home/jonathas/Repositories/memory-profiler/target/release/libmemory_profiler.so ./test 40
    

    Didn't generate any dat file at all. I have even tried adding a make or new calls to the fib func in order to introduce more allocations but I still got no results.

    opened by Jonathas-Conceicao 3
  • LD_PRELOAD undefined symbol: memfd_create

    LD_PRELOAD undefined symbol: memfd_create

    Hi, great project! I successfully compiled the project, but when executing this statement:

    LD_PRELOAD=./libbytehound.so ./bytehound serverv
    

    i got: ./bytehound: symbol lookup error: ./libbytehound.so: undefined symbol: memfd_create

    and every program is like this Did I do something wrong?

    Information about env: In docker container (uname -r = 5.10.104-linuxkit) already install gcc g++

    opened by kp-tux 3
  • [Suggestion] Make the case study scripts an interactive page

    [Suggestion] Make the case study scripts an interactive page

    Dearest Maintainer,

    Bytehound has been a joy to use. Thank you for this. I found the "Case study: Memory leak analysis" after searching for how to use bytehound. I have found the example scripting to be amazingly helpful. My ask would to to make that flow in to a page.

    "by backtrace" which gives the chart and then you can select the groups one at a time.

    A page for leaked until the end as well. maybe that is a filter on by backtrace.

    Last question I have is what does "memory lost to fragmentation" really mean? what does one do with this information?

    Either way! thanks for reading this. I looked at the ui folder and saw it was react. I am very lost in that world but one day might take the time.

    Thanks for the amazing software. It has been very helpful.

    Becker

    opened by sbeckeriv 0
  •  How to compile the 32-bit version of bytehound

    How to compile the 32-bit version of bytehound

    I got an error when using bytehound: ERROR: ld.so: object './libbytehound.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS64): ignored.

    I guess it's because my app is 32bit version but bytehound is 64bit. So how should I compile bytehound for use by a 32-bit program?

    System info: Linux dell-7060 5.15.0-46-generic #49~20.04.1-Ubuntu SMP Thu Aug 4 19:15:44 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

    opened by Gary-Hobson 4
  • implement sallocx

    implement sallocx

    hello again!

    i have made an attempt at sallocx. I don't know how to build a C test program with custom _rjem_ prefix, so i copied existing v05 and call jemalloc-sys function in there. This requires pub fn alloc from the common directory.

    about the test data, i don't understand how allocating 12 bytes, rounded up to 24:

      left: `24`,
     right: `16`', src/main.rs:12:9
    

    naively, i thought 1->8, so 12->16 as the closest byte-multiple.

    ps i think the "jemalloc-v05-unprefixed" test is uncommitted somehow.

    opened by hdhoang 0
  • unchanging dat file when applying to official ClickHouse binaries

    unchanging dat file when applying to official ClickHouse binaries

    (apologies for sparse details, I'll copy them more as i can formulate them concretely)

    we're trying out bytehound to analyze a memleak. However, the dat file always come out at 190MB, and have no update after CH finished starting up. Furthermore, the file has 0s runtime, 0B allocation.

    We run on debian11, but CH builds are mostly self-contained. There's a few mode of starting up CH, you can try a one-shot command this way:

    1. obtain a CH as clickhouse-common-static amd64.tgz <= 22.3 from https://packages.clickhouse.com/tgz/stable/ (approx 230MB)
    curl -LO https://packages.clickhouse.com/tgz/stable/clickhouse-common-static-22.3.9.19-amd64.tgz
    tar xf clickhouse-common-static-22.3.9.19-amd64.tgz
    ./clickhouse-common-static-22.3.9.19/usr/bin/clickhouse local -q 'select version()'
    22.3.9.19
    

    Later CH versions ignore LD_PRELOAD btw, so we'll have to use 22.3 here.

    1. run a 9-second script under bytehound (0.9.0), observe that the file is not updated after first Flushing
    env MEMORY_PROFILER_LOG=debug LD_PRELOAD=$HOME/libbytehound.so ./clickhouse-common-static-22.3.9.19/usr/bin/clickhouse local -q 'select toUnixTimestamp(now()); select sleep(3);select sleep(3);select sleep(3); select toUnixTimestamp(now());'
    bytehound: 04b2 04b2 INF Version: 0.9.0
    

    debug log:

    ```text bytehound: 04b2 04b2 INF Version: 0.9.0 bytehound: 04b2 04b2 INF Options: bytehound: 04b2 04b2 INF MEMORY_PROFILER_BASE_SERVER_PORT = 8100 bytehound: 04b2 04b2 INF MEMORY_PROFILER_CHOWN_OUTPUT_TO = None bytehound: 04b2 04b2 INF MEMORY_PROFILER_DISABLE_BY_DEFAULT = false bytehound: 04b2 04b2 INF MEMORY_PROFILER_ENABLE_BROADCAST = false bytehound: 04b2 04b2 INF MEMORY_PROFILER_ENABLE_SERVER = false bytehound: 04b2 04b2 INF MEMORY_PROFILER_GRAB_BACKTRACES_ON_FREE = true bytehound: 04b2 04b2 INF MEMORY_PROFILER_INCLUDE_FILE = None bytehound: 04b2 04b2 INF MEMORY_PROFILER_OUTPUT = memory-profiling_%e_%t_%p.dat bytehound: 04b2 04b2 INF MEMORY_PROFILER_REGISTER_SIGUSR1 = true bytehound: 04b2 04b2 INF MEMORY_PROFILER_REGISTER_SIGUSR2 = true bytehound: 04b2 04b2 INF MEMORY_PROFILER_USE_PERF_EVENT_OPEN = true bytehound: 04b2 04b2 INF MEMORY_PROFILER_USE_SHADOW_STACK = true bytehound: 04b2 04b2 INF MEMORY_PROFILER_WRITE_BINARIES_TO_OUTPUT = true bytehound: 04b2 04b2 INF MEMORY_PROFILER_ZERO_MEMORY = false bytehound: 04b2 04b2 INF MEMORY_PROFILER_GATHER_MMAP_CALLS = false bytehound: 04b2 04b2 INF MEMORY_PROFILER_BACKTRACE_CACHE_SIZE_LEVEL_1 = 16384 bytehound: 04b2 04b2 INF MEMORY_PROFILER_BACKTRACE_CACHE_SIZE_LEVEL_2 = 327680 bytehound: 04b2 04b2 INF MEMORY_PROFILER_CULL_TEMPORARY_ALLOCATIONS = false bytehound: 04b2 04b2 INF MEMORY_PROFILER_TEMPORARY_ALLOCATION_LIFETIME_THRESHOLD = 10000 bytehound: 04b2 04b2 INF MEMORY_PROFILER_TEMPORARY_ALLOCATION_PENDING_THRESHOLD = None bytehound: 04b2 04b2 INF Tracing will be toggled ON (for the first time) bytehound: 04b2 04b2 INF Found 'aligned_alloc' at: 0x0000000019C6F3E0 bytehound: 04b2 04b2 INF Found 'malloc_stats_print' at: 0x0000000019C81CE0 bytehound: 04b2 04b2 INF Found 'malloc_usable_size' at: 0x0000000019C81D80 bytehound: 04b2 04b2 INF Found 'memalign' at: 0x0000000019C72C20 bytehound: 04b2 04b2 INF Found 'dallocx' at: 0x0000000019C7E6C0 bytehound: 04b2 04b2 INF Found 'posix_memalign' at: 0x0000000019C6E5A0 bytehound: 04b2 04b2 INF Found 'free' at: 0x0000000019C72B60 bytehound: 04b2 04b2 INF Found 'mallctlnametomib' at: 0x0000000019C81AA0 bytehound: 04b2 04b2 INF Found 'malloc' at: 0x0000000019C6E4C0 bytehound: 04b2 04b2 INF Found 'nallocx' at: 0x0000000019C81720 bytehound: 04b2 04b2 INF Found 'sdallocx' at: 0x0000000019C815E0 bytehound: 04b2 04b2 INF Found 'mallctl' at: 0x0000000019C81960 bytehound: 04b2 04b2 INF Found 'mallctlbymib' at: 0x0000000019C81B60 bytehound: 04b2 04b2 INF Found 'mallocx' at: 0x0000000019C76220 bytehound: 04b2 04b2 INF Found 'sallocx' at: 0x0000000019C7E260 bytehound: 04b2 04b2 INF Found 'realloc' at: 0x0000000019C74620 bytehound: 04b2 04b2 INF Found 'valloc' at: 0x0000000019C73940 bytehound: 04b2 04b2 INF Found 'calloc' at: 0x0000000019C70180 bytehound: 04b2 04b2 INF Found 'rallocx' at: 0x0000000019C78B40 bytehound: 04b2 04b2 INF Found 'xallocx' at: 0x0000000019C7C340 bytehound: 04b2 04b2 INF Attaching prefixed jemalloc hooks... bytehound: 04b2 04b2 INF Symbol not found: "_rjem_malloc" bytehound: 04b2 04b2 INF Symbol not found: "_rjem_mallocx" bytehound: 04b2 04b2 INF Symbol not found: "_rjem_calloc" bytehound: 04b2 04b2 INF Symbol not found: "_rjem_sdallocx" bytehound: 04b2 04b2 INF Symbol not found: "_rjem_realloc" bytehound: 04b2 04b2 INF Symbol not found: "_rjem_rallocx" bytehound: 04b2 04b2 INF Symbol not found: "_rjem_nallocx" bytehound: 04b2 04b2 INF Symbol not found: "_rjem_xallocx" bytehound: 04b2 04b2 INF Symbol not found: "_rjem_malloc_usable_size" bytehound: 04b2 04b2 INF Symbol not found: "_rjem_mallctl" bytehound: 04b2 04b2 INF Symbol not found: "_rjem_posix_memalign" bytehound: 04b2 04b2 INF Symbol not found: "_rjem_aligned_alloc" bytehound: 04b2 04b2 INF Symbol not found: "_rjem_free" bytehound: 04b2 04b2 INF Symbol not found: "_rjem_sallocx" bytehound: 04b2 04b2 INF Symbol not found: "_rjem_dallocx" bytehound: 04b2 04b2 INF Symbol not found: "_rjem_mallctlnametomib" bytehound: 04b2 04b2 INF Symbol not found: "_rjem_mallctlbymib" bytehound: 04b2 04b2 INF Symbol not found: "_rjem_malloc_stats_print" bytehound: 04b2 04b2 INF Symbol not found: "_rjem_memalign" bytehound: 04b2 04b2 INF Symbol not found: "_rjem_valloc" bytehound: 04b2 04b2 INF Stage 1 initialization finished bytehound: 04b2 04b2 INF Initializing stage 2... bytehound: 04b2 04b2 INF Setting atexit hook... bytehound: 04b2 04b2 INF Registering SIGUSR1 handler... bytehound: 04b2 04b2 INF Registering SIGUSR2 handler... bytehound: 04b2 04b2 INF Stage 2 initialization finished bytehound: 04b2 04b2 DBG Opening perf events; pid=0, cpu=-1, frequency=0, stack_size=0, reg_mask=0x0000000000000000, event_source=SwDummy, inherit=false, start_disabled=false... bytehound: 04b2 04b2 DBG Maximum sample rate: 100000 bytehound: 04b2 04b2 DBG Allocating 16 + 1 pages for the ring buffer for PID 0 on CPU -1 bytehound: 04b2 04b2 DBG Perf events open with fd=5 bytehound: 04b2 04b2 INF Spawning event processing thread... bytehound: 04b2 04b2 INF Tracing was enabled bytehound: 04b2 04b3 INF Starting event thread... bytehound: 04b2 04b3 INF Data ID: 3ce8be8456ba326f23c1fa46e8d647ae bytehound: 04b2 04b3 INF File 'memory-profiling_clickhouse_1660033877_1202.dat' opened for writing bytehound: 04b2 04b3 INF Writing initial header... bytehound: 04b2 04b3 INF Writing wall clock... bytehound: 04b2 04b3 INF Writing uptime... bytehound: 04b2 04b3 INF Writing environ... bytehound: 04b2 04b3 INF Writing maps... bytehound: 04b2 04b3 INF Writing binaries... bytehound: 04b2 04b3 DBG Writing '/usr/lib/x86_64-linux-gnu/ld-2.31.so'... bytehound: 04b2 04b3 DBG Writing '/usr/lib/x86_64-linux-gnu/librt-2.31.so'... bytehound: 04b2 04b3 DBG Writing '/usr/lib/x86_64-linux-gnu/libdl-2.31.so'... bytehound: 04b2 04b3 DBG Writing '/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28'... bytehound: 04b2 04b3 DBG Writing '/usr/lib/x86_64-linux-gnu/libm-2.31.so'... bytehound: 04b2 04b3 DBG Writing '/home/osboxes/tgz/clickhouse-common-static-22.3.9.19/usr/bin/clickhouse'... 1660033877 bytehound: 04b2 04b3 DBG Writing '/usr/lib/x86_64-linux-gnu/libgcc_s.so.1'... bytehound: 04b2 04b3 DBG Writing '/usr/lib/x86_64-linux-gnu/libc-2.31.so'... bytehound: 04b2 04b3 DBG Writing '/usr/lib/x86_64-linux-gnu/libpthread-2.31.so'... bytehound: 04b2 04b3 DBG Writing '/usr/lib/x86_64-linux-gnu/libresolv-2.31.so'... bytehound: 04b2 04b3 DBG Writing '/home/osboxes/libbytehound.so'... bytehound: 04b2 04b3 INF Flushing... ```

    final sql & bytehound output:

    bytehound: 04b2 04b3 INF Flushing...
    0
    0
    0
    1660033886
    bytehound: 04b2 04b2 INF Exit hook called
    bytehound: 04b2 04b3 INF Flushing 0 bucket(s) on exit
    bytehound: 04b2 04b3 INF Flushing 0 bucket(s) on exit
    bytehound: 04b2 04b3 INF Event thread finished
    bytehound: 04b2 04b3 DBG Disabling thread 04b6...
    bytehound: 04b2 04b3 DBG Disabling thread 04b7...
    bytehound: 04b2 04b3 DBG Disabling thread 04b2...
    bytehound: 04b2 04b3 DBG Disabling thread 04b4...
    bytehound: 04b2 04b3 INF Tracing was disabled
    bytehound: 04b2 04b3 DBG Thread dropped: 04B3
    bytehound: 04b2 04b2 INF Exit hook finished
    bytehound: 04b2 04b6 DBG Thread dropped: 04B6
    bytehound: 04b2 04b4 DBG Thread dropped: 04B4
    bytehound: 04b2 04b7 DBG Thread dropped: 04B7
    

    dat file is not updated beyond 2 seconds

    stat -c '%Y %s' memory-profiling_clickhouse_1660033877_1202.dat
    1660033879 198359119
    

    and it contains no info image

    we would love to use bytehound's USR1 signaling to catch the relevant, long-term memleak. do you have any suggestion to gather more info?

    we also tried LD_PRELOAD both bytehound.so & system libjemalloc.so.2 at the same time, but it segfaults/aborts (details TBD). further more, in interactive mode (eg clickhouse local without -q), it aborts right away too.

    thanks for the tooling!

    opened by hdhoang 9
Releases(0.11.0)
  • 0.11.0(Nov 23, 2022)

  • 0.10.0(Nov 17, 2022)

    Major changes:

    • Performance improvements; CPU overhead of allocation-heavy heavily multithreaded programs was cut down by up to ~80%
    • You can now control whether child processes are profiled with the MEMORY_PROFILER_TRACK_CHILD_PROCESSES environment variable (disabled by default)
    • The fragmentation timeline was removed from the UI
    • mmap/munmap calls are now gathered by default (you can disable this with MEMORY_PROFILER_GATHER_MAPS)
    • Total actual memory usage is now gathered by periodically polling /proc/self/smaps
    • Maps can now be browsed in the UI and analyzed through the scripting API
    • Maps are now named according to their source using PR_SET_VMA_ANON_NAME (Linux 5.17 or newer; on older kernels this is emulated in user space)
    • Glibc-internal __mmap and __munmap are now hooked into
    • Bytehound-internal allocations now exclusively use mimalloc as their allocator
    • New scripting APIs:
      • AllocationList::only_alive_at
      • AllocationList::only_from_maps
      • Graph::start_at
      • Graph::end_at
      • Graph::show_address_space
      • Graph::show_rss
      • MapList
      • Map
    • Removed scripting APIs:
      • AllocationList::only_not_deallocated_after_at_least
      • AllocationList::only_not_deallocated_until_at_most
      • Graph::truncate_until
      • Graph::extend_until
    • Removed lifetime filters in the UI: only_not_deallocated_in_current_range, only_deallocated_in_current_range
    • Fixed a rare crash when profiling programs using jemalloc
    • Added support for aligned_alloc
    • Added support for memalign
    • Relative scale in the generated graphs is now always relative to the start of profiling
    • Gathered backtraces will now include an extra Bytehound-specific frame on the bottom to indicate which function was called
    • Minor improvements to the UI
    Source code(tar.gz)
    Source code(zip)
    bytehound-x86_64-unknown-linux-gnu.tgz(41.70 MB)
  • 0.9.0(Jul 25, 2022)

    Major changes:

    • Deallocation backtraces are now gathered by default; you can use the MEMORY_PROFILER_GRAB_BACKTRACES_ON_FREE environment variable to turn this off
    • Deallocation backtraces are now shown in the GUI for each allocation
    • Allocations can now be filtered according to where exactly they were deallocated
    • Allocations can now be filtered according to whether the last allocation in their realloc chain was leaked or not
    • Profiling of executables larger than 4GB is now supported
    • Profiling of executables using unprefixed jemalloc is now supported
    • New scripting APIs:
      • AllocationList::only_matching_deallocation_backtraces
      • AllocationList::only_not_matching_deallocation_backtraces
      • AllocationList::only_position_in_chain_at_least
      • AllocationList::only_position_in_chain_at_most
      • AllocationList::only_chain_leaked
    • The server subcommand of the CLI should now use less memory when loading large data files
    • The behavior of malloc_usable_size when called with a NULL argument now matches glibc
    • At minimum Rust 1.62 is now required to build the crates; older versions might still work, but will not be supported
    • The way the profiler is initialized was reworked; this should increase compatibility and might fix some of the crashes seen when trying to profile certain programs
    Source code(tar.gz)
    Source code(zip)
    bytehound-x86_64-unknown-linux-gnu.tgz(40.25 MB)
  • 0.8.0(Nov 16, 2021)

    Major changes:

    • Significantly lower CPU usage when temporary allocation culling is turned on
    • Each thread has now its own first-level backtrace cache; this might result in higher memory usage when profiling
    • The MEMORY_PROFILER_BACKTRACE_CACHE_SIZE environment variable knob was replaced with MEMORY_PROFILER_BACKTRACE_CACHE_SIZE_LEVEL_1 and MEMORY_PROFILER_BACKTRACE_CACHE_SIZE_LEVEL_2 to control the size of the per-thread caches and the global cache respectively
    • The MEMORY_PROFILER_PRECISE_TIMESTAMPS environment variable knob was removed (always gathering precise timestamps is fast enough on amd64)
    • The default value of MEMORY_PROFILER_TEMPORARY_ALLOCATION_PENDING_THRESHOLD is now unset, which means that the allocations will be buffered indefinitely until they're either culled or until they'll live long enough to not be eligible for culling (might increase memory usage in certain cases)
    • Backtraces are now not emitted for allocations which were completely culled
    • You can now see whether a given allocation was made through jemalloc, and filter according to that
    • You can now see when a given allocation group reached its maximum memory usage was, and filter according to that
    • New scripting APIs:
      • Graph::show_memory_usage
      • Graph::show_live_allocations
      • Graph::show_new_allocations
      • Graph::show_deallocations
      • AllocationList::only_group_max_total_usage_first_seen_at_least
      • AllocationList::only_jemalloc
    • New subcommand: extract (will unpack all of the files embedded into a given data file)
    • The strip subcommand will now not buffer allocations indefinitely when using the --threshold option, which results in a significantly lower memory usage when stripping huge data files from long profiling runs
    • malloc_usable_size now works properly when compiled with the jemalloc feature
    • reallocarray doesn't segfault anymore
    • The compilation should now work on distributions with an ancient version of Yarn
    Source code(tar.gz)
    Source code(zip)
    bytehound-x86_64-unknown-linux-gnu.tgz(36.15 MB)
  • 0.7.0(Aug 18, 2021)

    Major changes:

    • The project was rebranded from memory-profiler to bytehound
    • Profiling of applications using jemalloc is now fully supported (AMD64-only, jemallocator crate only)
    • Added built-in scripting capabilities which can be used for automated analysis and report generation; those can be accessed through the script subcommand
    • Added a scripting console to the GUI
    • Added the ability to define programmatic filters in the GUI
    • Allocation graphs are now shown in the GUI when browsing through the allocations grouped by backtraces
    • Improved support for tracking and analyzing reallocations
    • Improved paralellization of the analyzer's internals, which should result in snappier behavior on modern multicore machines
    • The cutoff point for determining allocations' lifetime is now the end of profiling for those allocations which were never deallocated
    • The squeeze subcommand was renamed to strip
    • You can now use the strip subcommand to strip away only a subset of temporary allocations
    • Information about allocations culled at runtime is now emitted on a per-backtrace basis during profiling
    • Fixed an issue where the shadow stack based unwinding was incompatible with Rust's ABI in certain rare cases
    • mmap calls are now always gathered in order (if you have enabled their gathering)
    • Improved runtime backtrace deduplication which should result in smaller datafiles
    • Many other miscellaneous bugfixes
    Source code(tar.gz)
    Source code(zip)
    bytehound-x86_64-unknown-linux-gnu.tgz(34.30 MB)
  • 0.6.1(Jun 10, 2021)

  • 0.6.0(Jun 9, 2021)

    Major changes:

    • Added a runtime backtrace cache; backtraces are now deduplicated when profiling, which results in less data being generated.
    • Added automatic culling of temporary allocations when running with MEMORY_PROFILER_CULL_TEMPORARY_ALLOCATIONS set to 1.
    • Added support for reallocarray.
    • Added support for unwinding through JITed code, provided the JIT compiler registers its unwinding tables through __register_frame.
    • Added support for unwinding through frames which require arbitrary DWARF expressions to be evaluated when resolving register values.
    • Added support for DWARF expressions that fetch memory.
    • Allocations are not tracked by their addresses anymore; they're now tracked by unique IDs, which fixes a race condition when multiple threads are simultaneously allocating and deallocating memory in quick succession.
    • mmap calls are now not gathered by default.
    • Rewrote TLS state management; some deallocations from TLS destructors which were previously missed by the profiler are now gathered.
    • When profiling is disabled at runtime the profiler doesn't completely shutdown anymore, and will keep on gathering data for those allocations which were made before it was disabled; when reenabled it won't create a new file anymore and instead it will keep on writing to the same file as it did before it was disabled.
    • The profiler now requires Rust nightly to compile.
    Source code(tar.gz)
    Source code(zip)
    memory-profiler-x86_64-unknown-linux-gnu.tgz(24.23 MB)
  • 0.5.0(Oct 7, 2019)

    Major changes:

    • Shadow stack based unwinding is now supported on stable Rust and turned on by default.
    • Systems where perf_event_open is unavailable (e.g. unpatched MIPS64 systems, docker containers, etc.) are now supported.
    • The mechanism for exception handling when using shadow stack based unwinding was completely rewritten using proper landing pads.
    • Programs which call longjmp/setjmp are now partially supported when using shadow stack based unwinding.
    • Shared objects dynamically loaded through dlopen are now properly handled.
    • Rust symbol demangling is now supported.
    • Fixed an issue where calling backtrace on certain architectures while using shadow stack based unwinding would crash the program.
    • The profiler can now be compiled with the jemalloc feature to use jemalloc instead of the system allocator.
    • The profiler can now be started and stopped programmatically through memory_profiler_start and memory_profiler_stop functions exported by libmemory_profiler.so. Those are equivalent to controlling the profiler through signals.
    Source code(tar.gz)
    Source code(zip)
    memory-profiler-x86_64-unknown-linux-gnu.tgz(23.39 MB)
  • 0.4.0(Jul 14, 2019)

    Major changes:

    • The profiler can now be compiled on Rust stable, with the caveat that the shadow stack based unwinding will be then disabled.
    • The profiler is now fully lazily initialized; if disabled with MEMORY_PROFILER_DISABLE_BY_DEFAULT the profiler will not initialize itself nor create an output file.
    • The signal handler registration can now be disabled with MEMORY_PROFILER_REGISTER_SIGUSR1 and MEMORY_PROFILER_REGISTER_SIGUSR2.
    • When the profiling is disabled at runtime it will more thoroughly deinitialize itself, and when reenabled it will create a new output file instead of continuing to write data to the old one.
    • The embedded server is now disabled by default and can be reenabled with the MEMORY_PROFILER_ENABLE_SERVER environment variable.
    • The base port of the embedded server can now be set with the MEMORY_PROILER_BASE_SERVER_PORT environment variable.
    • The MEMORY_PROFILER_OUTPUT now supports an %n placeholder.
    • The GUI has now a graph which shows allocations and deallocations per second.
    Source code(tar.gz)
    Source code(zip)
    memory-profiler-x86_64-unknown-linux-gnu.tgz(23.82 MB)
  • 0.3.0(Jun 6, 2019)

  • 0.2.0(May 28, 2019)

    Major changes:

    • Massive performance improvements. In the average case on AMD64 the cost per a single allocation was cut down to 20%; on ARM it was cut down to less than 50%.
    • The profiler no longer crashes when a memory operation is triggered from a destructor of an object residing in TLS.
    • The gathered timestamps are no longer as precise as they were; they should be at most off by ~250ms if your application isn't making a lot of allocations. You can restore the previous behavior if you need it by setting MEMORY_PROFILER_PRECISE_TIMESTAMPS to 1 at the cost of extra CPU time.
    Source code(tar.gz)
    Source code(zip)
    memory-profiler-x86_64-unknown-linux-gnu.tgz(24.29 MB)
Owner
Koute
Koute
A tracing profiler for the Sega MegaDrive/Genesis

md-profiler, a tracing profiler for the Sega MegaDrive/Genesis This program, meant to be used with this fork of BlastEm, helps you finding bottlenecks

null 15 Nov 3, 2022
Simple timings profiler

profl Simple timings profiler Example fn main() -> std::io::Result<()> { profl::init("timings.data"); let mut total = 0; for i in 0..1000

Broxus 1 Dec 9, 2021
Py-spy - Sampling profiler for Python programs

py-spy: Sampling profiler for Python programs py-spy is a sampling profiler for Python programs. It lets you visualize what your Python program is spe

Ben Frederickson 9.4k Nov 27, 2022
🐦 Friendly little instrumentation profiler for Rust 🦀

?? puffin The friendly little instrumentation profiler for Rust How to use fn my_function() { puffin::profile_function!(); ... if ... {

Embark 824 Dec 5, 2022
The axiom profiler for exploring and visualizing SMT solver quantifier instantiations (made via E-matching).

Axiom Profiler A tool for visualising, analysing and understanding quantifier instantiations made via E-matching in a run of an SMT solver (at present

Viper Project 18 Oct 18, 2022
Sampling profiler and tracer for Ruby (CRuby) which runs in BPF

rbperf rbperf is a low-overhead sampling profiler and tracer for Ruby (CRuby) which runs in BPF Build To build rbperf you would need a Linux machine w

Javier Honduvilla Coto 70 Nov 30, 2022
🐝🦀🔥 An ebpf based CPU profiler written in Rust

profile-bee ?? ?? ?? Profile Bee is an eBPF based CPU profiler written in Rust for performance and efficiency. Aya is used for building the BPF progra

Joshua Koo 4 Nov 7, 2022
bustd is a lightweight process killer daemon for out-of-memory scenarios for Linux!

bustd: Available memory or bust! bustd is a lightweight process killer daemon for out-of-memory scenarios for Linux! Features Small memory usage! bust

Pop!_OS 8 Oct 6, 2022
Rust binding of fortran Limited memory LBFGS subroutine

lbfgs-sys Rust binding of fortran L-BFGS-B subroutine. The orginal fortran subroutine is distributed under BSD-3 license. To know more about the condi

Naushad Karim 10 Sep 23, 2022
This crate allows writing a struct in Rust and have it derive a struct of arrays layed out in memory according to the arrow format.

Arrow2-derive - derive for Arrow2 This crate allows writing a struct in Rust and have it derive a struct of arrays layed out in memory according to th

Jorge Leitao 26 Nov 15, 2022
A memory efficient immutable string type that can store up to 24* bytes on the stack

compact_str A memory efficient immutable string type that can store up to 24* bytes on the stack. * 12 bytes for 32-bit architectures About A CompactS

Parker Timmerman 337 Nov 18, 2022
A rust library that makes reading and writing memory of the Dolphin emulator easier.

dolphin-memory-rs A crate for reading from and writing to the emulated memory of Dolphin in rust. A lot of internals here are directly based on aldela

Madison Barry 4 Jul 19, 2022
messloc is a drop in replacement for malloc that can transparently recover from memory fragmentation without any changes to application code.

messloc is a drop in replacement for malloc that can transparently recover from memory fragmentation without any changes to application code. Goals Al

null 10 Nov 2, 2022
A lite tool to make systemd work in any container(Windows Subsystem for Linux 2, Docker, Podman, etc.)

Angea Naming from hydrangea(アジサイ) A lite tool to make systemd work in any container(Windows Subsystem for Linux 2, Docker, Podman, etc.) WSL1 is not s

いんしさくら 17 Nov 16, 2022
Experimental Valve Index camera passthrough for Linux

Index camera passthrough Warning: This is still a work in progress, you could get motion sickness if you try it now The problem that the Index camera

yshui 21 Nov 20, 2022
Freebsd-embedded-hal - Like linux-embedded-hal but FreeBSD

freebsd-embedded-hal Implementation of embedded-hal traits for FreeBSD devices: gpio: using libgpio, with stateful and toggleable support, with suppor

null 2 Oct 1, 2022
Fox Ear is a Linux process behavior trace tool powered by eBPF

Fox Ear Fox Ear is a Linux process behavior trace tool powered by eBPF. Banner image by Birger Strahl on Unsplash. Features Log process and its subpro

Rui Li 75 Nov 9, 2022
Auto Fan Management Utility in Linux Systems for Monster Laptops

Auto Fan Management Utility in Linux Systems for Monster Laptops Monster Laptoplar için Linux Sistemlerde Oto Fan Yönetimi TR Monster laptoplar gömülü

null 2 Aug 22, 2022
Utility for controlling laptop backlight brightness on Linux.

Licht Utility for chaning laptop backlight brightness, supporting different stepping modes. Usage licht USAGE: licht [OPTIONS] <SUBCOMMAND> OPT

null 2 Jul 3, 2022