profile-bee 🐝 🦀 🔥

Profile Bee is an eBPF based CPU profiler written in Rust for performance and efficiency.

Aya is used for building the BPF programs that is attached to perf events for sampling. The beauty of this tool is that it runs from a single binary without the need to install additional libraries such as bcctools or libbpf on target hosts.

Stacktraces are retrieved in the user space program and symbols resolution is handled.

Stacks are counted and sorted and can be output with the follow choices

A SVG flamegraph (generated with inferno) you can load in your browser
Branden Gregg's Stack Collapsed format that can be loaded up using speedscope visualizer
D3 flamegraph JSON and HTML output
Your own custom format

Note: if symbols for your C/Rust programs doesn't appear correct, you may want to build your software with debug information.

With rustc that's adding a -g flag when you compile. Another thing to consider doing is emitting frame pointer by setting RUSTFLAGS="-Cforce-frame-pointers=yes" with building (or modifying ./cargo/config) and -fno-omit-frame-pointer for gcc. With framepointers, you could get symbols while saving on the cost of dwarf parsing (using --no-dwarf)

Usage

# Profile system wide for 5s, generating a html flamegraph
profile-bee --time 5000 --html flamegraphs.html

# Profile at 9999hz for 2s, writing output to profile.svg
profile-bee --svg profile.svg --frequency 9999 --time 2000

# Realtime flamegraphs
profile-bee --time 5000 --serve --skip-idle --stream-mode 1 # Goto http://localhost:8000/ and click "realtime-updates"

# Same as above, grouped by CPU ids
profile-bee --svg profile.svg --frequency 9999 --time 2000 --group-by-cpu

# Profile at 999hz for 10s, writing output to profile.txt
profile-bee --collapse profile.txt --frequency 999 --time 10000

# Kitchen sink of all output formats
profile-bee --time 5000 --html flamegraphs.html --json profile.json --collapse profile.txt --svg profile.svg

# Profile at 99hz for 5s, writing output to screen, idle CPU cycles not counted
cargo xtask run --release -- --collapse profile.txt --frequency 99 --time 5000 --skip-idle

# Profile using kprobe over a short interval of 200ms
profile-bee --kprobe vfs_write --time 200 --svg kprobe.svg

# Profile using a tracepoint over a interval of 200ms
profile-bee --tracepoint tcp:tcp_probe --time 200 --svg tracepoint.svg

# Profile specific pid (or cpu)
profile-bee --pid <pid> ...

Features

Rust and C++ symbols demangling supported (via gimli)
Some source mapping supported (via addr2line)
Simple symbol lookup cache
SVG Flamegraph generation (via inferno)
BPF based stacktrace aggregation for reducing kernel <-> userspace transfers
Basic Kernel and tracepoint probing
Group by CPUs
Profile target PIDs, CPU id, or itself
Static d3 flamegraph JSON and/or HTML output
Real time flamegraphs served over integrated web server (using warp)

Limitations

Linux only
Interpreted / JIT stacktraces not yet supported
VDSO and binary offsets not calculated

TODOs

Optimize CPU usage
Check stack correctness (compare with perf, pprof etc)
switch over to Perf buffers
implement uprobing/USDT
pid nesting
Off CPU profiling
Optimize symbol lookup via binary search
Publish to crates.io
Measure cache hit ratio
Stacktrace and Hashmap clearing
Missing symbols

Alternatives

Perf
Bcc's profile tool
Cargo flamegraph, utilizing perf without the hassle
Parca-agent, always on profiling with BPF, except using golang.

Development

Prerequisites

Install a rust stable toolchain: rustup install stable
Install a rust nightly toolchain: rustup install nightly
Install bpf-linker: cargo install bpf-linker

Build eBPF

cargo xtask build-ebpf

To perform a release build you can use the --release flag. You may also change the target architecture with the --target flag

Build Userspace

cargo build

Run

cargo xtask run

Some rust function names can be pretty verbose and makes it difficult to be displayed (as well as generates long stress).

Consider stripping generics or even crate prefixes out of function names for more concise labels.

See how it's done in FlameGraph or inferno

An example of long symbols

                                library/alloc/src/raw_vec.rs: alloc::raw_vec::RawVec<T,A>::ptr
      github.com-1ecc6299db9ec823/num_cpus-1.13.1/src/lib.rs: alloc::raw_vec::finish_grow
                                  library/core/src/result.rs: <core::result::Result<T,E> as core::ops::try_trait::Try>::branch
                             gimli-0.26.2/src/read/abbrev.rs: gimli::read::abbrev::AttributeSpecification::new
                                     core/src/num/nonzero.rs: core::num::nonzero::NonZeroUsize::new_unchecked
                                         core/src/ptr/mod.rs: core::ptr::slice_from_raw_parts
                                       core/src/array/mod.rs: core::array::<impl core::ops::index::IndexMut<I> for [T; N]>::index_mut
                                     core/src/slice/index.rs: <core::ops::range::Range<usize> as core::slice::index::SliceIndex<[T]>>::get_unchecked
                                    src/slice/iter/macros.rs: <core::slice::iter::Iter<T> as core::iter::traits::iterator::Iterator>::position
   github.com-1ecc6299db9ec823/gimli-0.26.2/src/endianity.rs: <gimli::endianity::RunTimeEndian as core::clone::Clone>::clone
                      gimli-0.26.2/src/read/endian_reader.rs: gimli::read::endian_reader::SubRange<T>::read_slice
                                    src/slice/iter/macros.rs: <core::slice::iter::Iter<T> as core::iter::traits::iterator::Iterator>::next
                           gimli-0.26.2/src/read/rnglists.rs: gimli::read::rnglists::RawRange::parse
                                        alloc/src/vec/mod.rs: <alloc::vec::Vec<T> as core::iter::traits::collect::FromIterator<T>>::from_iter
                                  library/alloc/src/alloc.rs: <alloc::alloc::Global as core::alloc::Allocator>::allocate
                                   core/src/ptr/const_ptr.rs: core::ptr::const_ptr::<impl *const T>::is_null
                      gimli-0.26.2/src/read/endian_reader.rs: <gimli::read::endian_reader::EndianReader<Endian,T> as gimli::read::reader::Reader>::len
                                           std/src/io/mod.rs: std::io::BufRead::read_line
                                         core/src/ptr/mod.rs: core::ptr::slice_from_raw_parts
                                    core/src/alloc/layout.rs: core::alloc::layout::Layout::array
                                     core/src/slice/index.rs: <core::ops::range::RangeTo<usize> as core::slice::index::SliceIndex<[T]>>::index_mut
                            gimli-0.26.2/src/read/aranges.rs: gimli::read::aranges::ArangeHeaderIter<R>::next
                             gimli-0.26.2/src/read/abbrev.rs: gimli::read::abbrev::get_attribute_size
                                       core/src/slice/raw.rs: core::slice::raw::from_raw_parts_mut
                               gimli-0.26.2/src/read/unit.rs: gimli::read::unit::skip_attributes
                                  library/core/src/result.rs: <core::result::Result<T,E> as core::ops::try_trait::Try>::branch
                                    core/src/ptr/metadata.rs: core::ptr::metadata::from_raw_parts_mut
                               gimli-0.26.2/src/read/unit.rs: gimli::read::unit::EntriesRaw<R>::read_abbreviation
                                    core/src/ptr/metadata.rs: core::ptr::metadata::from_raw_parts
                                    src/slice/iter/macros.rs: <core::slice::iter::Iter<T> as core::iter::traits::iterator::Iterator>::next
                                       core/src/slice/mod.rs: core::slice::<impl [T]>::get_unchecked
                                    core_arch/src/x86/avx.rs: core::core_arch::x86::avx::_mm256_load_si256
                                       core/src/slice/mod.rs: core::slice::<impl [T]>::get_unchecked
                                        core/src/str/iter.rs: <core::str::iter::Chars as core::iter::traits::double_ended::DoubleEndedIterator>::next_back
    github.com-1ecc6299db9ec823/lock_api-0.4.9/src/rwlock.rs: <<lock_api::rwlock::RwLock<R,T> as core::fmt::Debug>::fmt::LockedPlaceholder as core::fmt::Debug>::fmt
                                         core/src/ptr/mod.rs: core::ptr::drop_in_place<alloc::vec::Vec<u8>>
                                     core/src/ptr/mut_ptr.rs: core::ptr::mut_ptr::<impl *mut T>::is_null
                                   core/src/ptr/const_ptr.rs: core::ptr::const_ptr::<impl *const T>::copy_to_nonoverlapping
                               gimli-0.26.2/src/read/line.rs: gimli::read::line::LineRow::execute
                                    core/src/str/converts.rs: core::str::converts::from_utf8
                                    core/src/num/wrapping.rs: <core::num::wrapping::Wrapping<u64> as core::ops::arith::AddAssign>::add_assign
                                    src/slice/iter/macros.rs: <core::slice::iter::Iter<T> as core::iter::traits::iterator::Iterator>::next
                                  library/core/src/result.rs: <core::result::Result<T,E> as core::ops::try_trait::Try>::branch
                                        alloc/src/vec/mod.rs: alloc::vec::Vec<T,A>::extend_desugared
                              src/iter/adapters/enumerate.rs: <core::iter::adapters::enumerate::Enumerate<I> as core::iter::traits::iterator::Iterator>::next
                                  library/core/src/result.rs: <core::result::Result<T,E> as core::ops::try_trait::Try>::branch
                                  library/core/src/option.rs: <core::option::Option<T> as core::ops::try_trait::Try>::branch
                                     core/src/ptr/mut_ptr.rs: core::ptr::mut_ptr::<impl *mut T>::guaranteed_eq

----

core/src/slice/index.rs: <core::ops::range::RangeFrom<usize> as core::slice::index::SliceIndex<[T]>>::index
                                      core/src/future/mod.rs: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
                           tokio-1.21.2/src/runtime/enter.rs: tokio::runtime::enter::Enter::block_on
                       runtime/scheduler/multi_thread/mod.rs: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
                             tokio-1.21.2/src/runtime/mod.rs: tokio::runtime::Runtime::block_on
                bpf-profiling/profile-bee/bin/profile-bee.rs: profile_bee::main
                                    core/src/ops/function.rs: core::ops::function::FnOnce::call_once
    github.com-1ecc6299db9ec823/lock_api-0.4.9/src/rwlock.rs: <<lock_api::rwlock::RwLock<R,T> as core::fmt::Debug>::fmt::LockedPlaceholder as core::fmt::Debug>::fmt
                                library/alloc/src/raw_vec.rs: alloc::raw_vec::RawVec<T,A>::with_capacity_in
                                   core/src/ptr/const_ptr.rs: core::ptr::const_ptr::<impl *const [T]>::as_ptr
                                library/alloc/src/raw_vec.rs: alloc::raw_vec::RawVec<T,A>::current_memory
                             src/collections/btree/search.rs: alloc::collections::btree::search::<impl alloc::collections::btree::node::NodeRef<BorrowType,K,V,Type>>::search_node
                                  library/alloc/src/boxed.rs: alloc::boxed::Box<T,A>::try_new_uninit_in
                                  library/core/src/result.rs: <core::result::Result<T,E> as core::ops::try_trait::Try>::branch
                                       core/src/slice/mod.rs: core::slice::<impl [T]>::len
                                     core/src/str/pattern.rs: <core::str::pattern::CharSearcher as core::str::pattern::Searcher>::next_match
                                       core/src/slice/mod.rs: core::slice::<impl [T]>::get
                              src/iter/adapters/enumerate.rs: <core::iter::adapters::enumerate::Enumerate<I> as core::iter::traits::iterator::Iterator>::next
                              src/iter/adapters/enumerate.rs: <core::iter::adapters::enumerate::Enumerate<I> as core::iter::traits::iterator::Iterator>::next
                                     core/src/slice/index.rs: <usize as core::slice::index::SliceIndex<[T]>>::get_unchecked
                                     core/src/ptr/mut_ptr.rs: core::ptr::mut_ptr::<impl *mut T>::is_null
                              src/iter/adapters/enumerate.rs: <core::iter::adapters::enumerate::Enumerate<I> as core::iter::traits::iterator::Iterator>::next
                              src/iter/adapters/enumerate.rs: <core::iter::adapters::enumerate::Enumerate<I> as core::iter::traits::iterator::Iterator>::next
                                  library/alloc/src/slice.rs: alloc::slice::<impl [T]>::to_vec
                                     core/src/slice/index.rs: <core::ops::range::Range<usize> as core::slice::index::SliceIndex<[T]>>::get
                                     library/core/src/cmp.rs: core::cmp::impls::<impl core::cmp::PartialEq<&B> for &A>::eq
                                  library/core/src/option.rs: core::option::Option<T>::ok_or
                                         core/src/str/mod.rs: core::str::<impl str>::get_unchecked
                              src/iter/adapters/enumerate.rs: <core::iter::adapters::enumerate::Enumerate<I> as core::iter::traits::iterator::Iterator>::next
                                        alloc/src/vec/mod.rs: alloc::vec::Vec<T,A>::set_len
                                     core/src/ptr/mut_ptr.rs: core::ptr::mut_ptr::<impl *mut T>::guaranteed_eq
                              src/iter/adapters/enumerate.rs: <core::iter::adapters::enumerate::Enumerate<I> as core::iter::traits::iterator::Iterator>::next
                                    core/src/char/methods.rs: core::char::methods::<impl char>::encode_utf8
          github.com-1ecc6299db9ec823/aya-0.11.0/src/util.rs: aya::util::parse_kernel_symbols
                                     core/src/slice/index.rs: <core::ops::range::Range<usize> as core::slice::index::SliceIndex<[T]>>::get_unchecked
                                     core/src/slice/index.rs: <core::ops::range::Range<usize> as core::slice::index::SliceIndex<[T]>>::get
                                        alloc/src/vec/mod.rs: <alloc::vec::Vec<T,A> as core::ops::index::Index<I>>::index
----
                                     core/src/ptr/mut_ptr.rs: core::ptr::mut_ptr::<impl *mut T>::guaranteed_eq
                                 core/src/num/uint_macros.rs: core::num::<impl usize>::checked_mul
                             src/collections/btree/search.rs: alloc::collections::btree::search::<impl alloc::collections::btree::node::NodeRef<BorrowType,K,V,Type>>::find_key_index
                                    src/slice/iter/macros.rs: <core::slice::iter::Iter<T> as core::iter::traits::iterator::Iterator>::next
                             gimli-0.26.2/src/read/reader.rs: gimli::read::reader::Reader::read_u8_array
                                   core/src/ptr/const_ptr.rs: core::ptr::const_ptr::<impl *const T>::is_null
                                library/alloc/src/raw_vec.rs: alloc::raw_vec::RawVec<T,A>::allocate_in
                             src/collections/btree/search.rs: alloc::collections::btree::search::<impl alloc::collections::btree::node::NodeRef<BorrowType,K,V,alloc::collections::btree::node::marker::LeafOrInterna
l>>::search_tree
                                     core/src/num/nonzero.rs: core::num::nonzero::NonZeroUsize::get
                                  library/core/src/option.rs: core::option::Option<T>::ok_or
                                    src/slice/iter/macros.rs: <core::slice::iter::Iter<T> as core::iter::traits::iterator::Iterator>::next
                               gimli-0.26.2/src/read/unit.rs: gimli::read::unit::EntriesRaw<R>::is_empty
                                     library/core/src/cmp.rs: core::cmp::impls::<impl core::cmp::Ord for u64>::cmp
                                         core/src/ptr/mod.rs: core::ptr::slice_from_raw_parts_mut
                                    core/src/ptr/metadata.rs: core::ptr::metadata::from_raw_parts
                                  library/core/src/option.rs: core::option::Option<T>::map
                                        alloc/src/vec/mod.rs: alloc::vec::Vec<T,A>::set_len
                                       core/src/slice/raw.rs: core::slice::raw::from_raw_parts
                                    core/src/ptr/non_null.rs: core::ptr::non_null::NonNull<[T]>::slice_from_raw_parts
     github.com-1ecc6299db9ec823/addr2line-0.18.0/src/lib.rs: addr2line::Lines::parse
                                         core/src/ptr/mod.rs: core::ptr::read
                                library/alloc/src/raw_vec.rs: alloc::raw_vec::RawVec<T,A>::allocate_in
                                  library/alloc/src/slice.rs: <T as alloc::slice::hack::ConvertVec>::to_vec
                                  library/core/src/option.rs: core::option::Option<T>::is_none
                      gimli-0.26.2/src/read/endian_reader.rs: <gimli::read::endian_reader::EndianReader<Endian,T> as gimli::read::reader::Reader>::read_slice
                                    core/src/ptr/metadata.rs: core::ptr::metadata::from_raw_parts
                                   core/src/ptr/const_ptr.rs: core::ptr::const_ptr::<impl *const [T]>::len
                              library/core/src/intrinsics.rs: core::intrinsics::copy_nonoverlapping
                      gimli-0.26.2/src/read/endian_reader.rs: gimli::read::endian_reader::SubRange<T>::read_slice
                                    core/src/ptr/non_null.rs: core::ptr::non_null::NonNull<T>::cast
                               gimli-0.26.2/src/read/unit.rs: gimli::read::unit::skip_attributes
                                  library/core/src/result.rs: <core::result::Result<T,E> as core::ops::try_trait::Try>::branch
                      gimli-0.26.2/src/read/endian_reader.rs: gimli::read::endian_reader::SubRange<T>::read_slice
                               gimli-0.26.2/src/read/unit.rs: gimli::read::unit::parse_attribute
      github.com-1ecc6299db9ec823/gimli-0.26.2/src/leb128.rs: gimli::leb128::read::u16
                      gimli-0.26.2/src/read/endian_reader.rs: <gimli::read::endian_reader::EndianReader<Endian,T> as gimli::read::reader::Reader>::read_slice
                                     core/src/num/nonzero.rs: core::num::nonzero::NonZeroUsize::new_unchecked
                                       core/src/slice/mod.rs: core::slice::<impl [T]>::get_unchecked_mut
                                  library/core/src/option.rs: <core::option::Option<T> as core::ops::try_trait::Try>::branch
                                    src/slice/iter/macros.rs: <core::slice::iter::Iter<T> as core::iter::traits::iterator::Iterator>::next
                                  library/core/src/borrow.rs: <T as core::borrow::Borrow<T>>::borrow
                             gimli-0.26.2/src/read/abbrev.rs: gimli::read::abbrev::get_attribute_size
                                  library/core/src/result.rs: <core::result::Result<T,E> as core::ops::try_trait::Try>::branch
                                   core/src/ptr/const_ptr.rs: core::ptr::const_ptr::<impl *const T>::guaranteed_eq
                                   core/src/ptr/const_ptr.rs: <core::ops::range::Range<usize> as core::slice::index::SliceIndex<[T]>>::get_unchecked
                                    src/slice/iter/macros.rs: <core::slice::iter::Iter<T> as core::iter::traits::iterator::Iterator>::next
                                   core_arch/src/x86/avx2.rs: core::core_arch::x86::avx2::_mm256_movemask_epi8
                                       src/memchr/x86/avx.rs: memchr::memchr::x86::avx::memchr::matched
                                       src/memchr/x86/avx.rs: memchr::memchr::x86::avx::memchr
                              memchr-2.5.0/src/memchr/mod.rs: memchr::memchr::memchr
                                    core_arch/src/x86/mod.rs: core::core_arch::x86::m256iExt::as_i8x32
                                   core_arch/src/x86/avx2.rs: core::core_arch::x86::avx2::_mm256_cmpeq_epi8
                                       src/memchr/x86/avx.rs: memchr::memchr::x86::avx::memchr
                             gimli-0.26.2/src/read/abbrev.rs: gimli::read::abbrev::get_attribute_size
                                  library/core/src/borrow.rs: <T as core::borrow::Borrow<T>>::borrow
                             gimli-0.26.2/src/read/reader.rs: gimli::read::reader::Reader::read_u8_array
    github.com-1ecc6299db9ec823/lock_api-0.4.9/src/rwlock.rs: <<lock_api::rwlock::RwLock<R,T> as core::fmt::Debug>::fmt::LockedPlaceholder as core::fmt::Debug>::fmt
                                    core/src/ptr/non_null.rs: core::ptr::non_null::NonNull<T>::as_ptr
                                  library/alloc/src/alloc.rs: alloc::alloc::Global::grow_impl
                                    core/src/ptr/non_null.rs: core::ptr::non_null::NonNull<T>::as_ptr
                                    core/src/ptr/non_null.rs: core::ptr::non_null::NonNull<T>::new
                              gimli-0.26.2/src/read/dwarf.rs: gimli::read::dwarf::Unit<R>::new
                                         core/src/ptr/mod.rs: core::ptr::drop_in_place<alloc::raw_vec::RawVec<u8>>
                                    core/src/ptr/non_null.rs: <core::ptr::non_null::NonNull<T> as core::convert::From<core::ptr::unique::Unique<T>>>::from
                                     library/core/src/cmp.rs: core::cmp::impls::<impl core::cmp::Ord for u64>::cmp
                                   core/src/ptr/const_ptr.rs: core::ptr::const_ptr::<impl *const T>::offset
                                   core/src/ptr/const_ptr.rs: core::ptr::const_ptr::<impl *const T>::add
                                       src/memchr/x86/avx.rs: memchr::memchr::x86::avx::memchr

Support processes running from containers
Reading files directly results in

Can't read "/usr/local/bin/node" Some(Os { code: 2, kind: NotFound, message: "No such file or directory" })

See https://jvns.ca/blog/2018/01/26/spy-container/ on using /proc/pid/root to map into a different namespace
opened by zz85 0

Consider tidying / stripping from function names

opened by zz85 0

Some resources for Off-CPU profiling
https://www.brendangregg.com/offcpuanalysis.html

Bcc tools

https://github.com/iovisor/bcc/blob/master/tools/funclatency.py

https://github.com/iovisor/bcc/blob/master/tools/offcputime.py https://github.com/iovisor/bcc/blob/master/tools/offcputime_example.txt

https://github.com/iovisor/bcc/blob/master/tools/offwaketime.py https://github.com/iovisor/bcc/blob/master/tools/offwaketime_example.txt
opened by zz85 0
Compute offsets correctly

This would cause user stacks to not display correctly if offsets are not calculated correct.

I'll need to dig a little deeper into whether that's because of .text sections or relocations.

This commit may be interesting to reference how offsets can be computed https://github.com/mstange/samply/commit/193e336a275b4e1c82b18572c94ce602f4c24b7f

opened by zz85 0