below is an interactive tool to view and record historical system data.

I noticed when running below on live mode a few interesting things. First there are roughly 3-4 clock_gettime syscalls per epoll_wait.

strace -f -p `pgrep below`
...
[pid 10545] epoll_wait(7, [], 3, 0)     = 0
[pid 10545] clock_gettime(CLOCK_MONOTONIC, {tv_sec=1030, tv_nsec=824761942}) = 0
[pid 10545] clock_gettime(CLOCK_MONOTONIC, {tv_sec=1030, tv_nsec=824806781}) = 0
[pid 10545] clock_gettime(CLOCK_MONOTONIC, {tv_sec=1030, tv_nsec=824850152}) = 0
[pid 10545] clock_gettime(CLOCK_MONOTONIC, {tv_sec=1030, tv_nsec=824894012}) = 0
[pid 10545] epoll_wait(7, [], 3, 0)     = 0
[pid 10545] clock_gettime(CLOCK_MONOTONIC, {tv_sec=1030, tv_nsec=824976495}) = 0
[pid 10545] clock_gettime(CLOCK_MONOTONIC, {tv_sec=1030, tv_nsec=825019028}) = 0
[pid 10545] clock_gettime(CLOCK_MONOTONIC, {tv_sec=1030, tv_nsec=825062260}) = 0
[pid 10545] clock_gettime(CLOCK_MONOTONIC, {tv_sec=1030, tv_nsec=825104165}) = 0
[pid 10545] epoll_wait(7, [], 3, 0)     = 0

A quick and dirty bpftrace estimate of syscalls/sec, shows ~20k syscalls/sec:

 bpftrace -e 'tracepoint:raw_syscalls:sys_enter /comm == "below"/ { @syscalls = count(); }
    interval:s:1 { print(@syscalls); clear(@syscalls); }'
Attaching 2 probes...
@syscalls: 20399
@syscalls: 19499
@syscalls: 18809
@syscalls: 19378
@syscalls: 27254
@syscalls: 20061
@syscalls: 19144
@syscalls: 18824

Compared to clock_gettime syscalls (~15k syscalls/sec):

bpftrace -e 'tracepoint:syscalls:sys_enter_clock_gettime /comm == "below"/ { @syscalls = count(); }
    interval:s:1 { print(@syscalls); clear(@syscalls); }'
Attaching 2 probes...
@syscalls: 15518
@syscalls: 15510
@syscalls: 17648
@syscalls: 14906
@syscalls: 15070
@syscalls: 15713

Not that there is anything wrong with running clock_gettime, but when I run top/htop at the same time as below I notice it takes anywhere in the range of ~7-12% CPU. From digging around the code I'm guessing the store crate is probably where many of these calls are coming from:

rg -i 'now\(' | cut -d ':' -f 1 | sort | uniq -c
      1 below/common/src/cliutil.rs
     32 below/common/src/dateutil.rs
      2 below/model/src/collector.rs
      7 below/src/main.rs
      4 below/src/test.rs
      1 below/store/src/advance.rs
      9 below/store/src/cursor.rs
     11 below/store/src/lib.rs
      1 below/view/src/lib.rs

Hello, I have compiled below on latest ubuntu 22.04 jammy.

Looks like dependency libbpf-sys v0.6.2+v0.6.1 has an issue.

  Compiling crossbeam-deque v0.8.1
   Compiling signal-hook-mio v0.2.3
   Compiling libbpf-sys v0.6.2+v0.6.1
   Compiling zstd-sys v1.5.0+zstd.1.4.9
   Compiling vsprintf v2.0.0
   Compiling console v0.15.0
   Compiling crossterm v0.19.0
   Compiling darling_core v0.10.2
   Compiling term v0.7.0
   Compiling crossterm v0.20.0
   Compiling num v0.3.1
error: failed to run custom build command for `libbpf-sys v0.6.2+v0.6.1`

Caused by:
  process didn't exit successfully: `/tmp/cargo-install9Iwpfi/release/build/libbpf-sys-d272316adf9e8884/build-script-build` (exit status: 101)
  --- stderr
  make: *** No targets specified and no makefile found.  Stop.
  thread 'main' panicked at 'pkg-config is required to compile libbpf-sys using the vendored copy of libbpf', /home/paran/.cargo/registry/src/github.com-1ecc6299db9ec823/libbpf-sys-0.6.2+v0.6.1/build.rs:81:13
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
error: failed to compile `below v0.5.0`, intermediate artifacts can be found at `/tmp/cargo-install9Iwpfi`
paran@paran-virtual-machine:~/below$ uname -a
Linux paran-virtual-machine 5.15.0-33-generic #34-Ubuntu SMP Wed May 18 13:34:26 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

What contribution can I make to help this issue?

Thanks. Paran Lee.

I installed below with dnf on Fedora 34, and I get this error when I try to start it:

❯❯❯ sudo below live
Aug 21 14:33:45.414 ERRO
----------------- Detected unclean exit ---------------------
Error Message: Invalid file format: "/proc/1/cgroup"
-------------------------------------------------------------

I also see the same error message over and over in journald.

In case it matters, before this I encountered the issue described in #8103 (I had disabled cgroup v2 to install docker) but I fixed it (at least I think I did) and now I get the error above instead.

I'm not sure if I can provide good reproducible steps, as this is a fairly old Fedora install that has been updated and tweaked over the years. But I'd appreciate any help diagnosing the issue further.

Building below on Ubuntu 20.04 (focal) as well as 22.04 (jammy) fails with:

> cargo build --release
warning: /export/scratch/elkner/build/below/below/store/Cargo.toml: version requirement `5.0.1+zstd.1.5.2` for dependency `zstd-safe` includes semver metadata which will be ignored, removing the metadata is recommended to avoid confusion
warning: /export/scratch/elkner/build/below/below/store/Cargo.toml: version requirement `0.11.1+zstd.1.5.2` for dependency `zstd` includes semver metadata which will be ignored, removing the metadata is recommended to avoid confusion
   Compiling below v0.6.2 (/export/scratch/elkner/build/below/below)
   Compiling below-store v0.6.2 (/export/scratch/elkner/build/below/below/store)
   Compiling below-render v0.6.2 (/export/scratch/elkner/build/below/below/render)
   Compiling below-view v0.6.2 (/export/scratch/elkner/build/below/below/view)
   Compiling below-dump v0.6.2 (/export/scratch/elkner/build/below/below/dump)
error: failed to run custom build command for `below v0.6.2 (/export/scratch/elkner/build/below/below)`

Caused by:
  process didn't exit successfully: `/export/scratch/elkner/build/below/target/release/build/below-7b509743ed83d54a/build-script-build` (exit status: 101)
  --- stderr
  Warning: unrecognized map: .maps
  Warning: unrecognized map: license
  thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Generate("Failed to generate skeleton for /tmp/.tmpmPLpsb/exitstat.o: Failed to spawn rustfmt")', below/build.rs:36:10
  stack backtrace:
     0:     0x556421e5de2c - std::backtrace_rs::backtrace::libunwind::trace::h76d34c316a8bb75b
                                 at /build/rustc-ZOqcvC/rustc-1.61.0+dfsg1/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5
     1:     0x556421e5de2c - std::backtrace_rs::backtrace::trace_unsynchronized::hb782d75313085067
                                 at /build/rustc-ZOqcvC/rustc-1.61.0+dfsg1/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
     2:     0x556421e5de2c - std::sys_common::backtrace::_print_fmt::hee022c31c5bcfcc1
                                 at /build/rustc-ZOqcvC/rustc-1.61.0+dfsg1/library/std/src/sys_common/backtrace.rs:66:5
     3:     0x556421e5de2c - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h054c846eb7f0e1fb
                                 at /build/rustc-ZOqcvC/rustc-1.61.0+dfsg1/library/std/src/sys_common/backtrace.rs:45:22
     4:     0x556421e7f80e - core::fmt::write::hcef9c23bf27d039c
                                 at /build/rustc-ZOqcvC/rustc-1.61.0+dfsg1/library/core/src/fmt/mod.rs:1194:17
     5:     0x556421e48041 - std::io::Write::write_fmt::h6f6e0727690d342c
                                 at /build/rustc-ZOqcvC/rustc-1.61.0+dfsg1/library/std/src/io/mod.rs:1655:15
     6:     0x556421e4ac45 - std::sys_common::backtrace::_print::h591ece144505a7d5
                                 at /build/rustc-ZOqcvC/rustc-1.61.0+dfsg1/library/std/src/sys_common/backtrace.rs:48:5
     7:     0x556421e4ac45 - std::sys_common::backtrace::print::hbdcd241dc6539260
                                 at /build/rustc-ZOqcvC/rustc-1.61.0+dfsg1/library/std/src/sys_common/backtrace.rs:35:9
     8:     0x556421e4ac45 - std::panicking::default_hook::{{closure}}::hc934c90a9f702809
                                 at /build/rustc-ZOqcvC/rustc-1.61.0+dfsg1/library/std/src/panicking.rs:295:22
     9:     0x556421e4a8f9 - std::panicking::default_hook::h984be39605172602
                                 at /build/rustc-ZOqcvC/rustc-1.61.0+dfsg1/library/std/src/panicking.rs:314:9
    10:     0x556421e4b28f - std::panicking::rust_panic_with_hook::h04be8cf992436914
                                 at /build/rustc-ZOqcvC/rustc-1.61.0+dfsg1/library/std/src/panicking.rs:698:17
    11:     0x556421e5e167 - std::panicking::begin_panic_handler::{{closure}}::h47ebda0479239463
                                 at /build/rustc-ZOqcvC/rustc-1.61.0+dfsg1/library/std/src/panicking.rs:588:13
    12:     0x556421e5df44 - std::sys_common::backtrace::__rust_end_short_backtrace::h1690aafc8ab8529a
                                 at /build/rustc-ZOqcvC/rustc-1.61.0+dfsg1/library/std/src/sys_common/backtrace.rs:138:18
    13:     0x556421e4ada2 - rust_begin_unwind
                                 at /build/rustc-ZOqcvC/rustc-1.61.0+dfsg1/library/std/src/panicking.rs:584:5
    14:     0x556421c609b3 - core::panicking::panic_fmt::h8531284c14f462dc
                                 at /build/rustc-ZOqcvC/rustc-1.61.0+dfsg1/library/core/src/panicking.rs:143:14
    15:     0x556421c60a43 - core::result::unwrap_failed::h6972a430d3981bcd
                                 at /build/rustc-ZOqcvC/rustc-1.61.0+dfsg1/library/core/src/result.rs:1785:5
    16:     0x556421c61912 - core::result::Result<T,E>::unwrap::hb6712249d2833527
    17:     0x556421c6151e - build_script_build::main::h444f38e0706d0359
    18:     0x556421c61b03 - core::ops::function::FnOnce::call_once::hcba44e2289eae45f
    19:     0x556421c616c9 - std::sys_common::backtrace::__rust_begin_short_backtrace::h0a511523a6f97bc6
    20:     0x556421c60d09 - std::rt::lang_start::{{closure}}::h045e314370c806a1
    21:     0x556421e45d81 - core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once::he815eb44dc8530a2
                                 at /build/rustc-ZOqcvC/rustc-1.61.0+dfsg1/library/core/src/ops/function.rs:259:13
    22:     0x556421e45d81 - std::panicking::try::do_call::h65588aa29098f6c1
                                 at /build/rustc-ZOqcvC/rustc-1.61.0+dfsg1/library/std/src/panicking.rs:492:40
    23:     0x556421e45d81 - std::panicking::try::haa10bca2f4952c44
                                 at /build/rustc-ZOqcvC/rustc-1.61.0+dfsg1/library/std/src/panicking.rs:456:19
    24:     0x556421e45d81 - std::panic::catch_unwind::h1d908ddce340475e
                                 at /build/rustc-ZOqcvC/rustc-1.61.0+dfsg1/library/std/src/panic.rs:137:14
    25:     0x556421e45d81 - std::rt::lang_start_internal::{{closure}}::hdf74d953ee81ad25
                                 at /build/rustc-ZOqcvC/rustc-1.61.0+dfsg1/library/std/src/rt.rs:128:48
    26:     0x556421e45d81 - std::panicking::try::do_call::hac535a1b96e92d37
                                 at /build/rustc-ZOqcvC/rustc-1.61.0+dfsg1/library/std/src/panicking.rs:492:40
    27:     0x556421e45d81 - std::panicking::try::hcc24ddbc174e7e92
                                 at /build/rustc-ZOqcvC/rustc-1.61.0+dfsg1/library/std/src/panicking.rs:456:19
    28:     0x556421e45d81 - std::panic::catch_unwind::h4b14c208c01d9f23
                                 at /build/rustc-ZOqcvC/rustc-1.61.0+dfsg1/library/std/src/panic.rs:137:14
    29:     0x556421e45d81 - std::rt::lang_start_internal::heef479cc3aa3b396
                                 at /build/rustc-ZOqcvC/rustc-1.61.0+dfsg1/library/std/src/rt.rs:128:20
    30:     0x556421c60cf1 - std::rt::lang_start::had5835b9e151f469
    31:     0x556421c615d3 - main
    32:     0x7f9047436d90 - <unknown>
    33:     0x7f9047436e40 - __libc_start_main
    34:     0x556421c60c05 - _start
    35:                0x0 - <unknown>
warning: build failed, waiting for other jobs to finish...
warning: unused `Result` that must be used
   --> below/view/src/lib.rs:224:13
    |
224 | /             screens
225 | |                 .get_mut()
226 | |                 .screen_mut()
227 | |                 .unwrap()
228 | |                 .take_focus(cursive::direction::Direction::none());
    | |___________________________________________________________________^
    |
    = note: `#[warn(unused_must_use)]` on by default
    = note: this `Result` may be an `Err` variant, which should be handled

warning: `below-view` (lib) generated 1 warning
    Building [=======================> ] 333/335: below-store                 
142.436u 2.734s 0:54.00 268.8%	0+0k 17758+55902io 6034pf+0w

# .cargo/bin/below --version
below 0.5.0
# .cargo/bin/below
Apr 08 14:22:16.594 ERRO
----------------- Detected unclean exit ---------------------
Error Message: Os { code: 95, kind: Uncategorized, message: "Operation not supported" }: "cpu.pressure": Operation not supported (os error 95)
-------------------------------------------------------------
# cat /sys/fs/cgroup/cpu.pressure
cat: /sys/fs/cgroup/cpu.pressure: Operation not supported

os info: centos 9 stream 5.14.0-71.el9.x86_64 CONFIG_PSI_DEFAULT_DISABLED=y

Update outdated Cargo.lock

$ cargo build --locked
    Updating crates.io index
    Updating git repository `https://github.com/facebookexperimental/rust-shed.git`
error: the lock file below/Cargo.lock needs to be updated but --locked was passed to prevent this
If you want to try to generate the lock file without accessing the network, remove the --locked flag and use --offline instead.

Remove semver build metadata from Cargo.toml

$ cargo build
warning: below/below/store/Cargo.toml: version requirement `5.0.1+zstd.1.5.2` for dependency `zstd-safe` includes semver metadata which will be ignored, removing the metadata is recommended to avoid confusion
warning: below/below/store/Cargo.toml: version requirement `0.11.1+zstd.1.5.2` for dependency `zstd` includes semver metadata which will be ignored, removing the metadata is recommended to avoid confusion

Require Cargo.lock is up to date

Ensure that Cargo.lock is always up to date when building and that each time the same versions of the crates are used (reproducible builds).

Please always update and commit Cargo.lock after changing any Cargo.toml (cargo does it automatically when running cargo build and similar).

Summary: Add open source shim for gpu_stats so that we can remove some #[allow_unused] and #[cfg(fbcode_build)]. Those for view (UI) are a bit coupled. Tackling them will lead to a blank page for GPU until it's implemented. Therefore will leave that for the future.

Differential Revision: D36828570

Summary: Pull Request resolved: https://github.com/facebookexperimental/eden/pull/76

Use released version to fix cursive_core version conflicts.

Differential Revision: D27032206

I have a Dockerfile that I use for trying to build below located here https://gist.github.com/dc510ccd122e6c13d6271833dd1cbcee I keep getting the following error while trying to build the HEAD , the tag v0.5.0 builds just fine.

#15 105.6    Compiling num_enum_derive v0.5.4
#15 105.7    Compiling cursive v0.16.3
#15 107.1    Compiling cursive_buffered_backend v0.5.0
#15 107.7    Compiling below-common v0.5.0 (/src/below/below/common)
#15 107.8    Compiling num_enum v0.5.4
#15 108.2    Compiling btrfs v0.5.0 (/src/below/below/btrfs)
#15 108.4    Compiling below-gpu-stats v0.5.0 (/src/below/below/gpu_stats)
#15 108.7 error: `#[derive]` can't be used on a `#[repr(packed)]` struct that does not derive Copy (error E0133)
#15 108.7     --> below/btrfs/src/btrfs_api/open_source/btrfs_sys.rs:4997:10
#15 108.7      |
#15 108.7 4997 | #[derive(Debug)]
#15 108.7      |          ^^^^^
#15 108.7      |
#15 108.7      = note: `#[deny(unaligned_references)]` on by default
#15 108.7      = warning: this was previously accepted by the compiler but is being phased out; it will become a hard error in a future release!
#15 108.7      = note: for more information, see issue #82523 <https://github.com/rust-lang/rust/issues/82523>
#15 108.7      = note: this error originates in the derive macro `Debug` (in Nightly builds, run with -Z macro-backtrace for more info)
#15 108.7 
#15 108.8 error: could not compile `btrfs` due to 2 previous errors
#15 108.8 warning: build failed, waiting for other jobs to finish...
------
executor failed running [/bin/sh -c /root/.cargo/bin/cargo build --release --all-targets]: exit code: 101
make: *** [build] Error 1
(ins)lrfurtado-mbp:below_build lrfurtado$ 

When reading cgroup membership, we currently assume that the cgroup v2 line will be first in /proc/[pid]/cgroup. This is not necessarily the case. Instead let's take the first line that starts with "0::".

Tested on ubuntu where

$ cat /proc/1/cgroup
12:blkio:/init.scope
11:pids:/init.scope
8:memory:/init.scope
7:freezer:/
4:devices:/init.scope
2:cpu,cpuacct:/init.scope
1:name=systemd:/init.scope
0::/init.scope

This should fix #8105.

Summary: below samples have seconds granularity, which is enforced at the index entry timestamp. If two samples are recorded at the same second, we can't tell them apart. Zero-second collection interval will also mess up rate calculations. Let's make sure every two samples are at least 1s apart.

Reviewed By: boyuni

Differential Revision: D29642266

This should fix the warning mentioned in 8168. See discussion in Gentoo 28643 for more details.

Signed-off-by: Daniel Hodges [email protected]

Usually kernel threads are just clutter in the process view: not much one can do about it, not of interest because resource usage is always 0 no matter, which one.

So it makes sense to hide them per default (and optionally add a CLI option or key binding to show them on demand).

To find out, whether a process is a kernel_thread, one may inspect field 9 (flags) of /proc/$pid/stat - if the values bit 0x00200000 is set, it is a kernel thread.

systemd-cgls e.g. does it this way: https://github.com/jelmd/cgls/blob/main/misc.c#L911

If one installs below with default settings, its use is really annoying because one always needs to specify --config myconf.

It would be nice, if below would try per default to load ${HOME}/.below.conf (ignore if n/a), than supplement all unset values with the system config /etc/below.conf and finally set all unset values to their fallback values.

''' Compiling below v0.5.0 (/home/bhzhu/source/below/below) error: failed to run custom build command for below v0.5.0 (/home/bhzhu/source/below/below)

Caused by: process didn't exit successfully: /home/bhzhu/source/below/target/release/build/below-718216cef9c31e46/build-script-build (exit status: 101) --- stderr thread 'main' panicked at 'called Result::unwrap() on an Err value: Build("Failed to run regex on version string")', below/build.rs:19:47 stack backtrace: 0: 0x5555557ee5ec - std::backtrace_rs::backtrace::libunwind::trace::h26ded3cee286f9ec at /rustc/09cb29c64c2a0e15debf2d6fca2bc7c71a682033/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5 1: 0x5555557ee5ec - std::backtrace_rs::backtrace::trace_unsynchronized::h830d548378790fdc at /rustc/09cb29c64c2a0e15debf2d6fca2bc7c71a682033/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5 2: 0x5555557ee5ec - std::sys_common::backtrace::_print_fmt::h7356e27371ec940d at /rustc/09cb29c64c2a0e15debf2d6fca2bc7c71a682033/library/std/src/sys_common/backtrace.rs:66:5 3: 0x5555557ee5ec - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h3da18fd93237a444 at /rustc/09cb29c64c2a0e15debf2d6fca2bc7c71a682033/library/std/src/sys_common/backtrace.rs:45:22 4: 0x555555810c3c - core::fmt::write::h1b11e7c3b9f52b83 at /rustc/09cb29c64c2a0e15debf2d6fca2bc7c71a682033/library/core/src/fmt/mod.rs:1190:17 5: 0x5555557ea318 - std::io::Write::write_fmt::h2530d62e754fd8dc at /rustc/09cb29c64c2a0e15debf2d6fca2bc7c71a682033/library/std/src/io/mod.rs:1657:15 6: 0x5555557f0657 - std::sys_common::backtrace::_print::ha001eea1f09df837 at /rustc/09cb29c64c2a0e15debf2d6fca2bc7c71a682033/library/std/src/sys_common/backtrace.rs:48:5 7: 0x5555557f0657 - std::sys_common::backtrace::print::he19aad5eb563cdb7 at /rustc/09cb29c64c2a0e15debf2d6fca2bc7c71a682033/library/std/src/sys_common/backtrace.rs:35:9 8: 0x5555557f0657 - std::panicking::default_hook::{{closure}}::h4bec003d220dffc8 at /rustc/09cb29c64c2a0e15debf2d6fca2bc7c71a682033/library/std/src/panicking.rs:295:22 9: 0x5555557f031f - std::panicking::default_hook::hbf9de44b5ceb445d at /rustc/09cb29c64c2a0e15debf2d6fca2bc7c71a682033/library/std/src/panicking.rs:314:9 10: 0x5555557f0dba - std::panicking::rust_panic_with_hook::hba49b95df3772ed5 at /rustc/09cb29c64c2a0e15debf2d6fca2bc7c71a682033/library/std/src/panicking.rs:698:17 11: 0x5555557f0aa7 - std::panicking::begin_panic_handler::{{closure}}::he9fc0b4286f77c60 at /rustc/09cb29c64c2a0e15debf2d6fca2bc7c71a682033/library/std/src/panicking.rs:588:13 12: 0x5555557eea94 - std::sys_common::backtrace::__rust_end_short_backtrace::h07791585b84f81da at /rustc/09cb29c64c2a0e15debf2d6fca2bc7c71a682033/library/std/src/sys_common/backtrace.rs:138:18 13: 0x5555557f07a9 - rust_begin_unwind at /rustc/09cb29c64c2a0e15debf2d6fca2bc7c71a682033/library/std/src/panicking.rs:584:5 14: 0x5555555b9c13 - core::panicking::panic_fmt::he8540b011fb85aaa at /rustc/09cb29c64c2a0e15debf2d6fca2bc7c71a682033/library/core/src/panicking.rs:143:14 15: 0x5555555b9ca3 - core::result::unwrap_failed::h5b5a61b4f1a3fe39 at /rustc/09cb29c64c2a0e15debf2d6fca2bc7c71a682033/library/core/src/result.rs:1749:5 16: 0x5555555babe2 - core::result::Result<T,E>::unwrap::hbd8ff0f689370095 17: 0x5555555bcb7a - build_script_build::main::h64c087248aa8d4af 18: 0x5555555bb2c3 - core::ops::function::FnOnce::call_once::h3fe4fb01b7d47e7e 19: 0x5555555bb509 - std::sys_common::backtrace::__rust_begin_short_backtrace::h4b18e62f18ccc92a 20: 0x5555555bcac9 - std::rt::lang_start::{{closure}}::hf32d562c9a5ca0c2 21: 0x5555557edcd1 - core::ops::function::impls::<impl core::ops::function::FnOnce for &F>::call_once::hb16268c89c211835 at /rustc/09cb29c64c2a0e15debf2d6fca2bc7c71a682033/library/core/src/ops/function.rs:259:13 22: 0x5555557edcd1 - std::panicking::try::do_call::h42af4c24be01d864 at /rustc/09cb29c64c2a0e15debf2d6fca2bc7c71a682033/library/std/src/panicking.rs:492:40 23: 0x5555557edcd1 - std::panicking::try::hcddad39c2668977b at /rustc/09cb29c64c2a0e15debf2d6fca2bc7c71a682033/library/std/src/panicking.rs:456:19 24: 0x5555557edcd1 - std::panic::catch_unwind::h17be4d22360f6801 at /rustc/09cb29c64c2a0e15debf2d6fca2bc7c71a682033/library/std/src/panic.rs:137:14 25: 0x5555557edcd1 - std::rt::lang_start_internal::{{closure}}::he399a7bc5b46d1ed at /rustc/09cb29c64c2a0e15debf2d6fca2bc7c71a682033/library/std/src/rt.rs:128:48 26: 0x5555557edcd1 - std::panicking::try::do_call::he22e88faa301e128 at /rustc/09cb29c64c2a0e15debf2d6fca2bc7c71a682033/library/std/src/panicking.rs:492:40 27: 0x5555557edcd1 - std::panicking::try::h179e28dc63c29e66 at /rustc/09cb29c64c2a0e15debf2d6fca2bc7c71a682033/library/std/src/panicking.rs:456:19 28: 0x5555557edcd1 - std::panic::catch_unwind::h926c540a0059105b at /rustc/09cb29c64c2a0e15debf2d6fca2bc7c71a682033/library/std/src/panic.rs:137:14 29: 0x5555557edcd1 - std::rt::lang_start_internal::hcf3b4e612381f709 at /rustc/09cb29c64c2a0e15debf2d6fca2bc7c71a682033/library/std/src/rt.rs:128:20 30: 0x5555555bcab1 - std::rt::lang_start::hfb5e709917bc1f4e 31: 0x5555555bcc43 - main 32: 0x7ffff6f1f50d - __libc_start_main at /home/bhzhu/source/glibc-2.33/csu/../csu/libc-start.c:332:16 33: 0x5555555ba43a - _start at /home/bhzhu/source/glibc-2.33/csu/../sysdeps/x86_64/start.S:120 34: 0x0 - warning: build failed, waiting for other jobs to finish... error: build failed '''