Hello,

First, thx for this amazing tools. Thanks to it I was able to clean my computer from a lot of bull***

I open this issue on a subject i don't rely understand by I hope it can help.

Thx to bsd/apple clone things, i am able to clone dir/ file with cp -c .

Clone are, as understand it, a kind of weird hard link but when you write over it, it save the diff.

In my example, you can see that I clone the dankest movie of my library few time and df -h doesn't report a disk usage difference.

dust repeat the clones as there take more space on disk.

[Movies] df -h
Filesystem      Size   Used  Avail Capacity     iused      ifree %iused  Mounted on
/dev/disk1s1   466Gi   10Gi   90Gi    11%      484283 4881968597    0%   /
devfs          403Ki  403Ki    0Bi   100%        1404          0  100%   /dev
/dev/disk1s2   466Gi  352Gi   90Gi    80%     3172969 4879279911    0%   /System/Volumes/Data
/dev/disk1s5   466Gi   12Gi   90Gi    12%          12 4882452868    0%   /private/var/vm
map auto_home    0Bi    0Bi    0Bi   100%           0          0  100%   /System/Volumes/Data/home
/dev/disk2s2   105Mi  105Mi    0Bi   100%           3 4294967276    0%   /Volumes/Install Google Drive File Stream
drivefs         30Gi  7.0Gi   23Gi    24% 18446744069414596880 4294967295 146880675765702656%   /Volumes/GoogleDrive
drivefs         30Gi  7.0Gi   23Gi    24% 18446744069414740697 4294967295 11796403584574832%   /Volumes/GoogleDrive
[Movies] cp -cR Sonic.the.Hedgehog.2020.720p.HDRip.XviD.MP3-STUTTERSHIT clone1
[Movies] cp -cR Sonic.the.Hedgehog.2020.720p.HDRip.XviD.MP3-STUTTERSHIT clone2
[Movies] cp -cR Sonic.the.Hedgehog.2020.720p.HDRip.XviD.MP3-STUTTERSHIT clone3
[Movies] cp -cR Sonic.the.Hedgehog.2020.720p.HDRip.XviD.MP3-STUTTERSHIT clone4
[Movies] cp -cR Sonic.the.Hedgehog.2020.720p.HDRip.XviD.MP3-STUTTERSHIT clone5
[Movies] cp -cR Sonic.the.Hedgehog.2020.720p.HDRip.XviD.MP3-STUTTERSHIT clone6
[Movies] df -h
Filesystem      Size   Used  Avail Capacity     iused      ifree %iused  Mounted on
/dev/disk1s1   466Gi   10Gi   90Gi    11%      484283 4881968597    0%   /
devfs          403Ki  403Ki    0Bi   100%        1404          0  100%   /dev
/dev/disk1s2   466Gi  352Gi   90Gi    80%     3172987 4879279893    0%   /System/Volumes/Data
/dev/disk1s5   466Gi   12Gi   90Gi    12%          12 4882452868    0%   /private/var/vm
map auto_home    0Bi    0Bi    0Bi   100%           0          0  100%   /System/Volumes/Data/home
/dev/disk2s2   105Mi  105Mi    0Bi   100%           3 4294967276    0%   /Volumes/Install Google Drive File Stream
drivefs         30Gi  7.0Gi   23Gi    24% 18446744069414596880 4294967295 146880675765702656%   /Volumes/GoogleDrive
drivefs         30Gi  7.0Gi   23Gi    24% 18446744069414740697 4294967295 11796403584574832%   /Volumes/GoogleDrive
[Movies] dust
  46G ─┬ .
  24G  ├─┬ Star.Wars.The.Clone.Wars.S01.1080p.BluRay.x264-FLHD[rartv]
 1.1G  │ ├── Star.Wars.The.Clone.Wars.S01E22.1080p.BluRay.x264-FLHD.mkv
 1.1G  │ ├── Star.Wars.The.Clone.Wars.S01E20.1080p.BluRay.x264-FLHD.mkv
 1.1G  │ └── Star.Wars.The.Clone.Wars.S01E16.1080p.BluRay.x264-FLHD.mkv
 2.9G  ├─┬ Sonic.the.Hedgehog.2020.720p.HDRip.XviD.MP3-STUTTERSHIT
 2.9G  │ └── Sonic.the.Hedgehog.2020.720p.HDRip.XviD.MP3-STUTTERSHIT.avi
 2.9G  ├─┬ clone1
 2.9G  │ └── Sonic.the.Hedgehog.2020.720p.HDRip.XviD.MP3-STUTTERSHIT.avi
 2.9G  ├─┬ clone2
 2.9G  │ └── Sonic.the.Hedgehog.2020.720p.HDRip.XviD.MP3-STUTTERSHIT.avi
 2.9G  ├─┬ clone3
 2.9G  │ └── Sonic.the.Hedgehog.2020.720p.HDRip.XviD.MP3-STUTTERSHIT.avi
 2.9G  ├─┬ clone4
 2.9G  │ └── Sonic.the.Hedgehog.2020.720p.HDRip.XviD.MP3-STUTTERSHIT.avi
 2.9G  ├─┬ clone5
 2.9G  │ └── Sonic.the.Hedgehog.2020.720p.HDRip.XviD.MP3-STUTTERSHIT.avi
 2.9G  ├─┬ clone6
 2.9G  │ └── Sonic.the.Hedgehog.2020.720p.HDRip.XviD.MP3-STUTTERSHIT.avi
 1.4G  └─┬ Zero.Dark.Thirty.2012.1080p.BluRay.x265.10bit-z97
 1.4G    └── Zero.Dark.Thirty.2012.1080p.BluRay.x265.10bit-z97.mkv

I never ever developed in rust except 10 minute ago while try to see if metadata and filetype structure could help us here. seems not.

Actually, I have no clue to know if a file is a clone or not : (https://stackoverflow.com/questions/46417747/apple-file-system-apfs-check-if-file-is-a-clone-on-terminal-shell)

Would be glad to help you if you don't have any OSx to try thing out, but I don't think I would be able to PR anything.

Best Regards,

I cloned this repo to my machine and run cargo test, it gives the following errors:

warning: unused import: `std::panic`
 --> tests/tests_symlinks.rs:5:5
  |
5 | use std::panic;
  |     ^^^^^^^^^^
  |
  = note: `#[warn(unused_imports)]` on by default

warning: 1 warning emitted

warning: unused import: `std::panic`
 --> tests/tests_symlinks.rs:5:5
  |
5 | use std::panic;
  |     ^^^^^^^^^^
  |
  = note: `#[warn(unused_imports)]` on by default

warning: 1 warning emitted

    Finished test [unoptimized + debuginfo] target(s) in 0.03s
     Running target/debug/deps/dust-b9c46c4d4c140db8

running 13 tests
test display::tests::test_human_readable_number ... ok
test utils::tests::test_is_a_parent_of ... ok
test utils::tests::test_is_a_parent_of_root ... ok
test utils::tests::test_should_ignore_file ... ok
test utils::tests::test_should_ignore_file_on_different_device ... ok
test utils::tests::test_simplify_dir ... ok
test utils::tests::test_simplify_dir_dots ... ok
test utils::tests::test_simplify_dir_duplicates ... ok
test utils::tests::test_simplify_dir_rm_subdir ... ok
test utils::tests::test_simplify_dir_substring_names ... ok
test utils::tests::test_simplify_dir_rm_subdir_and_not_substrings ... ok
test display::tests::test_format_str_long_name ... ok
test display::tests::test_format_str ... ok

test result: ok. 13 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running target/debug/deps/integration-e9ab9987cbc75a52

running 0 tests

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

     Running target/debug/deps/test_exact_output-f8b1b747f15bb498

running 9 tests
test test_apparent_size ... FAILED
test test_main_basic ... FAILED
test test_main_multi_arg ... FAILED
test test_main_long_paths ... FAILED
test test_unicode_directories ... FAILED
test tests_symlinks::test_hard_sym_link ... ok
test tests_symlinks::test_recursive_sym_link ... ok
test test_substring_of_names_and_long_names ... FAILED
test tests_symlinks::test_soft_sym_link ... ok

failures:

---- test_apparent_size stdout ----
thread 'test_apparent_size' panicked at 'assertion failed: output.contains(&output_apparent_size())', tests/test_exact_output.rs:158:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- test_main_basic stdout ----
thread 'test_main_basic' panicked at 'assertion failed: output.contains(&main_output())', tests/test_exact_output.rs:57:5

---- test_main_multi_arg stdout ----
thread 'test_main_multi_arg' panicked at 'assertion failed: output.contains(&main_output())', tests/test_exact_output.rs:73:5

---- test_main_long_paths stdout ----
"0B     ┌── /tmp/test_dir/many/a_file    │░░░░░░░░░░░░░░░░░░░░░░░░░░░░█ │   0%\n 4.0K     ├── /tmp/test_dir/many/hello_file│█████████████████████████████ │ 100%\n 4.0K   ┌─┴ /tmp/test_dir/many             │█████████████████████████████ │ 100%\n 4.0K ┌─┴ /tmp/test_dir                    │█████████████████████████████ │ 100%"
"0B     ┌── /tmp/test_dir/many/a_file    │         ░░░░░░░░░░░░░░░░░░░█ │   0%\n 4.0K     ├── /tmp/test_dir/many/hello_file│         ░░░░░░░░░░██████████ │  33%\n 8.0K   ┌─┴ /tmp/test_dir/many             │         ████████████████████ │  67%\n  12K ┌─┴ /tmp/test_dir                    │█████████████████████████████ │ 100%"
thread 'test_main_long_paths' panicked at 'assertion failed: output.contains(&main_output_long_paths())', tests/test_exact_output.rs:119:5

---- test_unicode_directories stdout ----
thread 'test_unicode_directories' panicked at 'assertion failed: output.contains(&unicode_dir())', tests/test_exact_output.rs:243:5

---- test_substring_of_names_and_long_names stdout ----
thread 'test_substring_of_names_and_long_names' panicked at 'assertion failed: output.contains(&no_substring_of_names_output())', tests/test_exact_output.rs:198:5


failures:
    test_apparent_size
    test_main_basic
    test_main_long_paths
    test_main_multi_arg
    test_substring_of_names_and_long_names
    test_unicode_directories

test result: FAILED. 3 passed; 6 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.04s

error: test failed, to rerun pass '--test test_exact_output'

Commit: 1b3d0b272401b2eaa196891766d03a7667ea6917. OS: Arch Linux Rust: 1.50.0

Is cargo test the correct way to test dust? If not, then it should be documented.

$ export RUST_BACKTRACE=full

$ dust
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ThreadPoolBuildError { kind: IOError(Os { code: 11, kind: WouldBlock, message: "Try again" }) }', src/main.rs:161:10
stack backtrace:
   0: 0xe7c35e34 - <unknown>
   1: 0xe7bd64be - <unknown>
   2: 0xe7c1f300 - <unknown>
   3: 0xe7c36836 - <unknown>
   4: 0xe7c3709c - <unknown>
   5: 0xe7c36e2a - <unknown>
   6: 0xe7c36df6 - <unknown>
   7: 0xe7c36dcc - <unknown>
   8: 0xe7bd4e58 - <unknown>
   9: 0xe7bd9ea6 - <unknown>
  10: 0xe7ba09cc - <unknown>
  11: 0xe7b87540 - <unknown>
  12: 0xe7ba13b6 - <unknown>
  13: 0xe78be0b6 - <unknown>

$ dust -V
Dust 0.8.2

this started happening after the update installed dust from my package manager on termux

It does not take $COLUMNS into account and stuff gets wrapped being hardly readable under terminal less than 100 width. I have no issue with dust shortening the file name if the terminal is short.

1005M   │ ┌─┴ iams-dev-environment

 3.1G   ├─┴ hlai

 616M   │ ┌── student

 994M   │ │ ┌── node_modules

 1.4G   │ ├─┴ apspace

 608M   │ │   ┌── objects

 691M   │ │ ┌─┴ .git

 2.1G   │ │ │   ┌── Components

 2.3G   │ │ │ ┌─┴ stage

 2.3G   │ │ ├─┴ database

 3.6G   │ ├─┴ attendix

 6.7G   ├─┴ apu

I think at the very least, it should support terminal with 80 widths with a fallback even if stuff does not display well, without the terminal wrapped stuff up, it will still look pretty neat.

v0.4.3 builds and installs, but the changes in bdc3d40 break windows compilation (even when using 'nightly').

I just noticed when updating to the most recent version (v0.4.4).

I've looked through the code and it looks like using the unstable metadata.file_index() and metadata.volume_serial_number() as well as missing get_filesystem() are the culprits.

Here's the build output:

C:>cargo +nightly build
   Compiling du-dust v0.4.3 (C:\Users\Roy\OneDrive\Projects\rust\rs.dust)
error[E0425]: cannot find function `get_filesystem` in this scope
   --> src\utils\mod.rs:102:26
    |
102 |         if let Some(a) = get_filesystem(file_name) {
    |                          ^^^^^^^^^^^^^^ not found in this scope

error[E0658]: use of unstable library feature 'windows_by_handle'
  --> src\utils\platform.rs:28:52
   |
28 |         let windows_equivalent_of_inode = Some((md.file_index(), md.volume_serial_number()));
   |                                                    ^^^^^^^^^^
   |
   = note: for more information, see https://github.com/rust-lang/rust/issues/63010
   = help: add `#![feature(windows_by_handle)]` to the crate attributes to enable

error[E0658]: use of unstable library feature 'windows_by_handle'
  --> src\utils\platform.rs:28:69
   |
28 |         let windows_equivalent_of_inode = Some((md.file_index(), md.volume_serial_number()));
   |                                                                     ^^^^^^^^^^^^^^^^^^^^
   |
   = note: for more information, see https://github.com/rust-lang/rust/issues/63010
   = help: add `#![feature(windows_by_handle)]` to the crate attributes to enable

error[E0308]: mismatched types
  --> src\utils\platform.rs:27:5
   |
25 |   pub fn get_metadata(d: &DirEntry, use_apparent_size: bool) -> Option<(u64, Option<(u64, u64)>)> {
   |                                                                 --------------------------------- expected `std::option::Option<(u64, std::option::Option<(u64, u64)>)>` because of return type
26 |       use std::os::windows::fs::MetadataExt;
27 | /     d.metadata.as_ref().unwrap().as_ref().ok().map(|md| {
28 | |         let windows_equivalent_of_inode = Some((md.file_index(), md.volume_serial_number()));
29 | |         (md.file_size(), windows_equivalent_of_inode)
30 | |     })
   | |______^ expected `u64`, found enum `std::option::Option`
   |
   = note: expected enum `std::option::Option<(_, std::option::Option<(u64, u64)>)>`
              found enum `std::option::Option<(_, std::option::Option<(std::option::Option<u64>, std::option::Option<u32>)>)>`

error[E0658]: use of unstable library feature 'windows_by_handle'
  --> src\utils\platform.rs:54:19
   |
54 |     Some(metadata.volume_serial_number())
   |                   ^^^^^^^^^^^^^^^^^^^^
   |
   = note: for more information, see https://github.com/rust-lang/rust/issues/63010
   = help: add `#![feature(windows_by_handle)]` to the crate attributes to enable

error[E0308]: mismatched types
  --> src\utils\platform.rs:54:10
   |
54 |     Some(metadata.volume_serial_number())
   |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `u64`, found enum `std::option::Option`
   |
   = note: expected type `u64`
              found enum `std::option::Option<u32>`

error: aborting due to 6 previous errors

Some errors have detailed explanations: E0308, E0425, E0658.
For more information about an error, try `rustc --explain E0308`.
error: could not compile `du-dust`.

To learn more, run the command again with --verbose.

Do you know how I could fix this in CentOS?

[jalal@goku dust-0.5.4]$ cargo install du-dust
    Updating crates.io index
  Downloaded du-dust v0.5.4
  Downloaded 1 crates (86.4 KB) in 0.40s
  Installing du-dust v0.5.4
error: failed to compile `du-dust v0.5.4`, intermediate artifacts can be found at `/tmp/cargo-installux8cxe`

Caused by:
  failed to parse lock file at: /home/grad3/jalal/.cargo/registry/src/github.com-1ecc6299db9ec823/du-dust-0.5.4/Cargo.lock

Caused by:
  invalid serialized PackageId for key `package.dependencies`
[jalal@goku dust-0.5.4]$ ls
total 108K
drwxr-xr-x.  6 jalal cs-grad 4.0K Jan 16 10:12 tests
drwxr-xr-x.  3 jalal cs-grad 4.0K Jan 16 10:12 src
-rw-r--r--.  1 jalal cs-grad 2.3K Jan 16 10:12 README.md
drwxr-xr-x.  2 jalal cs-grad 4.0K Jan 16 10:12 media
-rw-r--r--.  1 jalal cs-grad  12K Jan 16 10:12 LICENSE
-rw-r--r--.  1 jalal cs-grad  155 Jan 16 10:12 .gitignore
drwxr-xr-x.  3 jalal cs-grad 4.0K Jan 16 10:12 .github
drwxr-xr-x.  2 jalal cs-grad 4.0K Jan 16 10:12 ci
-rw-r--r--.  1 jalal cs-grad  951 Jan 16 10:12 Cargo.toml
-rw-r--r--.  1 jalal cs-grad  14K Jan 16 10:12 Cargo.lock
drwx-----x. 68 jalal cs-grad  40K Mar 23 18:52 ..
drwxr-xr-x.  7 jalal cs-grad 4.0K Mar 23 18:52 .

Implements #103

Since I was added a new dep, I also sorted/updated the existing deps. If you want me to not do that in this PR I'll revert and just add the new dep.

Hi! 👋🏻 Thank you for making dust, it's awesome!

debian-11.1.0-arm64 virtualized in an Apple M1:

$ wget https://github.com/bootandy/dust/releases/download/v0.7.5/dust-v0.7.5-arm-unknown-linux-gnueabihf.tar.gz
$ tar vzxf dust-v0.7.5-arm-unknown-linux-gnueabihf.tar.gz
$ cd dust-v0.7.5-arm-unknown-linux-gnueabihf
$ chmod +x dust

$ ./dust
zsh: exec format error: ./dust

$ bash
$ ./dust
bash: ./dust: cannot execute binary file: Exec format error

However, building it from source works:

$ docker run --rm -v "$PWD":/usr/src/dust -w /usr/src/dust rust:latest cargo build --release
   ... 
   Compiling du-dust v0.7.5 (/opt/src)
    Finished release [optimized] target(s) in 1m 41s
$ ./target/release/dust
  ...
  49M       ├─┴ deps                                    │▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒█████████████████████████████ │  60%
  82M     ┌─┴ release                                   │████████████████████████████████████████████████ │ 100%
  82M   ┌─┴ target                                      │████████████████████████████████████████████████ │ 100%
  83M ┌─┴ .                                             │████████████████████████████████████████████████ │ 100%

• adds GitHub-Actions CI workflow (includes arm-linux, x86-linux, macos, and windows builds/testing)
  • automatic deployment to Github Releases when commit is version tagged (see example at https://github.com/rivy/rs.dust/releases/tag/v0.4.9; note: fully automatically generated)
• improves path manipulation portability
• builds / tests successfully locally and on all CI hosts
• wip / needs further investigation
  • feasability of (currently disabled) symlink tests on "windows"
  • investigate and/or update tests to handle minor size variations in testing output for different "windows" hosts
    • some one byte differences between local and CI host file sizes; ? bug or expected host variation
  • cargo tarpaulin inability to handle test coverage (use alternate framework?)

I often find support for exclude/ignore, but these tools seem to lack the opposite of allowing to only care about certain extensions (eg images png|gif|jpg|jpeg|webp or archives zip|tar|rar|7zip|zstd). Such functionality would be quite handy for me to get a better perspective of content breakdown for a server with user uploaded content.

Identifying how much disk usage a particular image type is using, or what the largest N images are. CLI tools like this one all seem to have the capability of doing such, but when the scanned content is not as well organized, or spread out among dirs where sibling/nested dirs can contain various other content or mixed types (eg if I only want to know about png disk usage), this is problematic without the ability to filter.

Dust is currently about 4 times slower than du.

It would be nice to know why and how it's slower. Did you add benchmarks of some sorts?

In a second phase, it's probably easy to optimize it?

upgrade all dependencies except clap (I saw in a comment you were maybe already working on switching to clap4)

it's pretty straightforward, only sysinfo required some code changes because they return the amount of avail memory in bytes now, not in kB.

On a system with 1,500,000 inodes used and 2GB RAM (with ~500MB free), dust is consistently OOM-killed. I'm not sure what the internals of dust look like, but it seems to me that it should be possible to operate with significantly less than that.

It looks like what's happening is that every node name is stored up-front when the directory tree is being walked, and then the summary is only done at the end. I'm not sure how much surgery would be required to improve the situation. One thought is to not store the PathBuf in the Node, and instead only get the name for the node when assembling the final output, by walking the directory tree one more time and matching inode numbers. This would probably want to be behind a --low-memory switch or something like that, since it's a tradeoff — it would take about twice a much time (walking the file tree twice), but use significantly less memory.

Another option, which is less of a tradeoff, is to store just the top section of the path, rather than the absolute path in the Node object, and only assemble absolute paths from the tree of nodes. I haven't looked deeply into how much the assumption of absolute paths is baked-in, so maybe this is a non-starter, though.

Would love for dust to be more useful on my file-filled and ram-constrained server :)

When dust is performing on a large set of file, it doesn't print anything to the terminal, which can confuse the user.

That's why i've implemented a small indicator. For the moment it doesn't show up if the process last less than 2 seconds (the timing is started at the indexing, because it's the operations that takes most of the time). If showed, it prints 3 things :

• A rolling thing, to indicate that the program doesn't freeze
• The operation
• Some more infos if needed

It can show two operations : Indexing, and Preparing. When in Indexing, it shows the number of file indexed and the total size of them (the size is based on node's information). Preparing state is set after Indexing, when the nodes are cleanup and filtered before computing the output.

When the indicator is stopped, it returns the output, just in case you want to use them.

I'm pretty sure there are things that are not done in the best way, and i'll change them if requested, but for now i just wanted your opinion on this small feature.

I've also added a modification to your .gitignore file. You can reject it as well.

OS: Arch Linux 6.0.6-zen1-1-zen x86_64

Hello, it seems that dust got slower recently.

To be more precise I run sudo dust -X home /, a little while ago this would finish within a few seconds (maybe around 30) but now it was left for a few hours and still didn't finish, pinning all my cores (Ryzen 5 5600X) to 100%

I downloaded and tried different versions and the last one that worked was v0.8.0.

Mind you all versions work if I run them in e.x. my home directory, but not in root.

Edit with some more info: It seems like it still worked fine on alpha-2, so it must be something in these commits.

Hi, I'm a screen reader user, and when running this tool, a lot of unicode symbols trip up my screen reader. Take this line: 291G ├─┴ Dropbox │ ████████████████████████████████████ │ 67% Results in me hearing: 291G plus? Dropbox symbol
symbol symbol symbol symbol symbol symbol symbol symbol symbol symbol symbol symbol symbol symbol symbol symbol symbol symbol symbol symbol symbol symbol symbol symbol symbol symbol symbol symbol symbol symbol symbol symbol symbol symbol symbol symbol │ 67% Is there a reason these particular characters are used? If there is, could a flag for screen readers be added? Thanks!