Probably issue in cyrillic characters in path

thread 'main' panicked at 'byte index 11 is not a char boundary; it is inside 'с' (bytes 10..12) of `/фото с iPhone/IMG_6740 1.JPG`', src/output.rs:12:14
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Is your feature request related to a problem? Please describe. Currently, there is no fine control over the directory scanning process. Using the Globwalk Builder, we can add options like --max-depth & --follow-links to offer more control over the directory scanning process.

Describe the solution you'd like https://docs.rs/globwalk/latest/globwalk/#advanced-globbing

cc: @beeb adding this issue here to discuss options

This change makes the result table sorted by ascending size, allowing the user to immediately see the largest file at the bottom of the (potentially large) output.

The fact that there are a lot of unwrap's in the scanner code makes it so that the app panics when I run it against my home directory:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }', [...]/deduplicator-0.0.3/src/scanner.rs:52:65
stack backtrace:
   0: rust_begin_unwind
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/panicking.rs:575:5
   1: core::panicking::panic_fmt
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/panicking.rs:65:14
   2: core::result::unwrap_failed
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/result.rs:1791:5
   3: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold
   4: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter
   5: rayon::iter::plumbing::Folder::consume_iter
   6: rayon::iter::plumbing::bridge_producer_consumer::helper
   7: <rayon::vec::IntoIter<T> as rayon::iter::IndexedParallelIterator>::with_producer
   8: rayon::iter::extend::<impl rayon::iter::ParallelExtend<T> for alloc::vec::Vec<T>>::par_extend
   9: deduplicator::scanner::duplicates
  10: deduplicator::app::App::init
  11: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  12: tokio::runtime::park::CachedParkThread::block_on
  13: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
  14: tokio::runtime::runtime::Runtime::block_on
  15: deduplicator::main

Faillible actions (especially against the filesystem) should be properly handled (most likely skipping the affected files during the scan).

Not sure if you'll agree to this change, but I was not able to scan my full home directory with deduplicator, probably because it was trying to keep too many open file descriptors. globwalk seems to handle this much better (the default limits the number of open descriptors to a sane value). With this change I was able to scan my full home dir.

I have not tested it extensively but it seems to work for me, even with the --types argument.

Describe the bug After the scanning is complete, the app hangs for a second before printing the output. This is clearer with large directories.

** Runtime Info ** Install Type: [e.g. cargo install] App Version: [e.g. v0.1.1]

Expected behavior Printing should be fast

Platform Details (please complete the following information):

• OS: Arch Linux
• Terminal Emulator: Kitty
• Shell: Bash

Describe the bug The memory usage while scanning a 127 GB directory of PDFs, Images & Videos shot up to 26 GiB from 4.8 GiB (Initial), causing the desktop manager (lightdm) to crash & restart.

Runtime Info App Arguments: none Install Type: cargo install App Version: 0.0.8

Expected behavior Reduced Memory Consumption.

Platform Details (please complete the following information):

• OS: Arch Linux (Kernel: 5.15.86-1-lts)
• Terminal Emulator: Kitty
• Shell: Zshell

Just had an idea of how to maybe improve performance. Instead of hashing all files, how about first differentiating them by using the file size (which is unlikely to be identical for two large files that are different) and only relying on the hash when they have the same size?

I think its generally expected feature of this tool, it can be interactively like "select file to delete: 1, 2, 3" with some --interactive option or just auto delete first match (keep only last one) with --remove.

Is your feature request related to a problem? Please describe. Currently, deduplicator is only installable via cargo (rust's build tool). Need to make pre-built binary download options available to make deduplicator easily accessible to more people.

Describe the solution you'd like Create workflows to cross compile binaries for the following platforms

• x86_64
• AArch64

Describe alternatives you've considered

• Distributing through linux package repositories (plans for the future)

Additional context N/A

OS: Windows 11 Enterprise OS version: 22000.1335

When I try to run app, I get an error "Error: unable to open database file (code 14)"

Example:

cargo run --release -- --dir=test_data
warning: unused imports: `Frame`, `Rect`
 --> src\app\ui.rs:5:45
  |
5 |     layout::{Constraint, Direction, Layout, Rect},
  |                                             ^^^^
...
9 |     Frame, Terminal,
  |     ^^^^^
  |
  = note: `#[warn(unused_imports)]` on by default

warning: unused import: `std::thread`
  --> src\app\mod.rs:14:5
   |
14 | use std::thread;
   |     ^^^^^^^^^^^

warning: unused import: `std::time::Duration`
  --> src\app\mod.rs:15:5
   |
15 | use std::time::Duration;
   |     ^^^^^^^^^^^^^^^^^^^

warning: unused imports: `Block`, `Borders`, `Widget`
  --> src\app\mod.rs:18:15
   |
18 |     widgets::{Block, Borders, Widget},
   |               ^^^^^  ^^^^^^^  ^^^^^^

warning: unused import: `Backend`
 --> src\app\ui.rs:4:15
  |
4 |     backend::{Backend, CrosstermBackend},
  |               ^^^^^^^

warning: associated function `cleanup` is never used
  --> src\app\mod.rs:39:8
   |
39 |     fn cleanup(term: &mut Terminal<CrosstermBackend<io::Stdout>>) -> Result<()> {
   |        ^^^^^^^
   |
   = note: `#[warn(dead_code)]` on by default

warning: associated function `render_cycle` is never used
  --> src\app\mod.rs:51:8
   |
51 |     fn render_cycle(term: &mut Terminal<CrosstermBackend<io::Stdout>>) -> Result<()> {
   |        ^^^^^^^^^^^^

warning: associated function `init_render_loop` is never used
  --> src\app\mod.rs:58:8
   |
58 |     fn init_render_loop(term: &mut Terminal<CrosstermBackend<io::Stdout>>) -> Result<()> {
   |        ^^^^^^^^^^^^^^^^

warning: associated function `init_terminal` is never used
  --> src\app\mod.rs:69:8
   |
69 |     fn init_terminal() -> Result<Terminal<CrosstermBackend<io::Stdout>>> {
   |        ^^^^^^^^^^^^^

warning: struct `EventHandler` is never constructed
 --> src\app\event_handler.rs:7:12
  |
7 | pub struct EventHandler;
  |            ^^^^^^^^^^^^

warning: associated function `init` is never used
  --> src\app\event_handler.rs:10:12
   |
10 |     pub fn init() -> Result<events::Event> {
   |            ^^^^

warning: associated function `handle_keypress` is never used
  --> src\app\event_handler.rs:21:8
   |
21 |     fn handle_keypress(keyevent: KeyEvent) -> Result<events::Event> {
   |        ^^^^^^^^^^^^^^^

warning: enum `Event` is never used
 --> src\app\events.rs:1:10
  |
1 | pub enum Event {
  |          ^^^^^

warning: struct `Ui` is never constructed
  --> src\app\ui.rs:12:12
   |
12 | pub struct Ui;
   |            ^^

warning: associated function `generate_file_list` is never used
  --> src\app\ui.rs:15:8
   |
15 |     fn generate_file_list() -> impl Widget {
   |        ^^^^^^^^^^^^^^^^^^

warning: associated function `generate_info_bar` is never used
  --> src\app\ui.rs:27:8
   |
27 |     fn generate_info_bar() -> impl Widget {
   |        ^^^^^^^^^^^^^^^^^

warning: associated function `generate_file_desc` is never used
  --> src\app\ui.rs:31:8
   |
31 |     fn generate_file_desc() -> impl Widget {
   |        ^^^^^^^^^^^^^^^^^^

warning: associated function `render_frame` is never used
  --> src\app\ui.rs:35:12
   |
35 |     pub fn render_frame(term: &mut Terminal<CrosstermBackend<io::Stdout>>) -> Result<()> {
   |            ^^^^^^^^^^^^

warning: `deduplicator` (bin "deduplicator") generated 18 warnings
    Finished release [optimized] target(s) in 0.30s
     Running `target\release\deduplicator.exe --dir=test_data`
Error: unable to open database file (code 14)
error: process didn't exit successfully: `target\release\deduplicator.exe --dir=test_data` (exit code: 1)

I tried to install app via cargo, also I tried to build app from source code, but I got the same error.

How can I fix it?

Is your feature request related to a problem? Please describe. If I don't want to exclude duplicates of a single file type, It's very difficult to do it.

Describe the solution you'd like Add a --exclude-types / -x to exclude certain file types from being scanned by deduplicator

Is your feature request related to a problem? Please describe. The Interactive mode allows the deletion of files one duplicate group at a time. When working with millions of files, this can be tedious.

Describe the solution you'd like In order to automate this process by using deduplicator in scripts, adding options like --keep-latest --keep-oldest can help.

Describe alternatives you've considered adding custom config files that can parse a DSL to decide which files to keep [idea for the future]

I don't own or have access to a windows machine. If anybody does have access to a windows machine, please help out by testing deduplicator on windows.

Required Information:

1. Benchmarks - Speed & Memory Efficiency
2. Bug Reports

Is your feature request related to a problem? Please describe. In order to avoid fixed issues from resurfacing, unit tests need to be added for each mod.

Describe the solution you'd like Add Tests for:

• hashing
• file scanning
• finding duplicates by size
• finding duplicates by hashes
• printing output (integration tests)
• interactive mode (integration tests)

Describe alternatives you've considered N/A

Additional context As the application is going through major changes, it's essential to make sure that it doesn't break.