Video Duplicate Finder

vid_dup_finder finds near-duplicate video files on disk. It detects videos whose frames look similar, and where the videos are roughly the same length (within ~5%).

vid_dup_finder will work with most common video file formats (any format supported by FFMPEG.)

How it works

Video Duplicate finder extracts several frames from the first minute of each video. It creates a "perceptual hash" from these frames using 'Spatial' and 'Temporal' information from those frames:

The spatial component describes the parts of each frame that are bright and dark. It is generated using the pHash algorithm described in here
The temporal component describes the parts of each frame that are brighter/darker than the previous frame. (It is calculated directly from the bits of the spatial hash)

The resulting hashes can then be compared according to their hamming distance. Shorter distances represent similar videos.

Requirements

Ffmpeg must be installed on your system and be accessible on the command line. You can do this by:

Debian-based systems: # apt install ffmpeg
Yum-based systems: # yum install ffmpeg
Windows:
1. Download the correct installer from https://ffmpeg.org/download.html
2. Run the installer and install ffmpeg to any directory
3. Add the directory into the PATH environment variable

Limitations

vid_dup_finder will find duplicates if minor changes have been made to the video, such as resizing, small colour corrections, small crops or faint watermarks. It will not find duplicates if there are larger changes (flipping or rotation, embedding in a corner of a different video etc)

To save processing time when working on large datasets, vid_dup_finder uses only frames from the first 30 seconds of any video. vid_dup_finder may return false positives when used on content of the same length and and a common first-30- seconds (for example a series of cartoons with a fixed into sequence)

False Positives

Because this library only checks the first 30 seconds of each video, if two videos are the same length and share the first 30 seconds of video content, they will be reported as a false match. This may occur for TV shows which contain opening credits.

License

Licensed under either of

Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

General purpose client/server networking library written in Rust, built on top of the QUIC protocol which is implemented by quinn

Overview "This library stinks!" ... "Unless you like durian" durian is a client-server networking library built on top of the QUIC protocol which is i

92 Dec 31, 2022

Comments

Add additional API for VideoHash
Today I finished implementing this library in czkawka and results can be seen here - https://github.com/qarmin/czkawka/pull/460

It was surprisingly easy to do(most of time took preparing GUI)

In app I use custom system to save cache to text files, so I need info about each element from VideoHash. I found that currenty it is not possible to do it, so I had to create custom fork.

Changes from this fork are visible here(I used rust formatter, so there is a lot diffs from this tool) - https://github.com/qarmin/vid_dup_finder_lib/commit/a4809772aea8f73c9a22da6fb43df50bfdd1b31d. It would be good to integrate this or similar changes to project, because this will allow me to work with upstream version of this library and get always the newest features.

Since I load hashes from file, then I need to create VideoHash object manually so I added this function

pub fn with_start_data( duration: u32, src_path: impl AsRef<Path>, hash: [u64; HASH_QWORDS], num_frames: u32, ) -> Self { VideoHash { hash, num_frames, src_path: src_path.as_ref().to_path_buf(), duration, } }

Also there is no possibility to get hash or num_frames from VideoHash so I added this functions

/// The hash. pub fn hash(&self) -> &[u64; HASH_QWORDS] { &self.hash } pub fn num_frames(&self) -> u32 { self.num_frames }

The last thing is that const HASH_QWORDS is not public. I need to check if number of loaded hash elements is equal to number of needed hash elements, so access to such constant would be really helpful(for now I have hardcoded number 19)
opened by qarmin 2