REPEATS(1)

NAME
repeats.pl and repeats.py - search for duplicate files

SYNOPSIS
repeats.pl [-h(elp)] [-1(_line_output)] [-a algorithm] [-i(nit_rs) bytes] [-l(inks_hard)] [-m(in_fs) bytes] [-r ramp_factor] [-v(erbose)] [-z(ero_include)] [file...] [directory...] repeats.py [-h(elp)] [-1(_line_output)] [-a algorithm] [-i(nit_rs) bytes] [-l(inks_hard)] [-m(in_fs) bytes] [-r ramp_factor] [-v(erbose)] [-z(ero_include)] [file...] [directory...]

DESCRIPTION
repeats.pl (written in perl and utilizing routines from the CryptX module), and repeats.py (written in python and utilizing routines from the digest module) all search for duplicate files in one or more specified directories, using a three-stage process. This process works as follows: Initially, all specified files, all files in the specified directories (and their subdirectories) are listed as potential duplicates. In the optional first stage, any files which are actually a hardlink to another file are removed, since they don't actually take up any more disk space. In the second stage, all files with a unique filesize are declared unique and are removed from the list. In the third, all files for which the first 256 bytes (adjustable with the -i option) have a unique filehash are declared unique and are removed from the list, and then the third state is repeated with larger and larger reads until all files have been verified as having a match or been declared as unique and removed from the list. This process is MUCH less disk and CPU intensive than creating full hashes for all files. The duff, dupd, fdupes, jdupes, and rdfind commands utilize similar strategies.

OPTIONS

-h  Print help and quit.

-1  Print each set of duplicate files on a single line.

-a hash_algorithm
    Specify which hash algorithm should be used. Choices are 1 (MD5), 2 (SHA1), 3 (SHA224), 4 (SHA256), 5 (SHA384), 6 (SHA512), 7 (BLAKE2B-256), and 8 (BLAKE2B-512). The default is 8, for BLAKE2B-512 hashes.

-i initial_readsize
    Specify the number of bytes read per file at the beginning of stage 3. The default 256.

-l  List files that are actually hardlinks as duplicates. Normally, only the first hardlink sharing an i-node number is included as a possible repeat. [This skips stage 2.]

-m minimum_filesize
    Specify the minimum size of file (in bytes) that will be considered for duplicates. This allows the user to focus only on larger files. The default value is 1, which includes all non-zero files.

-r ramp_factor
    In repeats.pl and repeats.py, stage 3 is run repeatedly with the number of bytes read in each round being multipled by the "ramp factor" value. The default value is 4.0.

-v  Verbose output. Write some statistics concerning number of potential duplicates found at each stage to stderr.

-z  Include zero-length files in the search. If there is more than one zero-length file, all of those files will be considered duplicates.

NOTES
If no directory is specified, the current directory is assumed. In terms of program history, the legacy sh- and C-based repeats utility was written first (in 2004). The repeats.pl utility was written in 2020 to explore new algorithms and it currently implements a multi-step stage 3 algorithm that requires less disc I/O than the legacy repeats did. The repeats.py utility was written in 2023 to see how Python performance differed from Perl. The Python version is slightly faster on Linux and slightly slower on Cygwin.

BUGS
It must be noted that it is theoretically possible (though freakishly improbable) for two different files to be listed as duplicates if the -p option is not used. If they have the same size and the same file hash, they will be listed as the duplicates. The odds of two different files (of the same size) being listed as duplicates is approximately 1.16e77 to 1 for the SHA256 hash. Using arguments similar to the classic "birthday paradox" (i.e., the probability of two people sharing the same birthday in a room of only 23 people is greater than 50%), it can be shown that it would take approximately 4.01e38 different files (of exactly the same size) to achieve similar odds. In other words, it'll probably never happen. Ever. However, it's not inconceivable. You have been warned. For the various hashes, the number of same-sized files required for the probability of a false positive to reach 50% are as follows: MD5: 2.17e19 files
SHA1: 1.42e24 files
SHA224: 6.11e33 files
SHA256: 4.01e38 files (default prior to version 1.2.0)
SHA384: 7.39e57 files
SHA512: 1.36e77 files
BLAKE2B-256: 4.01e38 files
BLAKE2B-512: 1.36e77 files (default for versions 1.2.0 and later) See https://en.wikipedia.org/wiki/Birthday_problem and https://en.wikipedia.org/wiki/Birthday_attack for more information. Also, repeats.py and repeats.pl currently lack logic to mark hardlinks (under the -l option) as duplicates without actually reading the entire file multiple times. This will be addressed in a future version of littleutils.

SEE ALSO
perl(1), python(1), CryptX(3pm), duff(1), dupd(1), fdupes(1), jdupes(1), rdfind(1)

COPYRIGHT
Copyright (C) 2004-2026 by Brian Lindholm. This program is free software; you can use it, redistribute it, and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 3, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.