OT Duplicate file finder

Everything search is a free program for windows - it's lightening fast - it indexes a hard disk in seconds.

and then if you want to try the alpha 1.5.xx version

On the same page as the 1.5 alpha download there is a link to what's new which in turn links to screenshots and brief instructions

This does have a duplicate file name search and when checking on my win

You highlight the column title "name" and right click to get a menu where one of the options is a duplicate file search. It may not be quite what you want as I see no way of quickly deleting the duplicate files but it could give you an almost instant indication about how many duplicates you have and the file sizes. The file sizes can be shown in size order and perhaps you could just identify/delete the main ones causing your lack of backup space.

For duplicate names on a pre-indexed disc perhaps, except that would miss any files that had been created, moved or renamed since the disc was indexed. Finding identical content will take a great deal longer - that involves finding all files of the exact same size and generating a hash to compare. This requires reading the whole file and processing the content. It would be possible to generate all the hashes in the background but then again you'd have to keep track of any changes to the file system once you've done that, which would require running a journaling layer on top of the existing file system, or hooking into an existing JFS.

The software indexes my laptop disk in seconds and in the tools menu/indexes/ there is a force index option that can be used prior to running the duplicate search

Finding identical content will take a great deal

It may only be searching the index for identical file names* but it does show the file sizes. (* or paths, file sizes, dates etc.)

Agreed it may not be what the OP wants in that it may not check the contents of the file but may provide a fast method of identifying possibly the few memory hungry files.

I thought I would try some of these programs in parallel.

CCleaner: it's still running after 10 days without any indication of when it might complete.

Auslogics Duplicate File Finder 10: this initially looked promising but the progress bar is permanently at 99% and the Processed memory is slowly counting up.

Fast Duplicate File Finder: after 10 days this is at 1%

Everything 1.5a: indexes files quickly but doesn't show length and can't sort (it just hangs)

SMF: this hashes 10-15 files per second, so is going to take a while Even at that rate after 10 days it should have finished. It hasn't.

Thanks for the ideas.

What settings did you use? Rather than insisting on doing a hash on every file, it might help just to match on file name, size and date to find the most obvious deletion candidates; and any that you are not sure about, leave to a second pass in which you check the hashes.

In parallel?? Five different applications fighting over the same resources?

It shows file size. Right click on the name/path etc heading and there is a list of additional options (although file size was set by default on my installation).

When is the last time this disk had CHKDSK run on it ?

This sounds like quite a large disk drive, with quite a few files on it.

Do you have any ideas on those two numbers ?

If you've been running five tools for ten days on that drive, you've probably beaten the piss out of the drive :-)

Task Manager, have a look at the allocations and I/O Read and I/O Write columns. You can add columns to Task Manager display, to augment what you know.

I would think memory has gone through the roof, and the programs are slowed down, by sticking an elbow into each other.

I have a copy of Everything 1.5a now. I will test on C: .

Name: Everything64.exe Size: 4772872 bytes (4661 KiB) SHA1: E28504F3C323E7F62F2988B784FC72EF427D29A6

Everything64.exe -create-filelist every_c.txt "C:"

notepad every_c.txt

(500KB memory growth per second, while it builds file list) (Took about 2 minutes perhaps...)

The program works by reading the $MFT first (gives filenames for paths). But size determination requires a stat() of each directory, and the tree walk is a lengthy process on a big disk. Using Task Manager, you wait until Everything64.exe disappears from the trace, then go looking for that output file.

My space is 58.1GB used on a 118GB partition, for 293,123 files.

And my fileset (58.1GB), can't quite fit on my RAMDrive. Alternately, an NVMe drive is just as fast as a RAMDrive. If you use storage devices that support high I/O rates, then yes, you can run five dedupe programs at once, and it won't take ten days.

Paul

Brand new disk with a NAS storage drive copied onto it. The idea was to find duplicates and then mirror the deletions on the NAS drive.

Yep, except that was the purpose of the drive and finding the ideal application.

I have 20GB of RAM, and yes it was at 80%

The drive is external through USB3 and it gets warm, not hot. It's also a conventional hard disk, so perhaps not very comparable with your test?

There are two ways to copy the initial drive.

Cloning software works in sequential cluster order. The software does not know it is copying files. It just has a list of sectors 1,3-8,23-30 to copy. It works sequentially, so there is less "grinding" of the disk drive.

However, future traversals across the clone, are just as non-optimal as the original drive. The drive is fragmented, the average stroke size is large. Analysis softwares scanning the disk, are going to run slower.

We made the cloning step fast. The five analysis runs will be slow.

This would be in alphabetical order. This will be slow, because the source disk was never in alphabetical order, and the source seeks will be all over the place. It might take two days or more, to copy 9,000,000 files this way. Versus 7 hours for (1).

Once the files are in alphabetical order, now the "path taken" for the five analysis runs, will also be in alphabetical order.

I have done some of these miserably large experiments, on purpose, and they are damn slow. This is why I know that there are hard choices to make.

Even to do this much.

Everything64.exe -create-filelist every_E.txt "E:"

could take a while. With the disk in the shape it is in currently.

This is also one of the reasons you don't buy those new 22TB drives :-)

If your scratch drive was an SSD, the seek time would be zero, and (1) versus (2) would not matter quite as much. You would do (1) in such a case. Because a random seek during the scan, would not cost you.

The best drive type, would be a 4TB NVMe. Not exactly a convenient form factor.

However, the operating system does not run at NVMe speed, so you have only managed to eliminate any suggestion of a hardware limitation. The NTFS seek speed is still only 4000 to 10000 files per second. The seek speed does not go to a million a second, because you bought an NVMe.

Summary:

Run just one test, with the new drive as is.

If it takes way way too long, then go back and revisit how the disk copy was done, and try to put the files in "access order", the same pattern of access the analysis programs will use.

With hard drives, it's better to use "-j 1" than "-j 6" (six threads of execution), because the hard drive will not like being thrown into six different areas of the hard drive on a continuous basis. An SSD on the other hand, does not mind that. And "-j 6" will work just fine on an SSD, as the seek time is in microseconds, not 8 milliseconds.

What I first noticed, when trying a little test here, is that Everything64 (which runs in the background), had seemingly ground to a halt. It did not exit, but it did not malloc more RAM either. By freeing up resources, a second run finished. But there wasn't a lot of room left exactly. This is why running five tools at once, is just asking for (various kinds of) trouble. This test file set, is not stored in a folder, or it would be totally unworkable. It uses a container instead... which is still slow, so no savings there.

Sorry for any confusion.

I've already copied the NAS drive to this external drive. It's pretty fast as it's connected through a USB3.1 port.

I'm not sure an internal drive will gain much in speed. It does have SATA 2 sockets on the MB but these are used for existing drives.

But won't compare file contents. The pure indexing is pretty quick.

I'm not sure it is in bad shape?

Yes, I fully accept that running numerous tests simultaneously is not ideal.

Many thanks for the ideas and input.