OT Duplicate file finder

SMF= Search My Files apparently does dupe finding , but I've never used that function, just complex multi-wildcard etc search term "db" search. Freeware

One needs to be careful. For example some DLL files are meant to be duplicates if installed by certain software. Many audio pro software, editors etc, tend to use a file called lame.dll, and they often like them local.

Ccleaner can look at duplicates, but then there are many small free offerings out there, and you really need won where you can have an undo system, in case the duplicate is required. Brian

Brian Gaff snipped-for-privacy@gmail.com posted

CCleaner v5.63.7540 is almost ideal, except it would nice to be able to adjust the "ignore files smaller than X" a bit more finely.

You can get duplicate finders that replace the file with a link to another copy, saves space if it's a larger file'

Why? Just find files with identical content, regardless of their names, and decide which to delete. That's going to be your big space saver. Then find files with identical names to check manually if they are different versions of the same content, where you can delete the older versions.

It's just a pity that Windows can't do proper file linking. A proper version would keep one file, allow you to delete it or any copies and actually hold on to that file until the last link was deleted. It'd also keep all the links valid if you moved the file.

That's called a hardlink.

mklink /? # for help

mklink /H destination source # similar to syntax order of "ln" on Linux

This would be checking of a file, for its alternate hardlink pathname.

fsutil hardlink list source \____ Since the file handles are equal in function, fsutil hardlink list destination / the same list should be returned for each.

C:\> fsutil hardlink list source.txt \Users\paul\Downloads\source.txt \Users\paul\Downloads\destination.txt

C:\> fsutil hardlink list destination.txt \Users\paul\Downloads\source.txt \Users\paul\Downloads\destination.txt

There is one set of clusters, two filenames.

If I edit source.txt, save it, then delete it, the destination.txt has my changes, because the clusters stored the changes while the handle "source.txt" was being used.

Hardlinks do not navigate outside of a partition. You can't hardlink C: to D: or hardlink one spindle to another spindle.

From Linux, inode numbers are "created as a datafill" from the NTFS $MFT filenum.

paul@WALLACE:/mnt/c/Users/paul/Downloads$ ls -i source.txt destination.txt

When printed in Hex, the number is interesting C000000004F0A

Using NFI.exe (a Microsoft utility you can download), this is the $MFT entry.

File 20232 20232 is 0x4F08 (so much for that theory!) \Users\paul\Downloads\source.txt $STANDARD_INFORMATION (resident) $FILE_NAME (resident) $FILE_NAME (resident) $DATA (resident)

It seems the Bash shell added a hex offset to the filenum value.

If you need to link outside a partition, that is a symbolic link (which is also available).

They've been making equivalents, after a fashion.

The contents of WinSxS are hardlinked into System32, and that is an area of your C: drive you could use for test. This is intended, to make the WinSxS directory a "maintenance" directory for Windows Update. The SxS stands for "Side By Side".

Paul

I use WinMerge.

Does the first, plus makes the second easier: graphic diff of two files...

Thomas Prufer

Not sure if it's available for windows, but fslint is a charm on *nix. It has a gui so may run under python/

There are some Snap versions for Ubuntu.

According to this person, it's Python 2, and hiding it in a Snap is how you get around the repository being at Python 3.6 or so. That's part of the reason the Snap is "unofficial".

There's a Rust version of a similar thing. Windows version available. The czkawka is supposed to have the GTK DLLs with it.

# For FFMPEG, you want a static build (take ffmpeg.exe and throw into czkawka bin dir).

Name: ffmpeg-5.1.2-full_build.7z <=== I got this, end of last year Size: 47431656 bytes (45 MiB) SHA256: 037BDB2183189DC0EFA642B1FDE7D6B33C6114036FA06FAF35F777E8DF07D863

I didn't really have a good collection to test with. it comes up negative.

I'm pretty sure NTFS can do that.

It can do symbolic links, junctions, etc., but it can be a recipe for disaster - causing the wrong (live) file to be deleted. You actually have to analyse things before making links or deleting things.

What is needed is a system that when you copy a file, automatically creates a link and not a copy of the file; that when the original file is moved, updates all links that point to it; that when the original file is deleted, ensures that it is only hidden if any links to it remain (or maybe moves it to one of the link locations and updates the other links to point to it); and that does there things automatically, without the user having to do anything.

It can do that too (aka hard links)

That isn't copying a file. That is making a link, and has a different command

Imagine your scenario, you insert a USB, make a link on it to your file and remove the USB, Golly no file is on it.

that when the original file

This is all part of the Unix/Linux LINK command. It creates explicit duplicate directory entries to the same inode structures and file data blocks. The last unlink command to the final directory entry removes the inode structures as well thus deleting the file and making its data sectors available for other files to use.

There are three linux commands, a hard link, which will only work if the new file entry is on the same disk partition as the old, which merely duplicates the directory entry. A soft or symbolic link, which is an instruction to look for the real file over there-> on possibly a different disk which may not even exist at that point in time, and a copy command, which physically duplicates the file contents and directory entries.

There is also a MOVE command which is smart enough to know that if its on the same partition, it only needs to change the directory entry, rather than actually MOVE the whole file.

In Winders hasn't got this, try installing Linux. Or Unix.

In practice "big storage" on SANs tends to do it a different way.

Have a de-dupe process that runs in the background that looks at the sector/cluster level for identical data, and fiddle with the pointers behind the scenes so that all such sectors/clusters are only stored once, then use copy-on-write so that if one of the sectors/clusters is modified it gets its own copy.

Of course this is essentially block storage, that operates below the filesystem, but it works well on most types of data.

Sorry, I was not at all clear. I had actually forgotten that Windows (in NTFS) can do Hard Links, as it is not an option normally presented to the user. I have done links in Linux, but only when setting up necessary links for my server setup.

Also I was referring to copying on a machine's fixed storage or within a server - such as when people put the same photo in multiple directories, one for pictures of the kids, one for pictures of a particular holiday, one for pictures of a particular activity, etc. when the photo fits into all those categories.

Windows can create Shortcuts from the GUI, but they are limited - similar to Soft Links, work in Windows Explorer, but may not work in some applications.

Windows can also create fully functional Symlinks (Soft Links) and Hard Links, but they are not GUI menu options.

Yes. I know of that, as I run Linux for my home server and also on my satellite TV boxes. That was what prompted me to comment about the lack of being able to simply do so on Windows (and I suspect on Linux) by an average user (i.e. from a GUI).

It does, but that has similar limitations to the Windows version - a hard link cannot be made to a directory, only a file and cannot be made across two partitions and (presumably) cannot be made from a GUI.

Soft-links overcome most of those restrictions, but can end up dangling if the original file is deleted or renamed.

On my Satellite TV boxes, I often move programmes that are still recording from the recordings' directory to a genre or series specific directory, while the recording is in progress. It works perfectly, even when the recording is from a different (diskless) box and just using the living-room box's shared hard-disk.

I can do these things in both, but neither (as far as I know) makes it easy for an average user to do.

For most users, it would need to be as easy to use as the current copy and paste, not requiring command-line access. Just another option on the menus. For instance, I can Copy in a window and Paste elsewhere or Copy and then Paste a Shortcut, but I can't Copy and Paste a Symlink or Copy and Paste a Hard Link. I might look at adding those options.

Many thanks.

It initially looked very promising, seemingly doing a limited hash to compare files but after 99 hours it's still doing 15 hashes / sec and half way through

This hard drive is 4TB and something like 8 million files.

It doesn't seem to do a compare on file name and file size first. Nearly every file is a "potential duplicate".

This seems a shareware style of software. My experience is they provide limited functionality and promise the earth with a paid subscription.

In the initial sales blub it doesn't mention duplicate files?

What I am trying to do is clean up the mess in versioned backups, so duplicate files in essentially the same directory tree is removed in each subsequent dated delta.