Everything Context menu find duplicates in selected folder

[eswul62]

Using Windows Explorer or Directory Opus I would like to :

r-click a folder (in the folder tree)

and have Everything searched that folder for duplicates with parameters dupe: dmdupe: sizedupe:

How to do that?

Thanks.

[raccoon]

Everything 1.5 Alpha introduces a new method of duplicate finding. Column-data duplicates. This effectively replaces the dupe:, dmdupe:, sizedupe: commands because those only work against the entire filesystem Index and cannot be refined to a specific drive or folder or search results.

For example:

dupe: is now superseded by the combination of sort:name find-dupes:

sizedupe: is now superseded by the combination of sort:size find-dupes:

dmdupe: is now superseded by the combination of sort:dm find-dupes:

You can stack these and include them with other keyword searches, and they will be acted upon from left to right.

Changing how your windows context menus work is beyond the scope of my help. I recommend using bookmarks within Everything for workflow.

[NotNull]

As @raccoon already mentioned: the dupe: functions work against the entire Everything database.
that means that you will get unexpected results. Fore example, if you search for: exact:abc.txt dupe: dmdupe: sizedupe:,
it will find all 3 abc.txt files with size 123 and date "yesterday", but also all 5 abc.txt files with size 456 and date "last year".


An alternative: import the following REG file and then you can choose "Search for exact duplicates" in the context menu.
(Check if the path in the regfile corresponds with the location of your Everything and edit the reg-file accordingly.)


Note this can be accomplished in a better way by using the Search Preprocessor functions, but I consider those "too alpha" to rely on at this stage of development.



ExactDupes.reg

Code: Select all

Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\SOFTWARE\Classes\*\Shell\SearchDupes]
"MUIVerb"="Search for exact duplicates"

[HKEY_CURRENT_USER\SOFTWARE\Classes\*\Shell\SearchDupes\Command]
@="cmd.exe /c for %%F in (\"%1\") do start \"\"  \"C:\\Program Files\\Everything 1.5a\\Everything64.exe\"    -search \"  exact:\"\"\"%%~nxF\"\"\"     size:%%~zF    dm:\"\"\"%%~tF\"\"\"      \""

[NotNull]

OK, I completely overlookes the "r-click a folder (in the folder tree) and have Everything searched that folder for duplicates" part....
What I wrote is for right-clicking a file and find exact duplicates (based on name, size, date) of that file anywhere.
Don't think that is possible with Everything (but will think about it).

You will be better off with a dedicated duplicate finder utility like AllDup.

[therube]

I'm probably not understanding, but...

You can stack these and include them with other keyword searches, and they will be acted upon from left to right.

Explain that a bit?

sort:name find-dupes: sort:size find-dupes:

Should that find name dupes that are only [also] the same size?

[raccoon]

yes

[therube]

But that is not the case.
.
.

Everything - sort by name by size.png (19.87 KiB) Viewed 414352 times

[void]

Unfortunately, dupe: dmdupe: sizedupe: will not work with a folder search as these search functions find duplicates in the ENTIRE index.

With Everything 1.4, please try the following:

• In the Registry Editor (Start -> regedit), navigate to:
  HKEY_CLASSES_ROOT\Directory\shell
• Create the following key:
  Find dupes in Everything...
• Create the following key:
  command
• Set the (Default) data to:
  "C:\Program Files\Everything\Everything.exe" -path "%1" -search sizedupe:

Please note: this will find files with a duplicated size anywhere in the entire INDEX and limit the results to the specified path.





For Everything 1.5, please try the following:

• In the Registry Editor (Start -> regedit), navigate to:
  HKEY_CLASSES_ROOT\Directory\shell
• Create the following key:
  Find dupes in Everything 1.5...
• Create the following key:
  command
• Set the (Default) data to:
  "C:\Program Files\Everything 1.5a\Everything64.exe" -path "%1" -search "find-dupes: sort:size"

sort:name find-dupes: sort:size find-dupes:

Sorting and find-dupes are applied after the search completes.
Only the right-most sort: and find-dupes: command is used.

[raccoon]

sort:name find-dupes: sort:size find-dupes:

Sorting and find-dupes are applied after the search completes.
Only the right-most sort: and find-dupes: command is used.

They didn't stack before? Should they be stacked? The same way that using column header right-click "Find xxx Duplicates" can be stacked?

For that matter, can we make column header right-click "Find xxx Duplicates" literally append the search terms "sort:xxx find-dupes:" to the search bar for continuity and teaching moment? Interactive learning.

[void]

They didn't stack before?

No.

Should they be stacked?
The same way that using column header right-click "Find xxx Duplicates" can be stacked?

Probably, yes.

For that matter, can we make column header right-click "Find xxx Duplicates" literally append the search terms "sort:xxx find-dupes:" to the search bar for continuity and teaching moment?

I will trial stacking dupe search commands in the search box.
Thanks for the suggestion.

[eswul62]

Everything 1.5 Alpha introduces a new method of duplicate finding. Column-data duplicates. This effectively replaces the dupe:, dmdupe:, sizedupe: commands because those only work against the entire filesystem Index and cannot be refined to a specific drive or folder or search results.

For example:

dupe: is now superseded by the combination of sort:name find-dupes:

sizedupe: is now superseded by the combination of sort:size find-dupes:

dmdupe: is now superseded by the combination of sort:dm find-dupes:

You can stack these and include them with other keyword searches, and they will be acted upon from left to right.

Changing how your windows context menus work is beyond the scope of my help. I recommend using bookmarks within Everything for workflow.

Thank you so much!
Ahum . . .
I was not aware of this to be honest.

I do have a separate tool to check for duplicates (purchase), however, I use it only in case of checking for duplicates within multiple folders on multiple drives, such scenarios.
It takes quite a lot clicks selecting folders, blah blah and... waiting for the results and so.

For the fast results I'd like to use Everything.

Yesterday I was working within 1 folder, with quite a number of duplicates, with different names though.
Sort on size and then judge/select and delete them one by one, leaving 1 in place.

Again thanks for the above.

Note sure how to combine them though, meaning: show ONLY files that are duplicates both size AND date (at the same time).

As far as I can see:
sort:size find-dupes: sort:dm find-dupes:

The top -say- 15/16 rows are duplicate modified date wise, I am seeking a way to find the ones like the below 6 (same size/date)

SnagIt-12012022 140905.png (31.49 KiB) Viewed 414299 times

Any suggestions?

[eswul62]

@void - thank you very much!
This may work for the installed version, I am also running a portable version, side by side.
So I am trying to find an alternative way, that I may use in future cases.

Whereas I initially thought it was limited to 1 folder, I just now noticed it involves more folders.

Feel embarrassed to confess that these 'sort:size find-dupes:' an d 'sort:dm find-dupes:'
were new to me. Ahum...

But I believe I've pretty much covered the matter now by using:

[example]

X:\Canon\|"Y:\Photoprojects\Holidays and weekends\2005\" sort:size find-dupes: sort:dm find-dupes:
or
X:\Canon\|"Y:\Photoprojects\Holidays and weekends\2005\" France find-dupes:
(and sort manually)

[therube]

For that matter, can we make column header right-click "Find xxx Duplicates" literally append the search terms "sort:xxx find-dupes:" to the search bar for continuity and teaching moment? Interactive learning.

I'm totally against that.
Cause IMO all that would do is to clutter up the search bar & more importantly history.
If manually typed in, I'm OK with it ending up in history, but if it is automatically entered only cause you selected a menu/context menu item, I wouldn't care for that.

(Even the same way that, Search for {Name|Size|Date...} does, I don't care for that either.)

[void]

Everything 1.5.0.1296a makes some changes to dupe: and finddupes:

You can now specify a property with dupe: search function:

dupe:<property-name>

For example:

dupe:size
dupe:date-modified
dupe:dm (dm is an alias for date modified)

The old behavior of dupe: stays the same, that is dupe: (without a property-name) finds files/folders where another file/folder with the exact same name exists in the index.



Like dupe:, you can now specify a property with finddupes: search function:

finddupes:<property-name>

For example:

finddupes:size
finddupes:dm


This will avoid the messy sort:<property-name> finddupes: syntax.
However, the old behavior of sort:<property-name> finddupes: will still work when no property-name is specified with finddupes:



You can now specify multiple dupe:<property-name> and finddupes:<property-name> in a search.

duplicates are found in order from left to right.

For example:

dupe:size dupe:dm dupe:name



You can now also specify the sort order without affecting the dupe command:

For example:

dupe:size dupe:dm dupe:name sort:path

For that matter, can we make column header right-click "Find xxx Duplicates" literally append the search terms "sort:xxx find-dupes:" to the search bar for continuity and teaching moment? Interactive learning.

I did try this, but it felt slow and clunky.
Each time you add a dupe: to the search, Everything must do a full requery (slow).
whereas with the column header -> right click -> find dupes, Everything only needs to find the duplicates in the current view without a requery.
The search edit pollution was noticeable as therube predicted.

(Even the same way that, Search for {Name|Size|Date...} does, I don't care for that either.)

Ideally, a filter for each column would be nice.
It is on my TODO list (column header split buttons).

X:\Canon\|"Y:\Photoprojects\Holidays and weekends\2005\" sort:size find-dupes: sort:dm find-dupes:

The following should now give the desired results:
X:\Canon\|"Y:\Photoprojects\Holidays and weekends\2005\" dupe:size dupe:dm

[raccoon]

Awesome new shiny tools, @void! I can't wait to try them out after I've finished playing with the other new shiny tools.

A common request around duplicate finding is, the ability to make each object conforms to multiple <property-name> matches, and not that some objects have duplicate names with another object, while also having duplicate sizes with entirely different objects of different names, etc. I guess this type of request would have to be called a serial or cascading duplicate search. Right now, object filtering happens in parallel and without an attention span for the previous dupe matches.

Is this something you'd consider poking at?

dupe:name;size;dm

[EricB]

I would second that! A duplicate subsearch for drilling down would be awesome.

Another thought: often I'm looking for Name duplicates for ebooks (epub format, which is actually a zipfile). Sometimes only a single epub property is different (the .opf file in the epub, which contains the book description/summary). This leads to minimal file size differences, most of time only 0,5 - 1k. Would it be possible to add a modifier to a duplicate search to allow for such a defined file size difference?

[raccoon]

From what you described, I don't think that it would be any more revealing than a simple name duplicate search. There's no way to tell whether that 1,000 bytes difference is a change in the book description/metadata, or the book itself.

What I would do instead is bulk-extract all of your .epub files using 7-zip -> Extract to "*\" so that the contents of each book is extracted to its own individual folders. Then use Everything to locate exact duplicates. This is revealed by the same filenames, filesizes, between the different folder Paths. You can also add a SHA-1 hash column to spot exact content duplicates without relying on names and sizes alone.

[EricB]

Well, I know from experience that if the size difference is only so little, the difference in content of the zip would be negligible (for this specific case of ebook comparison). It would make it easier to inspect name duplicates; whenever there is such a small size difference, it is safe (in my experience) to consider it as a duplicate and remove it. If the size difference is more than that, further inspection is needed. I use Total Commander for that, you can easily compare the contents of two zip files using the SyncDirs feature. No need to unpack. Works for many archive types.

I already implemented the hash method you describe, albeit using MD5. I added it as a property to the index, so it is immediately available at all times. This indeed allows me to filter out files that are the same in content but have a different name.

Further explanation: the small size difference for ebooks use case stems from previous use of Calibre. It updates some of the ebook properties, causing slight changes in the original epub file. A tad clumsy when you are in the process of deduplicating and ordering your collections.... I found that the combined use of Everything and Total Commander works better in that respect. I might import everything again in Calibre after all that work is done (it is a good library program for sure).

[void]

dupe:name;size;dm

I will consider this.
Thank you for the suggestion.

[therube]

bulk-extract all of your .epub files using 7-zip -> Extract to "*\"

Something similar from a command line:

fasti7.bat:

Code: Select all

777 x %* -o*  %2 %3 %4


:: with %2 %3, you can do something like
:: 777 x *.zip  -o*  *.exe  *.chm
:: which would ONLY extract .exe & .chm
:: files from the .zip, rather then (by
:: default) ALL files...

:: like for with sysinternal files, you'd
:: want the .exe & .chm, but nothing else

("777" is simply a renamed 7z.exe)
("fasti7" is named based on an old DOS tool, FASTI - Easy ZIP, ARJ, LZH unpacker creates directories for contents of each archive.)

[void]

Everything 1.5.0.1297a adds support for dupe:<property1;property2;property3>

For example:

dupe:size;dm;name
dupe:size;sha-256

Everything will find duplicates for the first property before gathering any other properties. (so the sha-256 property will only be loaded for files with the same size)
Up to a maximum of 3 properties can be specified.


dupe:

[therube]

dupe:size;dm;name

That is a global scope (unlike say, Find Name Duplicates)?

Maybe not, but...

NOT'ing is not allowed?

Code: Select all

dupe:size;dm;!name

---

But...

OK, yes it is, I think.
But then you can do something like 'Find Size Duplicates' to filter out the global-ness.
(Depending what you do, how you go about it, it can be a bit confusing...)

[wrisco]

So fast! This works great: dupe:size;dm

NOT'ing is not allowed?

Yes, I noticed that too, would be great.
I'm thinking of a way to find duplicates of size and date modified, excluding the ones that are duplicated on the same drive.

Code: Select all

regex:path:^([a-z]) dupe:size;dm !dupe:size;dm;regular-expression-match-1

or like @therube said:

Code: Select all

regex:path:^([a-z]) dupe:size;dm;!regular-expression-match-1

I wonder if this would be possible..?

[void]

Thank you for the suggestions!

Everything 1.5.0.1298a adds support for !

For example:
dupe:size;name;!dm
(Find files with the same size and name, but with a different date modified)



I also fixed the sorting after using dupe:xxx;yyy;zzz
Previous version were resorting by xxx only.
Everything will now sort by xxx, then yyy and then zzz.

I also fixed the case: search modifier.
This was previously not working at all.
By default Everything will perform a case-insensitive dupe search.



You can now also specify the sort order with -ascending/-descending suffixes, for example:

dupe:size-ascending;name-descending
The order does not affect which results are returned, only the order in which the results are displayed.

regex:path:.:(.*) dupe:size;dm;!regular-expression-match-1

will now work as expected.



Currently, Only one dupe:xxx;yyy;zzz command can be used.
I have put on my TODO list to allow multiple dupe:xxx;yyy;zzz commands.

The right click column header -> Find duplicates does not affect the initial query, this simply applies the dupe command on the current results.



dupe:

[jimspoon]

It would be great to have the ability to limit the results to files in a certain directory tree, plus all their dupes whereever else they may be, as when we were discussing "dupesizewithpath:" Maybe have a modifier with a name like "dupewithpath:" or "dupeofpath:" or "comparetopath:".

[wrisco]

Everything 1.5.0.1298a adds support for !
I also fixed the sorting after using dupe:xxx;yyy;zzz

This is great! Works really well and it's super useful!

Code: Select all

regex:path:^([a-z]) dupe:!size;!dm;!regular-expression-match-1

This seems to do what I want; show files and folders that are not duplicated on another drive.

[eswul62]

Super!

As always, thank you very much indeed.

[raccoon]

Currently, Only one dupe:xxx;yyy;zzz command can be used.
I have put on my TODO list to allow multiple dupe:xxx;yyy;zzz commands.

milestone. 1299a - added support for multiple dupe:xxx;yyy;zzz commands.

@void, Question, what is the correct way to break out of "dupe mode" back to the query? The method I've been using is toggling Search -> Match Case both on/off again.

[void]

If you are using Column Header -> Find Duplicates..., double click DUPE in the status bar to clear the dupe command.

When using dupe: in your query, I usually just add a space to the end of the query string to update the results.

[raccoon]

Thanks!

[void]

Everything 1.5.0.1301a removes the old dupe:, sizedupe: behavior of searching the entire index.

These functions will now find duplicates in the current results.
I don't think anyone will miss the old behavior.
-If you want the old behavior just use dupe:/sizedupe: on its own without any other search terms or filters.

Added name-dupe: search function.
Added path-dupe: search function.
Improved finding duplicates from the advanced search dialog.

namedupe: and sizedupe: will now combine into dupe:name;size

[raccoon]

clarity. is the old behavior of dupe: and sizedupe: removed or just deprecated from documentation? Or when you say "on its own" do you mean without any other search terms (path filtering)?

[void]

In other words, the old behavior is removed.

The old behavior was to search for duplicates in the ENTIRE index, not the current results.

So a search like:

C:\windows\ dupe:

would return unexpected results, because the 'duplicated' results would be from the entire index, not under C:\windows.



Now, the behavior is to find the duplicates within the current results.

If you specify dupe: by itself, with no other search terms or filters, it will give the old behavior, that is name duplicates in the ENTIRE index.

[kjhdfg]

Similarly, using Dopus I would like to:
right-click a single File (not a folder), and display it against all its "duplicates" (identical size), in any other location (all over C:, D: etc) in Everything.

Is that actually doable with a selected file instead of a folder?

Thank you!

[void]

Please consider the following context menu item:

• In the Registry Editor (Start -> regedit), navigate to:
  HKEY_CLASSES_ROOT\*\shell
• Create the following key:
  Find dupes in Everything 1.5...
• Navigate to:
  HKEY_CLASSES_ROOT\*\shell\Find dupes in Everything 1.5...
• Create the following key:
  command
• Navigate to:
  HKEY_CLASSES_ROOT\*\shell\Find dupes in Everything 1.5...\command
• Set the (Default) data to:
  "C:\Program Files\Everything 1.5a\Everything64.exe" -s size:[getsize:"%1"]

Right click your file and click Find dupes in Everything 1.5...
Everything will populate the search with:
size:<size-of-the-selected-file>

This will be fast, and work well for large files.

If you want exact duplicates, consider testing the sha256 sum too:
"C:\Program Files\Everything 1.5a\Everything64.exe" -s* size:[getsize:"%1"] sha256:[getproperty:"%1",sha256]
This will be slower, but accurate.

If you want the same extension, please try the following search:
"C:\Program Files\Everything 1.5a\Everything64.exe" -s* size:[getsize:"%1"] ext:[get-property:"%1","extension"]

[eswul62]

... Find dupes in Everything 1.5...

Thank you so much !

Exactly this was also something I was looking for, for quite a while.
Actually I was about to ask for the same thing.

So far I tried using a workaround: select and copied the file name to clipboard, then within Everything pasted the file name, searching for possible duplicates.
However, regularly this did not work, because the 'duplicate' has been renamed and given a more 'descriptive' name, i.e. description of what the file is about.

Like:
DSC1234567.JPG (10.295.143 bytes)
is a duplicate of
\Photos\John Doe - vacation 2022.JPG (10.295.143 bytes)
\Backup\Photos\John Doe - vacation 2022.JPG (10.295.143 bytes)

Hence the original DSC file can optionally be deleted.

Anyway, thanks again!

[void]

Just making a note here, the advanced search in Everything 1.5.0.1325a or later will now use the dupe:size;name;dm syntax.

[eswul62]

Great!

FWIW only, within Directory Opus I created a context menu item that does a quick search for duplicates of a selected file, based on size/ext.

SnagIt-27102022 143954.png (47.3 KiB) Viewed 403755 times

"D:\Everything\Everything64.exe" -s* size:[getsize:"%1"] ext:[get-property:"%1","extension"]