Help with comparing files - solved - A tool to avoid downloading files you're already downloaded

Off-topic posts of interest to the "Everything" community.
Post Reply
Stamimail
Posts: 1122
Joined: Sat Aug 31, 2013 9:05 pm

Help with comparing files - solved - A tool to avoid downloading files you're already downloaded

Post by Stamimail »

I have a big list of file links for download, and I already downloaded part of the list in the past. I don't want to download again what is already exist in my computer. So I need to compare the filenames in the computer to the filenames at the end of the links, and after that to omit from the links list the duplicates.

The list of file links look like this (txt file):
http:// ... /000001.gif
http:// ... /000002.gif
http:// ... /abcdef.zip

Trying to get the filenames of the computer by Everything wasn't so easy for me.
Ctrl+Shift+C is getting FullPath, and not the Filenames.
So I used export to csv to get it.

BTW. export to csv doesn't give you to export selected files in Results. Also, there are many columns with values that doesn't have meaning for the simple user. The simple user need the values in the same Format as in Everything. Also, there is a problem with displaying unicode characters in Excel by this method.

Anyway, now that I have the both lists, links list and computer filenames list, It's difficult to me to make this comparison and get the new links list. I need help here. Thank You.
Last edited by Stamimail on Wed Feb 22, 2017 10:16 am, edited 1 time in total.
therube
Posts: 4977
Joined: Thu Sep 03, 2009 6:48 pm

Re: Help with comparing files

Post by therube »

I'm not quite following & you may need to massage your data first, but UNIX has a uniq (unique) command.

(Ah, it's been so long, I'm not even remembering how to use it!)

Code: Select all

Usage: uniq [OPTION]... [INPUT [OUTPUT]]
Discard all but one of successive identical lines from INPUT (or
standard input), writing to OUTPUT (or standard output).

Mandatory arguments to long options are mandatory for short options too.
  -c, --count           prefix lines by the number of occurrences
  -d, --repeated        only print duplicate lines
  -D, --all-repeated[=delimit-method] print all duplicate lines
                        delimit-method={none(default),prepend,separate}
                        Delimiting is done with blank lines.
  -f, --skip-fields=N   avoid comparing the first N fields
  -i, --ignore-case     ignore differences in case when comparing
  -s, --skip-chars=N    avoid comparing the first N characters
  -u, --unique          only print unique lines
  -w, --check-chars=N   compare no more than N characters in lines
      --help     display this help and exit
      --version  output version information and exit

A field is a run of whitespace, then non-whitespace characters.
Fields are skipped before chars.
Maybe its something like this...

have:

Code: Select all

one
two
three
havenot:

Code: Select all

one
two
three
four

Code: Select all

cat.exe  have  havenot  >  combined
sort.exe  combined

uniq  -d  combined
& that should show you only the files that you do not already have, four, in this instance.
void
Developer
Posts: 16751
Joined: Fri Oct 16, 2009 11:31 pm

Re: Help with comparing files

Post by void »

Please try comparing the downloaded list and the todownload list in Everything by using the !filelist: search:
  • In Everything, search for your current downloaded copy of the files, for example, search for:
    d:\downloads\
  • From the File menu, click Export....
  • Change save as type to EFU Everything file list.
  • Type in downloaded.efu and click Save.
  • Convert your "to download" list to a list of filenames as they would appear when downloaded.
  • You should a have a full path and filename on each line, for example:
    Filename
    d:\downloads\000001.gif
    d:\downloads\000002.gif
    d:\downloads\abcdef.zip
  • At the top of the list add a header with the word Filename (as above)
  • Save the list as todownload.efu.
  • Open this file in Everything.
  • In Everything, from the Search menu, click Advanced Search....
  • Down the bottom of the search window, to the right of Search for a list of filenames, click Browse....
  • Select your downloaded.efu and click OK.
  • Click OK.
  • You might not be able to see the text in the text box, however there is text there.
  • In the Search box, press the home key to move the caret to the start of the edit box.
  • Type in a single !
  • Everything should now list all the files in todownload.efu that are not in downloaded.efu.
  • Now it's just a matter of converting the filenames back to their url. Which should be easy enough after exporting to a text file.
BTW. export to csv doesn't give you to export selected files in Results.
Exporting will always export all the current results, never the selection.
Also, there are many columns with values that doesn't have meaning for the simple user. The simple user need the values in the same Format as in Everything.
Please make sure you export as CSV.
The values will appear almost exactly as they do in Everything.
Size is formatted without commas.
Dates are formated with ISO8601: YYYY-MM-DD HH:MM:SS.
Excel should have no issue parsing these values.
Also, there is a problem with displaying unicode characters in Excel by this method.
Everything exports CSV as UTF-8.

To correctly open UTF-8 CSV files in Excel:
  • In Excel, from the Data menu, from the Import External Data submenu, click Import data....
  • Select your exported csv and click Open.
  • Change File origin to 65001 : Unicode (UTF-8).
  • Click Next.
  • Check Comma.
  • Click Next.
  • Click Finish.
  • Click OK to put the data in the existing worksheet at $A$1.
ovg
Posts: 294
Joined: Thu Oct 27, 2016 7:19 pm

Re: Help with comparing files

Post by ovg »

Stamimail
I wrote AHK script for your task and I need samples of both your files for debugging.
salazor
Posts: 258
Joined: Tue Jun 17, 2014 10:52 am

Re: Help with comparing files

Post by salazor »

Stamimail wrote:Trying to get the filenames of the computer by Everything wasn't so easy for me.
Ctrl+Shift+C is getting FullPath, and not the Filenames.
From Options - Context Menu - choose "Copy Name to Clipboard" and check "show this item" then set any shortcut
or
from Menu - File - choose "Copy Name to Clipboard" (must be checked one file or more) to work.
Stamimail
Posts: 1122
Joined: Sat Aug 31, 2013 9:05 pm

Re: Help with comparing files

Post by Stamimail »

ovg wrote:Stamimail
I wrote AHK script for your task and I need samples of both your files for debugging.
Sample of todownload:

Code: Select all

http://cloud.addictivetips.com/wp-content/uploads/2013/04/Everything-Search-Engine.png
https://screenshots.en.sftcdn.net/en/scrn/74000/74327/everything-21.jpg
https://www.pendriveapps.com/wp-content/uploads/everything-file-search-tool.jpg
http://www.majorgeeks.com/index.php?ct
http://cloud.addictivetips.com/wp-content/uploads/2009/12/EverythingDesktopSearch.jpg
https://screenshots.en.sftcdn.net/en/scrn/74000/74327/everything-19.jpg
http://i.i.cbsi.com/cnwk.1d/i/tim/2012/09/05/Foreman_12696521_8411_ddd.jpg
http://www.reyniersaudio.com/blog/wp-content/uploads/2011/03/Search-Everything.png
http://img.brothersoft.com/screenshots/softimage/e/everything_search_engine-229653-1239257527.jpeg
http://cdn.nirmaltv.com/images/Search-Everything.png
http://www.refolder.com/wp-content/uploads/2009/02/ese-01.png
https://i.kinja-img.com/gawker-media/image/upload/s--Ed6j4W0N--/c_fit,fl_progressive,q_80,w_636/18rg09rdob9rwjpg.jpg
http://images.pcworld.com/reviews/graphics/164021-every-big_350.jpg
https://bd23.https.cdn.softlayer.net/80BD23/142.4.51.106/blog/wp-content/uploads/2011/03/everything_search.png
https://screenshots.en.sftcdn.net/en/scrn/74000/74327/everything-23.jpg
http://cloud.addictivetips.com/wp-content/uploads/2013/04/Everything-Search-Engine_Filters.png
http://www.refolder.com/wp-content/uploads/2009/02/ese-02.png
http://img.creativemark.co.uk/uploads/images/580/15580/img3File.png
https://i.ytimg.com/vi/Ykq7X6_wmGY/hqdefault.jpg
https://lookeen.com/blog/wp-content/uploads/2015/04/Lookeen-Free-Desktop-Screenshot_EN.png
https://screenshots.en.sftcdn.net/en/scrn/74000/74327/everything-8.jpg
http://cloud.addictivetips.com/wp-content/uploads/2013/04/Everything-Search-Engine_FTP.png
http://www.technosamrat.com/wp-content/uploads/2008/09/locate-your-files-easily-with-everything-search-engine-for-windows.jpg
http://blogote.com/wp-content/uploads/2011/02/everything_software_windows_screenshot_022.jpg
http://www.technixupdate.com/wp-content/uploads/images/SearchinsidethefilecontentwithEverything_13BB0/everythingsearch.jpg
http://4.bp.blogspot.com/_SIq17kU32GU/S-3cVn3iOkI/AAAAAAAAATM/toj5ItyExjU/s1600/Everything.png
https://ttcshelbyville.files.wordpress.com/2015/02/1st-screen.jpg
https://www.voidtools.com/Everything.Search.Window.png
http://cdn.gtricks.com/2011/09/Everything_Search.jpg
http://www.majorgeeks.com/index.php?ct
http://cloud.addictivetips.com/wp-content/uploads/2013/04/Everything-File-List-Editor.png
http://i1-news.softpedia-static.com/images/news2/everything-review-488745-2.jpg
http://img.technospot.net/everything-file-indexer.png
http://cdn.makeuseof.com/wp-content/uploads/2012/07/10-Everything.jpg
https://i.ytimg.com/vi/cb3MfuRnRms/maxresdefault.jpg
http://cdn.ghacks.net/wp-content/uploads/2013/06/everything-search.png
http://www.voidtools.com/images/search.mount.video.example.png
Sample of downloaded:

Code: Select all

C:\Everything\Everything-Search-Engine_FTP.png
C:\Everything\locate-your-files-easily-with-everything-search-engine-for-windows.jpg
C:\Everything\everything_software_windows_screenshot_022.jpg
T:\Everything\everything-file-indexer.png
D:\Users\שם משתמש\Desktop\Everything-1.4.0.713b.x64\Search-Everything.png
D:\Users\שם משתמש\Desktop\תוכנות\Everything\686 - תרגום Everything\ese-01.png
D:\Users\שם משתמש\Desktop\תוכנות\Everything\686 - תרגום Everything\makelng\everything-8.jpg
T:\Everything\עברית - 713\Foreman_12696521_8411_ddd.jpg
D:\Users\שם משתמש\Desktop\תוכנות\Everything\801 - תרגום חדש\Template\Search-Everything.png
D:\Users\שם משתמש\Desktop\תוכנות\Everything\801 - תרגום חדש\Template\תיקיה חדשה\everything_search_engine-229653-1239257527.jpeg
D:\Users\שם משתמש\Desktop\תוכנות\Everything\818\everything-review-488745-2.jpg
@salazor
I had forgotten from that.
Another way to get it is by using F2 of renaming.
ovg
Posts: 294
Joined: Thu Oct 27, 2016 7:19 pm

Re: Help with comparing files

Post by ovg »

2Stamimail

Try script. https://yadi.sk/d/rxmEGCUD3EK2Sn
Syntax: ToDownload List_of_Links List_of_Existing_Files. List of existing files should be in UTF-8 encoding.
Output: ToDownload.txt - List of Files to Download
Stamimail
Posts: 1122
Joined: Sat Aug 31, 2013 9:05 pm

Re: Help with comparing files

Post by Stamimail »

@ovg
Almost done.
I've tested the script ToDownload.exe on the 2 lists above.
There was one miss:

Code: Select all

http://4.bp.blogspot.com/_SIq17kU32GU/S-3cVn3iOkI/AAAAAAAAATM/toj5ItyExjU/s1600/Everything.png
The file Everything.png is not listed in the List_of_Existing_Files, and this link to download Everything.png was omitted from the Output.
It will be nice to paste here the code for future tweaks.
ovg
Posts: 294
Joined: Thu Oct 27, 2016 7:19 pm

Re: Help with comparing files

Post by ovg »

2Stamimail

Source is in attachment.
PS. New (optimized) version 23-03-2017
Attachments
ToDownload.zip
(490 Bytes) Downloaded 927 times
Last edited by ovg on Thu Mar 23, 2017 10:42 am, edited 1 time in total.
Stamimail
Posts: 1122
Joined: Sat Aug 31, 2013 9:05 pm

Re: Help with comparing files

Post by Stamimail »

Thanks. I finished with my task.
Got the same results by ovg tool, and void approach but with syntax:
!"filename1.ext" !"filename2.ext" !"file-name-3.gif" !"file name_4.jpg" ... (Match WholeWord)

@ovg
Do you want to make a new thread about this tool in AutoHotkey Forum?
A tool to avoid downloading files you're already downloaded.
I want to subscribe to this thread till this tool will have a comfortable GUI. (Dialog box with Path & Browse to (the lists) txt files, and 2 TextBoxes for directly paste the lists to the dialog. Cancel & Process button to make the Output.txt)
Stamimail
Posts: 1122
Joined: Sat Aug 31, 2013 9:05 pm

Re: Help with comparing files - solved - A tool to avoid downloading files you're already downloaded

Post by Stamimail »

@void
Is it possible to treat web links in seacrh box as looking for a file?
I mean, for example while enter in:
https://i.kinja-img.com/gawker-media/im ... 9rwjpg.jpg
Everything actually will search:
18rg09rdob9rwjpg.jpg
Is there any other use for web links in SearchBox?

As for multiple search list, maybe it can be made somehow by Search History, so that the user can search each item in the list one by one.
Last edited by Stamimail on Mon Mar 20, 2017 6:01 pm, edited 1 time in total.
ovg
Posts: 294
Joined: Thu Oct 27, 2016 7:19 pm

Re: Help with comparing files - solved - A tool to avoid downloading files you're already downloaded

Post by ovg »

Stamimail wrote:Do you want to make a new thread about this tool in AutoHotkey Forum?
Sorry for delayed reply, I've missed your post somehow ....
Honestly I don't want to do this ... but if you want - you can DIY - I will not mind. You can freely use script.
In addition, I prefer cli utilities - they are very easy to integrate with Total Commander. Sorry ...
Stamimail
Posts: 1122
Joined: Sat Aug 31, 2013 9:05 pm

Re: Help with comparing files - solved - A tool to avoid downloading files you're already downloaded

Post by Stamimail »

@ovg
No problem, and thanks for the script, it was very useful.
ovg
Posts: 294
Joined: Thu Oct 27, 2016 7:19 pm

Re: Help with comparing files - solved - A tool to avoid downloading files you're already downloaded

Post by ovg »

2Stamimail
Glad to help you!
therube
Posts: 4977
Joined: Thu Sep 03, 2009 6:48 pm

Re: Help with comparing files - solved - A tool to avoid downloading files you're already downloaded

Post by therube »

Is it possible to treat web links in seacrh box as looking for a file?
So you're saying the "file name" of a currently opened web page (URL)?

So if you open the page (the URL), https://i.kinja-img.com/gawker-media/im ... 9rwjpg.jpg
You want to be able to search Everything for, "18rg09rdob9rwjpg.jpg".

Well that's basically what how to add everything as a search engine in firefox?
does, only that does a search for highlighted text on a web page, rather then the page URL itself.

I would think it could be modified to get the "basename" of the page URL?
Do you know JavaScript? (Me neither.)
therube
Posts: 4977
Joined: Thu Sep 03, 2009 6:48 pm

Re: Help with comparing files - solved - A tool to avoid downloading files you're already downloaded

Post by therube »

Try this:

Code: Select all

javascript:var%20url%20=%20window.location.pathname;%20var%20Qr%20=%20url.substring(url.lastIndexOf('/')+1);%20%20alert(Qr);if(Qr)location.href='es:'+(Qr);void(0)
If its working, the alert() can come out.

Code: Select all

javascript:var%20url%20=%20window.location.pathname;%20var%20Qr%20=%20url.substring(url.lastIndexOf('/')+1);if(Qr)location.href='es:'+(Qr);void(0)

https://www.squarefree.com/bookmarklets/
http://befused.com/javascript/get-filename-url
http://stackoverflow.com/questions/3820 ... javascript


Well its not going to be "exactly" right, as it fails on a URI that ends with a /, so www.example.com/test/ would not return "test", but www.example.com/test (without the trailing slash) works. (And I'm sure there are other places it will fail too ;-).)
ovg
Posts: 294
Joined: Thu Oct 27, 2016 7:19 pm

Re: Help with comparing files - solved - A tool to avoid downloading files you're already downloaded

Post by ovg »

2Stamimail

Try new version of script with a choice of files via standard Open/Save dialogs .... May be it will be helpful to you.
When start script without parameters, Open/Save dialogs appears. But you can still use command line parameters.

https://yadi.sk/d/wx4ilVeb3GGn9S
Last edited by ovg on Thu Mar 23, 2017 10:37 am, edited 1 time in total.
Stamimail
Posts: 1122
Joined: Sat Aug 31, 2013 9:05 pm

Re: Help with comparing files - solved - A tool to avoid downloading files you're already downloaded

Post by Stamimail »

@therube
I meant just to copy the full URL from the browser and paste it like it is in the Searchbox of Everything.
It's more easy to select/copy the whole URL than to select just the filename at the end of the URL.

@ovg
I tested it. Nice, now it can be done without the use of CMD.
3 Comments:
1. I think there is a bug.
The last link in "List of Links" is always displayed in Result (see the last result), despite its filename was in the "List of Downloaded Files".
2. "Save result to:" Dialog box
It saves the Results file without an extension. I think Save as txt file is expected.
3. "Save result to:" Dialog box
If the user press Cancel, it creates unwanted todownload.txt file.
therube
Posts: 4977
Joined: Thu Sep 03, 2009 6:48 pm

Re: Help with comparing files - solved - A tool to avoid downloading files you're already downloaded

Post by therube »

URL: http://www.voidtools.com/forum/viewtopi ... 770#p17770


"File Name":

Code: Select all

javascript:var%20url%20=%20window.location.pathname;%20var%20Qr%20=%20url.substring(url.lastIndexOf('/')+1);%20%20alert(Qr);if(Qr)location.href='es:'+(Qr);void(0)
Result: viewtopic.php

---


"Path + File Name":

Code: Select all

javascript:Qr%20=%20window.location.pathname;%20%20alert(Qr);if(Qr)location.href='es:'+(Qr);void(0)
Result: /forum/viewtopic.php

---


"Window URL":

Code: Select all

javascript:Qr%20=%20window.location.href;%20%20alert(Qr);if(Qr)location.href='es:'+(Qr);void(0)
Result: http://www.voidtools.com/forum/viewtopi ... 770#p17770


Image
ovg
Posts: 294
Joined: Thu Oct 27, 2016 7:19 pm

Re: Help with comparing files - solved - A tool to avoid downloading files you're already downloaded

Post by ovg »

@Stamimail
Try the new version, please. I fixed the bug with the last link, and your requests are also ready.
https://yadi.sk/d/wx4ilVeb3GGn9S
Stamimail
Posts: 1122
Joined: Sat Aug 31, 2013 9:05 pm

Re: Help with comparing files - solved - A tool to avoid downloading files you're already downloaded

Post by Stamimail »

@ovg
I've tested it a little. Yes, your program looks ready.
Thanks a lot. :D
ovg
Posts: 294
Joined: Thu Oct 27, 2016 7:19 pm

Re: Help with comparing files - solved - A tool to avoid downloading files you're already downloaded

Post by ovg »

@Stamimail
Check do you have the latest version? 1427 bytes 23-03-17 16:12.
I change Return to ExitApp and add two .Close() statements at the very end of script.
ovg
Posts: 294
Joined: Thu Oct 27, 2016 7:19 pm

Re: Help with comparing files - solved - A tool to avoid downloading files you're already downloaded

Post by ovg »

@Stamimail
For fun: I've added Progress Bar to indicate scanning process.
https://yadi.sk/d/jnKUs3or3GHnNv
It seems that all :mrgreen:
Stamimail
Posts: 1122
Joined: Sat Aug 31, 2013 9:05 pm

Re: Help with comparing files - Solved - A tool to avoid downloading files you're already downloaded

Post by Stamimail »

@ovg
Thank you for informing me you updated the link.
Downloaded. It works. Thanks.

Edit: [I saw you posted before me...]
Downloaded also the progessbar version. It works. Thanks.
It seems that all :mrgreen:
This was the original:
I want to subscribe to this thread till this tool will have a comfortable GUI. (Dialog box with Path & Browse to (the lists) txt files, and 2 TextBoxes for directly paste the lists to the dialog. Cancel & Process button to make the Output.txt)
I'm trying to say there is always what to improve. but don't bother yourself, unless you see another people interested.
Have Fun.
ovg
Posts: 294
Joined: Thu Oct 27, 2016 7:19 pm

Re: Help with comparing files - solved - A tool to avoid downloading files you're already downloaded

Post by ovg »

[color=#0000FF][b]Stamimail[/b][/color] wrote:....this tool will have a comfortable GUI. (Dialog box with Path & Browse to (the lists) txt files, and 2 TextBoxes for directly paste the lists to the dialog. Cancel & Process button to make the Output.txt)
Sometimes dreams come true :D
Please try new GUI version of the script.
https://yadi.sk/d/Z4Qczt6l3GL6NE. Drag'n'Drop also should working ....
Stamimail
Posts: 1122
Joined: Sat Aug 31, 2013 9:05 pm

Re: Help with comparing files - solved - A tool to avoid downloading files you're already downloaded

Post by Stamimail »

therube wrote:
Is it possible to treat web links in seacrh box as looking for a file?
So you're saying the "file name" of a currently opened web page (URL)?

So if you open the page (the URL), https://i.kinja-img.com/gawker-media/im ... 9rwjpg.jpg
You want to be able to search Everything for, "18rg09rdob9rwjpg.jpg".

Well that's basically what how to add everything as a search engine in firefox?
does, only that does a search for highlighted text on a web page, rather then the page URL itself.

I would think it could be modified to get the "basename" of the page URL?
Do you know JavaScript? (Me neither.)
Your solution is good for searching a page URL from within a browser.
In my case it was different. I had a list of URLs in a txt file.
There are cases it is more simple to just copy the link and put it in the Search Box.
Perhaps needs to add a new option in Options > Search Tab:
[ ] treat web links in search box as looking for a file
NotNull
Posts: 5461
Joined: Wed May 24, 2017 9:22 pm

Re: Help with comparing files - solved - A tool to avoid downloading files you're already downloaded

Post by NotNull »

How does this CMD command work for you?

Code: Select all

FOR /f "delims=" %X IN (2download.txt) DO @findstr /i /e "\%~nxX" downloaded.txt > nul || echo %X>> remaining.txt
If you want to use in a CMD batchfile, replace %X with %%X

(I hope I did understand you question correctly ...)


EDIT: You could combine this with WGET. WGET is used to download files off the Internet (or download complete websites):

Instead of echo %X >> remaining,txt, you could do: WGET %X
Or - if you already have the "remaining.txt" list- : WGET --input-file=remaining.txt

WGET can be found here for example: http://gnuwin32.sourceforge.net/packages/wget.htm

EDIT2: Changed the code to remove a trailing <space> at the end of the flename.
Last edited by NotNull on Mon Jun 05, 2017 10:19 am, edited 1 time in total.
Stamimail
Posts: 1122
Joined: Sat Aug 31, 2013 9:05 pm

Re: Help with comparing files - solved - A tool to avoid downloading files you're already downloaded

Post by Stamimail »

Hi NotNull,
This topic is already solved a while ago, as mentioned.
Now I just wanted to suggest a new option and bring it into a consideration.
Nevertheless I didn't want to let you down, so I did 2 tests.
In my tests I compared the results of your script with the results of ovg's tool.
The results are the same! Bravo. Except for one thing. In your results there is a single <space> that added at the end of each link (unwanted).
Thanks for the script.
NotNull
Posts: 5461
Joined: Wed May 24, 2017 9:22 pm

Re: Help with comparing files - solved - A tool to avoid downloading files you're already downloaded

Post by NotNull »

Stamimail wrote: This topic is already solved a while ago, as mentioned.
Now I just wanted to suggest a new option and bring it into a consideration.
Ah, OK, didn't pick up on that one (didn't read all of the massages, just a (very) quick scan).
Then you can happily ignore my message :-)

Not that it matters anymore, but the issue withe the extra space can be solved by replacing "echo %X >>" with "echo %X>>"
Stamimail
Posts: 1122
Joined: Sat Aug 31, 2013 9:05 pm

Re: Help with comparing files - solved - A tool to avoid downloading files you're already downloaded

Post by Stamimail »

I pretty sure your code will be useful for me or for someone some day.
I think it would be better to edit and fix your code in its place above. (I guess this someone also won't read the whole messages. :) )
Thank you for posting.
NotNull
Posts: 5461
Joined: Wed May 24, 2017 9:22 pm

Re: Help with comparing files - solved - A tool to avoid downloading files you're already downloaded

Post by NotNull »

Stamimail wrote:I pretty sure your code will be useful for me or for someone some day.
I think it would be better to edit and fix your code in its place above. (I guess this someone also won't read the whole messages. :) )
Thank you for posting.
Good thinking! Changed the code ..
Stamimail
Posts: 1122
Joined: Sat Aug 31, 2013 9:05 pm

Re: Help with comparing files - solved - A tool to avoid downloading files you're already downloaded

Post by Stamimail »

@ovg
Thank you very much for ToDownload_GUI.
I used it many times.

I thought about more features to add that can make it better:

1. As mentioned, a TextBox or other Control for pasting the lists from Clipboard (another option to load in addition to load list from a file)
2. Currently ToDownload works like this:
First List = get a list with URLs
Second List = get a list with Filenames (FullPaths)
Third List = Output

You can make it better by getting not only one kind of list, but like this:
First List = get a list with URLs|Filenames
Second List = get a list with URLs|Filenames
Third List = Output (with URLs|Filenames)

3. In the scanning/proccessing Dialog, at the end ,a summary how many files outputed.
ovg
Posts: 294
Joined: Thu Oct 27, 2016 7:19 pm

Re: Help with comparing files - solved - A tool to avoid downloading files you're already downloaded

Post by ovg »

Stamimail wrote:Thank you very much for ToDownload_GUI.
You are welcome!
Stamimail wrote:a TextBox or other Control for pasting the lists from Clipboard
I 'll see what I can do here. But please keep in mind:
I'm newbie with AHK and implementing this can take some time.
Stamimail wrote: You can make it better by getting not only one kind of list, but like this:
First List = get a list with URLs|Filenames
Second List = get a list with URLs|Filenames
Third List = Output (with URLs|Filenames)
Sorry, I didn't understand what you mean. Could you please explain more carefully?
Stamimail
Posts: 1122
Joined: Sat Aug 31, 2013 9:05 pm

Re: Help with comparing files - solved - A tool to avoid downloading files you're already downloaded

Post by Stamimail »

Sure,
The following example is probably not going to be the common use of the script but intended only to explain what the script should does.
What I did?
I took the 2 lists above and replaced/switched the 5-8th lines between those lists.

Sample of ToDownload:

Code: Select all

http://cloud.addictivetips.com/wp-content/uploads/2013/04/Everything-Search-Engine.png
https://screenshots.en.sftcdn.net/en/scrn/74000/74327/everything-21.jpg
https://www.pendriveapps.com/wp-content/uploads/everything-file-search-tool.jpg
http://www.majorgeeks.com/index.php?ct
D:\Users\שם משתמש\Desktop\Everything-1.4.0.713b.x64\Search-Everything.png
D:\Users\שם משתמש\Desktop\תוכנות\Everything\686 - תרגום Everything\ese-01.png
D:\Users\שם משתמש\Desktop\תוכנות\Everything\686 - תרגום Everything\makelng\everything-8.jpg
T:\Everything\עברית - 713\Foreman_12696521_8411_ddd.jpg
http://img.brothersoft.com/screenshots/softimage/e/everything_search_engine-229653-1239257527.jpeg
http://cdn.nirmaltv.com/images/Search-Everything.png
http://www.refolder.com/wp-content/uploads/2009/02/ese-01.png
https://i.kinja-img.com/gawker-media/image/upload/s--Ed6j4W0N--/c_fit,fl_progressive,q_80,w_636/18rg09rdob9rwjpg.jpg
http://images.pcworld.com/reviews/graphics/164021-every-big_350.jpg
https://bd23.https.cdn.softlayer.net/80BD23/142.4.51.106/blog/wp-content/uploads/2011/03/everything_search.png
https://screenshots.en.sftcdn.net/en/scrn/74000/74327/everything-23.jpg
http://cloud.addictivetips.com/wp-content/uploads/2013/04/Everything-Search-Engine_Filters.png
http://www.refolder.com/wp-content/uploads/2009/02/ese-02.png
http://img.creativemark.co.uk/uploads/images/580/15580/img3File.png
https://i.ytimg.com/vi/Ykq7X6_wmGY/hqdefault.jpg
https://lookeen.com/blog/wp-content/uploads/2015/04/Lookeen-Free-Desktop-Screenshot_EN.png
https://screenshots.en.sftcdn.net/en/scrn/74000/74327/everything-8.jpg
http://cloud.addictivetips.com/wp-content/uploads/2013/04/Everything-Search-Engine_FTP.png
http://www.technosamrat.com/wp-content/uploads/2008/09/locate-your-files-easily-with-everything-search-engine-for-windows.jpg
http://blogote.com/wp-content/uploads/2011/02/everything_software_windows_screenshot_022.jpg
http://www.technixupdate.com/wp-content/uploads/images/SearchinsidethefilecontentwithEverything_13BB0/everythingsearch.jpg
http://4.bp.blogspot.com/_SIq17kU32GU/S-3cVn3iOkI/AAAAAAAAATM/toj5ItyExjU/s1600/Everything.png
https://ttcshelbyville.files.wordpress.com/2015/02/1st-screen.jpg
https://www.voidtools.com/Everything.Search.Window.png
http://cdn.gtricks.com/2011/09/Everything_Search.jpg
http://www.majorgeeks.com/index.php?ct
http://cloud.addictivetips.com/wp-content/uploads/2013/04/Everything-File-List-Editor.png
http://i1-news.softpedia-static.com/images/news2/everything-review-488745-2.jpg
http://img.technospot.net/everything-file-indexer.png
http://cdn.makeuseof.com/wp-content/uploads/2012/07/10-Everything.jpg
https://i.ytimg.com/vi/cb3MfuRnRms/maxresdefault.jpg
http://cdn.ghacks.net/wp-content/uploads/2013/06/everything-search.png
http://www.voidtools.com/images/search.mount.video.example.png
Sample of Downloaded:

Code: Select all

C:\Everything\Everything-Search-Engine_FTP.png
C:\Everything\locate-your-files-easily-with-everything-search-engine-for-windows.jpg
C:\Everything\everything_software_windows_screenshot_022.jpg
T:\Everything\everything-file-indexer.png
http://cloud.addictivetips.com/wp-content/uploads/2009/12/EverythingDesktopSearch.jpg
https://screenshots.en.sftcdn.net/en/scrn/74000/74327/everything-19.jpg
http://i.i.cbsi.com/cnwk.1d/i/tim/2012/09/05/Foreman_12696521_8411_ddd.jpg
http://www.reyniersaudio.com/blog/wp-content/uploads/2011/03/Search-Everything.png
D:\Users\שם משתמש\Desktop\תוכנות\Everything\801 - תרגום חדש\Template\Search-Everything.png
D:\Users\שם משתמש\Desktop\תוכנות\Everything\801 - תרגום חדש\Template\תיקיה חדשה\everything_search_engine-229653-1239257527.jpeg
D:\Users\שם משתמש\Desktop\תוכנות\Everything\818\everything-review-488745-2.jpg
You can make it better by getting not only one kind of list, but like this:
First List = get a list with URLs|Filenames
Second List = get a list with URLs|Filenames
Third List = Output (with URLs|Filenames)
One example for a common use of such a script is:
When you want to download similar content from 2 websites, and you want to avoid the same content\files to be downloaded twice. In this case you have 2 lists with only URLs lines. No filenames.)
ovg
Posts: 294
Joined: Thu Oct 27, 2016 7:19 pm

Re: Help with comparing files - solved - A tool to avoid downloading files you're already downloaded

Post by ovg »

2Stamimail

Please try the new version: https://yadi.sk/d/BDPILaLf3Mzwjo
Stamimail wrote: 2. Currently ToDownload works like this:
First List = get a list with URLs
Second List = get a list with Filenames (FullPaths)
Third List = Output

You can make it better by getting not only one kind of list, but like this:
First List = get a list with URLs|Filenames
Second List = get a list with URLs|Filenames
Third List = Output (with URLs|Filenames)

3. In the scanning/proccessing Dialog, at the end ,a summary how many files outputed.
This items are done
Last edited by ovg on Tue Sep 19, 2017 10:52 am, edited 1 time in total.
ovg
Posts: 294
Joined: Thu Oct 27, 2016 7:19 pm

Re: Help with comparing files - solved - A tool to avoid downloading files you're already downloaded

Post by ovg »

2Stamimail

New version: https://yadi.sk/d/UmU0SINO3N33Xg
Item 1 is done >
Stamimail wrote:pasting the lists from Clipboard
Before submit Checkbox be shure that corresponding list is in the Clipboard
Post Reply