Mapped drives not fully indexed

[Snoopie]

Hello,

Been using your software for a few years now without a hitch for my work computer; however recently our IT team switched our File hosting service which has essentially broken Everything's ability to index the new drives completely. For reference with our previous servers, everything's index total was nearly 6 million files. After the service change and mapping the new drives I performed a "Force Rebuild" and the index total was 2.4 million files.
Everything is just finding some of the subfolders within the Mapped drives and only a few files within certain subfolders.

CMD net use readout:


Windows Explorer window showing drives mapped:


Statistics within Everything:


With today's latest release of 1.5a alpha (v1318a) I once again cleared out the index and just tried adding one of the drives using the "Folders" Index option because none of the drives show up under "Network Drives" in the settings.
The program was able to index 120,914 items but that is nowhere near the actual file count in that drive.

Example of selective index results:
NO found results 3 subfolders in:


Found results on another part of the F Drive 4 subfolders in:


Image links if I messed up the linking above:
1. https://ibb.co/T8mRkJx
2. https://ibb.co/GcBRTGf
3. https://ibb.co/nBfbhzp
4. https://ibb.co/WH3kLdC
5. https://ibb.co/r7fmF3h

[void]

Are you excluding any files/folders under Tools -> Options -> Exclude?
Are you excluding hidden/system files/folders under Tools -> Options -> Exclude?

Please try running Everything in verbose debug mode and forcing a rebuild:

• In Everything 1.5, from the Tools menu, under the Debug menu, click Start Debug Logging.
• From the Tools menu, under the Debug menu, check Verbose.
• From the Tools menu, click Options.
• Click the Indexes tab on the left.
• Click Force Rebuild.
  --wait for the reindex to complete--
• from the Tools menu, under the Debug menu, click Stop Debug Logging.
  --this will open your Everything Debug Log.txt
• This log will contain all the folders scanned.
• What folders are not scanned?
  -Is there anything special about them? (eg: folder junction/strange attributes?), can you access these folders from a command prompt..

[Snoopie]

No exclusions checked.

The log after the index rebuild may be found here
(link removed by void)
Note the link expires Sept 19, 2022.

There is nothing special about the unscanned folders. No special permissions or read/write access limitations that I'm aware of as I can access them via windows explorer and edit files within. The folders/files scanned seem random as within the main drive currently in the index there are several subfolders - the main ones being client specific. Everything seems to have randomly indexed only a few of the folders within that main folder structure.

The technology provider for the new servers is Egnyte: https://www.egnyte.com/ with an Azure backend. From a day to day usage standpoint the network drives function like any local Network drive, there isn't any local caching of files or anything that would cause me to think files are put into some sort of cold storage if no one accesses them recently

Far as accessing the drives via CMD I believe I'm using the correct command but get an error thru CMD:

[void]

Thanks for the debug log.


There are a bunch of FindFirstFileEx 1237 ERROR_RETRY errors.
Maybe this error is thrown if we are exceeding some rate limit?

I'll make Everything wait for up to one minute and keep retrying on these folders...
I will have an update for testing soon..
I'll post here once it is ready for testing.

[void]

Everything 1.5.0.1319a will now wait and try again when requested by the system to retry scanning a folder.


Everything will keep trying to scan a folder for up to one minute.
After one minute Everything will timeout and skip the folder.


You can customize this timeout with the find_first_file_retry_timeout ini setting.


Please force a rebuild from Tools -> Options -> Indexes.
Please let me know if you have any missing folders with this version.

[xxxxx]

I have a related issue, where everything 1.5.0.1319a doesn't see my network drives mapped as smb: shares (tried with `everything service: ON` and also with `everything service: OFF` + `run as admin: ON`)

ss_20220919_NUC8_PLorMG2Zzs.png (306.57 KiB) Viewed 15339 times

it used to work well with 1.5.0.1317a ( using `Everything service : ON').
with 1.5.0.1318a, I experienced multiple freezes/unresponsive behavior when trying to access network drives.

[void]

Thanks for the bug report xxxxx,

I am looking into the issue.

For now, please make sure Everything is not running as an administrator.
-If you are running Everything as an administrator, it will be unable to see your mapped network drives.

Please make sure Everything is installed correctly and running as a standard user:

• In Everything, from the Tools menu, click Options.
• Click the General tab on the left.
• Check Store settings and data in %APPDATA%\Everything.
• Uncheck Run as administrator.
• Check Everything Service. (Please make sure this is tick-checked and not square-checked)
• Click OK.
• Exit Everything (right click the Everything tray icon and click Exit).
• Restart Everything.

[xxxxx]

hmmm... so I just tried to install 1.5.0.1319a again and now it's working This is weird, I tried a couple times before, the only thing I did differently this time was to jump from 1.5.0.1317a portable to 1.5.0.1319a portable, my previous attempts were upgrading from 1.5.0.1318a to 1.5.0.1319a. (running in portable mode as standard user with everything service ON.)

[Snoopie]

Appreciate the new 1319 alpha build. I've installed it and set the index to rebuild with the default config. After about 10 hours I cancelled it as it was still showing 33% thru that particular "F" drive I've chosen as my test subject (previous version indexed the drive in 2-3 hours partially).

I edited the find_first_file_retry_timeout to 10,000 ms and pointed it to a folder within another of the main drives that I had trouble with previously getting indexed. This folder has about 320 subfolders and 2645 files within; previous index attempts failed to pickup any of the subfolders but the new 1319 version did it successfully in about 3 minutes.



I'll keep playing around with the timeout setting on the F drive test subject where I'm anticipating 500,000+ cloud files in this drive. Ideally I would want to my cloud drives scanned on a weekly basis but if it's a full day of indexing a 500k file server I'll probably have to adjust expectations once I scan the servers with 1M+ files.

Will report back

[void]

Thanks for the information Snoopie,

The current folder being indexed is shown under Tools -> Options -> Folders in the top right.

It doesn't look like it will be practical for Everything to index 500k files if FindFirstFile api is rate limited.

I am curious, does the Properties window from your screenshot take about 2 minutes to show those results?
I would expect the same amount of time it takes Windows Explorer to gather those properties as it takes Everything to index this folder.

[Snoopie]

@void - To answer your question, yes, the properties window for the folder in my previous screenshot took about 2-3 minutes to populate the file count. Matching about the time it took Everything to index the folder+subfolders.

I've played around with the FindFirstFile settings and still having some issues with the index finding all files within 4-5+ levels of subfolders within the main folder I added to in the Program options. I have it down to 5 seconds as that generally is able to index some of the larger folders within a network Drive I need within a 8-10 hour timeframe; but with mixed results.

[void]

Thanks for your reply and information Snoopie,

Sounds like Windows Explorer experiences the same rate limiting.

I don't have a good solution at this stage.



Maybe run an Everything Server on another PC (that is always on) to slowly rescan your cloud files?
Add the Everything Server index to your local Everything search client index from Tools -> Options -> Network index.

[Snoopie]

I appreciate the response and troubleshooting the last few weeks. I think you're correct that this would require an always on machine performing the index and having local clients use the server index.
I'll propose the solution to my IT department and reach out to inquire about the site license.