Any way to limit network bandwidth when indexing a network drive?

[jevery7]

When Everything is indexing a mapped network drive, the overall bandwidth usage is low, usually in the area of around 200kbps. It's a 1Gbps network, with typical sustained transfer speeds to/from the server drive of >950Mbps.
System is Windows 10 Pro 22H2.
Everything v1.4.1.1024.

But Everything's indexing of that drive inhibits virtually all other network activity on my computer. Even just something like using dir at a command prompt can take up to ten seconds before it slowly starts showing folder contents, which is normally a nearly-instantaneous action. Or opening up Autodesk Inventor's Projects listing: It has to process about 50 text-based project files, each of which is only about 10kB. This takes it maybe 10-15 seconds normally. If Everything is indexing, it can take several minutes, just to hit up some tiny text files.

Setting the everything.exe process priority to anything below Normal has no effect, which is understandable.
I assume that, while its overall bits-per-second is low, it is making a vast number of small requests, which is bogging down the computer's network.

Is there a setting or registry tweak for Everything to reduce its network-usage priority?

I also do have it set up to:
- Run as a service.
- Run on startup.
- Index at 3am.

But when I log in in the morning, it starts indexing, as if it never did the 3am indexing. So there's also that.

[void]

Thank you for your feedback jevery7,

Everything uses FindFirstFile to index your network drives.

Strange for this API to kill network bandwidth..
Do you notice high CPU usage on the client or server? high interrupts?
Might be firewall filtering? antivirus filtering?
Please check any system logs with eventvwr.
I typically see indexing of 1 million files take about 1 minute.



Enabling thread_mode_background (idle IO thread priority) may help:

• Copy and paste the following into your Everything search box:
  /folder_update_thread_mode_background=1
• Press ENTER in your Everything search box.
• If successful, folder_update_thread_mode_background=1 is shown in the status bar for a few seconds.

Enabling may make Everything take hours to index your network drive.



Please check the Everything verbose debug logs:

• Copy and paste the following into your Everything search box:
  /debug
• Press ENTER in your Everything search box.
  ---this will show a debug console---
• Copy and paste the following into your Everything search box:
  /verbose
• Press ENTER in your Everything search box.
• The currently folder being rescanned is shown in Green text.
• Is Everything slow overall? or is Everything getting stuck on a particular folder?

Please try Everything 1.5.
While Everything 1.5 still uses FindFirstFile, it enables large fetch mode which may help improve performance.



Everything will only index your network drive at 3am if Everything is currently running.
If this scheduled update is missed, Everything will rescan your network drive the next time Everything is started.

To disable this rescan on startup when a scheduled rescan is missed:

• Copy and paste the following into your Everything search box:
  /folder_update_rescan_asap=0
• Press ENTER in your Everything search box.
• If successful, folder_update_rescan_asap=0 is shown in the status bar for a few seconds.

[jevery7]

- First a note: I've used Everything for years now. Excellent program and productivity tool.
Only at some point in the past several months has this become an issue. I'm not sure if it's a Windows update that did something? I had been looking into Server Message Block (SMB) at one point; I was reading a long thread about some network performance issues that someone was having, which started in a Windows update to either 21H1 or 22H2. They were trying different SMB settings in PowerShell. That may have been unrelated though.

- CPU usage is low. Everything.exe is <1%.

- Setting /folder_update_thread_mode_background=1 looks like it did implement correctly, but I did not notice any change in behavior. Network remains sluggish during indexing.

- Everything itself is still quick and responsive. What gets slow is when I right-click on something on the network drive: It can take 30-60 seconds for the normal right-click menu to appear, until indexing is done. If indexing isn't running, right-click is almost instant.

- Debug, verbose mode: Is this log written to a file somewhere?

The vast majority of green lines are things like:
MSG: 0000000000051510 1401 0000000000000000 0000000000000000
MSG: 00000000003f147c 0113 000000000000002d 0000000000000000
or EVENT: and some more strings of hex.
Occasionally there is a filename or folder location.
I don't see it getting stuck on any specific file or folder.

- Rescan: I have Everything set to run as a service. Shouldn't that mean that it's running as soon as the computer finishes booting, even if I haven't logged in?


Thank you for all the help and pointers. I'll try out Everything 1.5 in the meantime.

[jevery7]

Everything 1.5: I'm noticing an immediate change.
When Everything 1.4 was indexing, network activity was fairly slow.
Everything 1.5: 2-4Mbps up/down with the network drive. Network activity remains responsive, as if Everything wasn't even indexing.
So, v1.5 may have solved this entirely!


"large fetch mode" - reminds me of some of the SMB settings I was trying.
Set-SmbClientConfiguration -EnableLargeMtu $true
Set-SmbClientConfiguration -MaxCmds 100


Edit: It took about 50 minutes to index 1.8 million items on the network drive.
I checked, and folder_update_thread_mode_background was still at the default of 0 for the v1.5 installation.

[horst.epp]

- Rescan: I have Everything set to run as a service. Shouldn't that mean that it's running as soon as the computer finishes booting, even if I haven't logged in?

No, the service does nothing for itself.
Its only function is to allow the client to index NTFS volumes without enhanced rights.
Indexing starts when you log in and the client is set to "Start Everything on system startup"

[jevery7]

Huh, I guess I spoke too soon.
v1.5a is showing the same behavior as v1.4.
Network activity is down below 200kbps typical, but extremely slow. Even just changing folders with the commandline can take 10-30 seconds.
As soon as all everything.exe processes are terminated, the network returns to normal.
Indexing process

Server-side, no unusual activity is detected.

The verbose debug log has lines of text streaming by fairly rapidly. As far as I can see anyway, lines show folder or filenames though.

No, the service does nothing for itself.
Its only function is to allow the client to index NTFS volumes without enhanced rights.
Indexing starts when you log in and the client is set to "Start Everything on system startup"

Ok, understood.

[jevery7]

I saw this red line in the debug log:
gelongpathname 3 x:\xxxxxxxxxx\xxxxxxxxx\xxxxxxxxxxxxxxxxxx\xxxxxxxxx\xxxxxxxxxxxxxx\xxxxxxxxxxxxxx.xxx

Then something similar showed up awhile later at a slightly different path:
getlongpathname 2 x:\xxxxxxxxxx\xxxxxxxxx\xxxxxxxxxxxxxxxxxx\xxxxxxxxxxxxxxxxxx.xxx

Not sure if the path length is important or not, but that's the number of characters in each path.

What's "getlongpathname" and that number? Red lines, so I'm guessing they're errors?


But I also saw that around the same time, Wireshark logged this.
"Find Request File: [unknown] SMB2_FIND_ID_BOTH_DIRECTORY_INFO Pattern:" and then a folder name, and
"Find Request SMB2_FIND_FULLDIRECTORY_INFO Pattern:" and then an asterisk.

The SMB2 part got my attention because this whole network slowdown problem started, as far as I recall, with a Windows Update late in 2022, and I saw reports elsewhere of issues related to some change in Server Message Block and odd slowdowns with network drives.
But Everything is the only program I've run into that causes issues, maybe because of its "aggressive" indexing behavior, versus something to do with SMB?

https://learn.microsoft.com/en-us/windo ... e-transfer (relevant?)

[void]

gelongpathname 3 x:\xxxxxxxxxx\xxxxxxxxx\xxxxxxxxxxxxxxxxxx\xxxxxxxxx\xxxxxxxxxxxxxx\xxxxxxxxxxxxxx.xxx

Everything uses ReadDirectoryChanges to monitor changes to your network drive.

Everything will look up the long path with GetLongPathName when a ReadDirectoryChanges event occurs.

Perhaps ReadDirectoryChanges is causing the bandwidth issue.
Please try disabling monitoring on this drive:

• In Everything, from the Tools menu, click Options.
• Click the Folders tab on the left.
• Select your Network drive.
• Uncheck monitor changes.
• Does the poor bandwidth issue persist?

How many changes are you making to the network drive? and how often?

[void]

Everything 1.5.0.1352a adds a read_directory_changes_get_long_path_name ini setting.

Disabling may help performance.

To disable path expansion with GetLongPathName on ReadDirectoryChanges events:

• In Everything 1.5, from the Tools menu, click Options.
• Click the Advanced tab on the left.
• To the right of Show settings containing, search for:
  read dir
• Select read_directory_changes_get_long_path_name.
• Set the value to: false
• Click OK.

[jevery7]

I tried both of these: Unchecked Monitor Changes, and then also tried read_directory_changes_get_long_path_name. (and combinations with one, the other, both, neither)
And also tried the combinations before and after terminating all everything64.exe processes and restarting Everything.

No change in network behavior.

Some progress though: I finally noticed and tried Fast Rescan.

Everything's network usage appears to remains the same while it's rescanning, about 1.0-1.5Mbps in Task Manager, but the network otherwise performs normally. Very near to full transfer speed (900-990Mbps sustained) and minimal lag.
Verbose debug logging shows a great deal of scan activity, but far fewer red messages. (Also: read_directory_changes_get_long_path_name is now back to the default value of true, and Monitor Changes is also enabled)

[jevery7]

Every time I think it's working... nope.

Fast Rescan has helped immensely, until this:

Today I transferred a large folder to the network drive, while Everything was running.
12.7GB, 6219 files, 836 folders.

The entire computer was sluggish while it was transferring, and since the transfer finished, Everything was bogging the network again.
Overall system network activity remained light, <300kbps background activity.

But just things like opening small files <1MB, or a program that needs to read a 10kB license text-file on the network drive, were all extremely laggy. That license-file reading should be <5 seconds, but it's more like 30-45 seconds.
The instant Everything64.exe is terminated, that lag vanishes.

Absolutely bizarre.

<I still think it's something to do with that SMB2/SMB3 Windows 22H2 issue. https://techcommunity.microsoft.com/t5/ ... 2#comments >

[void]

Everything 1.5.0.1353a adds the following ini settings:

find_first_file_large_fetch
folder_scan_sleep



Disabling find_first_file_large_fetch might help with bandwidth issues.

To disable find_first_file_large_fetch:

• In Everything 1.5, from the Tools menu, click Options.
• Click the Advanced tab on the left.
• To the right of Show settings containing, search for:
  find
• Select find_first_file_large_fetch.
• Set the value to: false
• Click OK.

Enabling folder_scan_sleep might help with bandwidth issues.

To enable folder_scan_sleep:

• In Everything 1.5, from the Tools menu, click Options.
• Click the Advanced tab on the left.
• To the right of Show settings containing, search for:
  sleep
• Select folder_scan_sleep.
• Set the value to: 1000
  where 1000 is the number of milliseconds to wait between folder scans.
• Click OK.

Please let me know if any of the above help with your network bandwidth issues.

[jevery7]

Thank you for this, I'll try this out.

Edit/updates, 2023-09-25:
- With folder_scan_sleep set to 1000, not unexpectedly, a network drive scan takes a long time. It was scanning for 8hrs and never finished.
- Enabling Fast Rescan in the Network Drives section seems to be what's mostly remedied this. There's still occasional slowdowns, but they're not severe.

- I'm now sure that this is an SMB2 issue, something either with Windows 10, or there was an update on the server, which is running some manner of Linux:
I did a test, transferring about 700,000 1kB files to someone else's Windows 10 computer on the same network, and it went fast, basically maxing out the 1Gbps network.
Then I tried transferring those same files to the network server. After about 30-40 seconds of high transfer speed, it abruptly dropped to just a few Mbps.

- In another test, while doing a full index of the network drive, I ran Wireshark.
For about 30-40 seconds, SMB2 transactions were happening every millisecond, maybe less. It was fast.
Then it abruptly slowed down, and SMB2 transactions were happening approximately every 4-7 seconds.

So this whole time, it may well have been a changed configuration setting on the server, or an update that affected its SMB2 handling, or a Windows 10 update that affected its SMB2 handling.
Everything simply exposed it because it's one of the few programs I regularly use that sends out so many small requests so quickly.