(1.4/1.5a) Moved folders' contents missing from index, evade 'monitor changes', 'fast rescan', etc.

[malventano]

Recently did some sorting/cleanup of a large network share and found folder sizes to be misreported as smaller than expected. Digging deeper, I noted that when moving a folder within the share, Everything (both 1.4 and 1.5a) will lose track of the contents of that moved folder. I believe this is due to how a local folder move is handled by the share (a single operation updates the inode of the folder(s), so the contents are not changed). Everything is seeing the old folder disappear (so its children are removed from the index), and the 'new' (moved) folder appears in the new location immediately afterward, but the only 'new' item is the folder (not its contents). End result is that after a folder move, Everything's index is no longer in sync with the contents of the network share. I've confirmed that the buffer is not overfilling (rescan on full buffer has no effect).

The only way to resync is to manually do a full rescan of the share (not a great workaround, especially for larger shares), but when reproducing with 1.5a I found another issue: 'Fast rescan' fails to scan the moved folder contents due to the way that the last modified dates are changed when a folder is moved. The moved folder itself retains its original date, and the folder it is moved into is the one updated to the current date. Everything sees the moved folder as having a date older than the last rescan, so it doesn't scan those branch contents, and so the moved folder contents continue to remain out of sync with the index even after the manual rescan.

Two different fixes are required to address both 'monitor changes' and 'fast rescan' features:
- For 'Attempt to monitor changes', Everything should do a partial branch scan of every newly appearing folder. Especially those appearing a short time after another folder was removed.
- For 'Fast rescan' to catch moved folder contents, the entire branch (all subdirs) of a folder with a newer modified date needs to be scanned. This is because the moved folder itself will retain the old modified date.

Hope this info helps to improve this most excellent search tool!

Allyn Malventano
Storage Technical Analyst, Intel

[void]

Thank you for the issue report Allyn Malventano,

When you move a network folder, Everything will see the following events:

ADD new-location\folder-name
DEL old-location\folder-name



Everything 1.4 will:
Miss any subfolders and files from the move.
Pickup any missed folders and files at the next scheduled rescan. (rescans occur daily)

Everything 1.5 will:
Add new-location\folder-name to a rescan queue and should pickup the moved folder, subfolder and files quickly.



It's interesting to hear Everything 1.5 is not picking up the moved folder contents for you.
What type of hardware is used to host the network share?



I am looking into an issue with fast rescan missing moved folders.
Thank you for the bug report.

[malventano]

What type of hardware is used to host the network share?

It's a debian linux smbd share. Fairly standard/simple config, but to be safe, I have reproduced on an NTFS share with identical results.

Watching 1.4 and 1.5a side by side, 1.5a does appear to be scanning some depth of the moved folder, but not all depth.

Experiment initial state (same on both 1.4 and 1.5a, performed on an NTFS share):

20220429-131844.png (30.53 KiB) Viewed 22608 times

1.4 behavior when moving 'folder' from 'location 1' to 'location 2':

20220429-131906.png (12.01 KiB) Viewed 22608 times

(only the moved folder is indexed and no depth is scanned, as expected)

1.5 result on same move operation:

20220429-131910.png (19.96 KiB) Viewed 22608 times

(scan depth appears to be stopping prematurely, only catching contents of the first layer subdirs of the moved folder)

Separately (re: 'fast rescan'), I have reproduced on NTFS shares, and in the above example, a fast rescan with 1.5a does not correct the issue (result after the move was still unchanged and the additional layers were not scanned).

All of the above produces an identical result on both NTFS and Linux-SMBD shares.

ADD new-location\folder-name
DEL old-location\folder-name

Is there perhaps some optimization that can be done, where if such a pair is received together, indexed children of the del folder can initially have their paths switched over to the add folder? This would minimize Everything's re-sorting load on larger operations and make the initial index update much faster after a move, and the add folder can still be rescanned to double-check for a coincidental add/del that wasn't actually a move.

[void]

Thank you for the hardware information malventano,

Thank you for the NTFS test information.

Everything 1.5.0.1312a fixes the issue with only partial folders being added after a move.

Everything was not building the full path correctly when rescanning.



Fast Rescan will also work as expected now.

Please force a rebuild of your network drive to pickup any partially rescanned folders in previous versions:
In Everything, type in the following search and press ENTER:
/reindex T:
where T: is your network drive.

Is there perhaps some optimization that can be done, where if such a pair is received together, indexed children of the del folder can initially have their paths switched over to the add folder?

I have put this on my TODO list.
Thank you for the suggestion.

[malventano]

Fast Rescan will also work as expected now.

I attempted to confirm this by allowing the index to become out of sync with the test folders and then performing a fast rescan, but I noted an odd configuration limit:

20220503-190327.png (4.14 KiB) Viewed 22508 times

With 'Attempt to monitor changes' disabled, shouldn't 'Fast rescan' still be allowed?
Also note there are still rescan issues carried forward into Fast rescan, as indicated below.

Everything 1.5.0.1312a fixes the issue with only partial folders being added after a move.
Everything was not building the full path correctly when rescanning.

Understood. The partial rescan did work correctly with my test folder example from before, but when I retried the original move in the live environment, the behavior remained the same as before (rescan is very fast but only goes two layers deep). I've again duplicated the live environment files on both NTFS and Linux SMBD with the same results. With further investigation, it appears the rescan fails its depth when there is more than one folder in any subdir down the rescan branch. For example, the rescan will only progress one layer past the dir with two folders present, and this issue again will evade a 'Fast rescan':
Before:

20220503-185320.png (27.16 KiB) Viewed 22508 times

(note that I added another folder 'layer 2(2)' alongside folder 'layer 2')

After 'folder' moved from 'location 1' to 'location 2':

20220503-185354.png (20.79 KiB) Viewed 22508 times

(note anything after layer 3 is now gone, and the now mismatched index will also evade a 'Fast rescan')

Is there perhaps some optimization that can be done, where if such a pair is received together, indexed children of the del folder can initially have their paths switched over to the add folder?

I have put this on my TODO list.
Thank you for the suggestion.

If you would like additional testing of this optimization once implemented, please ping me back on this thread and I can test performance on a few very large file/folder count shares.

[void]

Thank you for the feedback malventano,

With 'Attempt to monitor changes' disabled, shouldn't 'Fast rescan' still be allowed?

Yes, this is an oversight.
I will allow Fast Rescan to be toggled even when 'Attempt to monitor changes' is disabled in the next alpha update.

(note anything after layer 3 is now gone, and the now mismatched index will also evade a 'Fast rescan')

From what I am seeing, this appears to be an issue with SMB cache.
Everything sees a new folder is created (folder in location 2) from ReadDirectoryChanges.
FindFirstFile on "location 2\folder" succeeds
FindFirstFile on "location 2\folder\*" fails (either 2: path not found or 123: invalid filename)
If I wait for 5 seconds, FindFirstFile will succeed on "location 2\folder\*"


Setting HKLM\System\CurrentControlSet\Services\LanmanWorkstation\Parameters\FileNotFoundCacheLifetime to 0 fixes the issue.
Create a new DWORD value if FileNotFoundCacheLifetime does not exist.
FileNotFoundCacheLifetime



In the next alpha update Everything will wait 30 seconds until FindFirstFile on "location 2\folder\*" succeeds (if FindFirstFile on "location 2\folder" succeeds)

If you would like additional testing of this optimization once implemented, please ping me back on this thread and I can test performance on a few very large file/folder count shares.

Thank you for your offer to help with testing.

[malventano]

From what I am seeing, this appears to be an issue with SMB cache.
Everything sees a new folder is created (folder in location 2) from ReadDirectoryChanges.
FindFirstFile on "location 2\folder" succeeds
FindFirstFile on "location 2\folder\*" fails (either 2: path not found or 123: invalid filename)
If I wait for 5 seconds, FindFirstFile will succeed on "location 2\folder\*"

Setting HKLM\System\CurrentControlSet\Services\LanmanWorkstation\Parameters\FileNotFoundCacheLifetime to 0 fixes the issue.
Create a new DWORD value if FileNotFoundCacheLifetime does not exist.
FileNotFoundCacheLifetime

In the next alpha update Everything will wait 30 seconds until FindFirstFile on "location 2\folder\*" succeeds (if FindFirstFile on "location 2\folder" succeeds)

My question about this interaction would be: If 'Fast rescan' was running into the same issue, then why did subsequent manual (fast) rescans still fail to pick up the missing folder? Those rescans were done >5s after the move, yet they still missed the changes.

[void]

Fast rescan would currently miss the change because:

FindFirstFile on "location 2\folder" succeeds
"location 2\folder" is added to the index with the correct date modified.


FindFirstFile on "location 2\folder\*" currently fails (hits FileNotFoundCache)
Everything will think "location 2\folder" is empty.


On the next fast rescan update:
Everything will see the date modified has not changed for "location 2\folder"
Everything will assume "location 2\folder" is still empty.


The next update will ensure "location 2\folder\*" succeeds so Everything will never get in this out-of-sync state.

[void]

Everything 1.5.0.1313a fixes an issue with re-adding moved folders.

Everything will now wait for the SMB cache to update when re-adding moved folders.

There's still a sync issue where a parent folder has a newer cache than a child.
Changes to children could be missed if they are made between the two parent/child cache snapshots.
This should be rare.

I have put on my TODO list to bypass the SMB cache.

[malventano]

Everything 1.5.0.1313a fixes an issue with re-adding moved folders.

Retried the above example with .1313a - the issue remains as it was before (not fixed). Tried waiting several minutes for SMB caches - no effect.

I've included a zip of the example folder structure for reproduction. Extract to a share and then move the 'folder' from 'location 1' to 'location 2' and note that Everything will miss anything below layer 3 (and will evade a Fast rescan).

[void]

Thank you for testing 1313a malventano,

Thank you for the sample folder structure.
I was able to produce the issue my end.

Everything was not processing the list of folders to rescan correctly.
This issue will be fixed in the next alpha update.

[malventano]

Thank you for the sample folder structure.
I was able to produce the issue my end.

Glad to help. Does this mean the SMB cache was not actually an issue? Or perhaps the SMB cache fix will still be useful for other related cases?

This issue will be fixed in the next alpha update.

I'll keep a lookout and retest once available. If successful on the example (for both NTFS and Linux+SMBD), I will re-perform the original folder moves on some much larger production shares to torture test the fixes .

[void]

Glad to help. Does this mean the SMB cache was not actually an issue? Or perhaps the SMB cache fix will still be useful for other related cases?

SMB cache syncronization is still an issue for versions prior to 1313a.

I just did a test with your sample folder structure and the folder named "folder" was missing when this SMB cache fix is temporarily disabled.

It's only noticeable if you move a folder, then move it again within 5 seconds.
There's still possible sync issues when the SMB cache for a subfolder is older than a parent folder.

Thank you for your help with testing.

[void]

Everything 1.5.0.1314a fixes an issue with rescanning moved folders.

Everything should now correctly rescan the full sub folder structure after a folder is moved.