High memory / RAM usage when indexing docker image files

[gregor.al]

First I would like to thank you for creating such a great tool.
I don't use it every day, but when I do it does what it is supposed to do.

I have identified a problem when Docker images are present on the system.
When indexing the: "C:\ProgramData\Docker\windowsfilter\" folder Everything never completes.

Indexing this folder(s) never completes. ( tested on two of my machines )
This probably depends on the docker images that are cached there.

If left unchecked Everything.exe will consume all available memory ( it was at around 60 GB this morning ).

My suggestion would be to exclude docker image/volume folder ( "C:\ProgramData\Docker\windowsfilter\*" ) by default.
If this folder is excluded, then memory usage stays reasonable at around 600 MB for 2.5 M files ( in my particular case ).

I have 32 GB of physical memory and 50 GB swap file.
If I allow Everything to scan the: "C:\ProgramData\Docker\windowsfilter\" folder, it will never complete this scan and consume over 60 GB of memory.

It would be nice if there was an option to limit the maximum amount of memory used by the app.
Everything could then pause indexing and ask the user if indexing should continue ( preferably the user could enter a new memory limit ).

I have used /debug and /verbose:

Version.info: Everything, Version 1.4.1.877 (x64), Windows NT 6.2, Processors 4, IsAdmin 1, AppData 0, Service 0

If I turn on verbose it fills the debug window with messages like this one:
scan C:\ProgramData\Docker\windowsfilter\c8a24937d9f3826ef9a0ce653b4309757e5101b161bbf356b271e44f0f4b4c97\Files\Windows\WinSxS\amd64_netrass.inf.resources_31bf3856ad364e35_10.0.18362.1_en-us_4e11a3c2fae813bf\*

When I do want to search for files inside the docker file "swamp", allow Everything to index this folder.
Using /debug and /verbose I see that "new results" are sent in about 5 to 10 second intervals:

Code: Select all

new results
...
update index C:
...
updated C: in 0.000613 seconds
processed usn records in 0.000483 seconds
DB_WAIT: _db_ready_proc waiting...
DB_WAIT: _db_ready_proc waited 0.000055 seconds
new results
update index C:
...
updated C: in 0.000640 seconds
processed usn records in 0.003218 seconds
DB_WAIT: _db_ready_proc waiting...
DB_WAIT: _db_ready_proc waited 0.000082 seconds
new results
...
update index C:
...
updated C: in 0.003631 seconds
processed usn records in 0.002812 seconds
DB_WAIT: _db_ready_proc waiting...
DB_WAIT: _db_ready_proc waited 0.000173 seconds
new results
...

But if I forget to close the application after I'm done, it will consume all available memory and cause other applications to crash.

Here are three screenshots that show the memory usage going to 14.6 GB.
This continues and will grow to 60 GB or more if not interrupted.

After most of the virtual memory is used other apps start crashing.

I understand the issue and I don't expect you to fix this ( other than maybe exclude the docker folder by default ).

There is a workaround ( exclude "C:\ProgramData\Docker\windowsfilter\" folder from index ).

I hope this information might help others if they encounter this issue.

Best regards,
Gregor

Everything at 6GB

Everything at 6 GB memory usage . 2021 04 13.png (84.49 KiB) Viewed 25956 times

Everything at 8.5GB

Everything at 8.5 GB memory usage . 2021 04 13.png (83.38 KiB) Viewed 25956 times

Everything at 14.6GB

Everything at 14.6 GB memory usage . 2021 04 13.png (81.61 KiB) Viewed 25956 times

[gregor.al]

I have also tried with version: 1.5.0.1253a.x64
The behaviour of this version is similar, but not the same.

Observations:
- indexing does not seem to complete
- memory usage is still growing indefinitely, but is climbing much slower

The docker images in question are composed of 20 plus layers.
I believe this means there can be 20 layers of symbolic links to files in a parent layers.

This is an example of what the app is indexing:

Code: Select all

scan C:\ProgramData\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\297ac60d14d3b6c66fcc6327477752a022a3c0512ac3e7427bc4be2f940bbea7\Files\Windows\WinSxS\wow64_microsoft-windows-d..rformancemonitoring_31bf3856ad364e35_10.0.18362.1_none_59d98bf7a146bfef

The above looks to be a scan of a single filename.
Seems like a circular reference ( or something of similar sort ) to me.

The "filename" above is over 3569 characters long and one "sub-folder" is repeated many times.
( 067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter )

It appears that:

Code: Select all

"C:\ProgramData\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter"
refers to:
"C:\ProgramData\Docker\windowsfilter\"
or:
"C:\Users\All Users\Docker\windowsfilter"

This does not appear to be a problem with the number of actual files, but rather a problem with symbolic link resolution.
It appears the indexing routine gets caught in a symlink circle.

Another folder is also problematic: "C:\ProgramData\Microsoft\Windows\Containers\BaseImages\"

Code: Select all

scan C:\ProgramData\Microsoft\Windows\Containers\BaseImages\66a56fbb-e030-4e5c-8c64-5152aef2d9eb\BaseLayer\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\055e28a7e56023f7c5e217105f66cf44f6be396fa32479e32c77c3bba12fa8e1\Files\UpgradeToolKit
scan C:\ProgramData\Microsoft\Windows\Containers\BaseImages\66a56fbb-e030-4e5c-8c64-5152aef2d9eb\BaseLayer\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\10220e9648692ba2bcdc1ad6cf1a63f20e632ee1005ba0a140dfe621a8ce61b5\Files\Windows\WinSxS\amd64_microsoft.packagema..provider.powershell_31bf3856ad364e35_10.0.18362.1_none_dd6f9a7e673153fb
scan C:\ProgramData\Microsoft\Windows\Containers\BaseImages\66a56fbb-e030-4e5c-8c64-5152aef2d9eb\BaseLayer\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\067ae320d0ca27f839a883a66fee2c398f832cb56bb139e82d6c12911fb78eb3\Files\Users\All Users\Docker\windowsfilter\0b2de2a3eb5c4a24910a4af02ea34a63d54fd28d72066fa1287b3cb3b2c99e80\Files\Users\All Users\Docker\windowsfilter\585976eecb23c9a80e1eec12607643d16ee27d8d91e66bad29a4891a6a158d98\Files\Windows\WinSxS\wow64_microsoft-windows-i..l-keyboard-00040c00_31bf3856ad364e35_10.0.14393.0_none_ab41d0a479ae3742

The new version appears to consume much less memory per file ( total memory used is in MB not in GB ):

Everything 1.5.0.1253a at 554 MB memory usage

Everything 1.5.0.1253a.x64 at 554 MB memory usage . 2021 04 13.png (17.08 KiB) Viewed 25935 times

Hope this additional info is helpful.
Best regards,
Gregor

[gregor.al]

Theere are four problematic docker folders:

Code: Select all

C:\ProgramData\Docker\windowsfilter\
C:\ProgramData\Microsoft\Windows\Containers\BaseImages\
C:\Users\All Users\Docker\windowsfilter\
C:\Users\All Users\Microsoft\Windows\Containers\BaseImages\

Version: 1.5.0.1253a.x64
This version appears to have an issue when sorting filenames.
It keeps sorting, and sorting, and sorting, ...

I have attached a partial copy of debug messages.

Everything 1.5.0.1253a.x64 . partial verbose messages . 2021 04 13.txt.zip

debug messages

(3.26 KiB) Downloaded 640 times

[gregor.al]

UPDATE: There are potentially five problematic docker folders that users experiencing this issue should add to excluded folder list:

Code: Select all

C:\ProgramData\Docker\windowsfilter\
C:\ProgramData\Docker\image\windowsfilter
C:\ProgramData\Microsoft\Windows\Containers\BaseImages\

C:\Users\All Users\Docker\windowsfilter\
C:\Users\All Users\Microsoft\Windows\Containers\BaseImages\

[therube]

What is "Docker"?
What creates Docker?

Likewise /Containers/BaseImages/?

I gather this is on Windows 10?

What does the scan command do?


(When I look on a Win10 machine, I don't see Docker at all, nor any BaseImages [subdirectory to Containers].)



"A Docker image is a file, comprised of multiple layers, that is used to execute code in a Docker container. An image is essentially built from the instructions for a complete and executable version of an application, which relies on the host OS kernel. When the Docker user runs an image, it can become one or multiple instances of that container.

Docker is an open source OS-level virtualization software platform primarily designed for Linux, Windows and MacOS."
https://searchitoperations.techtarget.c ... cker-image


(I'm still not quite sure...)


Are you running Everything within this "Docker" environment or outside?
(I seem to recall, at times, different softwares that had issues running into loops when run in a Sandboxie environment, needing to exclude particular directories, or Windows Registry trees, from being read...)

[gregor.al]

A bit more information about the environment where Everything might run into problems.

My environment:
- Windows 10 64 bit is the host
- I have Docker Desktop installed and I have some docker images based on Windows cached ( pulled from docker repository )
- Everything is indexing files on the host

The issue is not directly related to docker and it might occur in other situations as well.

What is "Docker"?

Docker is an application or software package that enables running multiple containers on a single machine.

Containers are lightweight Virtual Machines, that have no graphical user interface and are usually accessed over a network connection.
Containers are often used to serve web pages or web services.
Containers are are also used for development, which is why I have it installed on my development machine.

What does the scan command do?

scan is not a command, but rather a log entry from Everything.exe.
It shows detailed information about what Everything application is doing behind the scenes.

You can view what Everything is doing by entering the following two strings into the Everything search box:

Code: Select all

/debug
/verbose

I mentioned docker in the description, because docker uses a lot of symlinks "under the hood".
So there is a high chance that users that use docker and everything might run into issues.

The core of the issue is related to the fact that there is a symlink that points "backward" to one of its parents.

There was and there still is a similar SymLink still in use inside modern windows:

Code: Select all

"C:\Users\some-user\AppData\Local\Application Data\"     is a symlink to     "C:\Users\some-user\AppData\Local\"

This is why microsoft has ( by default ) hidden and prevented listing the contents of:

Code: Select all

"C:\Users\some-user\AppData\Local\Application Data\"

But for compatibility with older software it is still possible to open any of the files and subfolders inside that folder:

Code: Select all

"C:\Users\some-user\AppData\Local\Application Data\Vendor\"

Try to open any of the following folders and see that each of these folder contains the same files:
( replace some-user with your actual username )

Code: Select all

"C:\Users\some-user\AppData\Local\Microsoft"
"C:\Users\some-user\AppData\Local\Application Data\Microsoft"
"C:\Users\some-user\AppData\Local\Application Data\Application Data\Microsoft"
"C:\Users\some-user\AppData\Local\Application Data\Application Data\Application Data\Microsoft"
"C:\Users\some-user\AppData\Local\Application Data\Application Data\Application Data\Application Data\Microsoft"
"C:\Users\some-user\AppData\Local\Application Data\Application Data\Application Data\Application Data\Application Data\Microsoft"
"C:\Users\some-user\AppData\Local\Application Data\Application Data\Application Data\Application Data\Application Data\Application Data\Microsoft"
...

Microsoft's own tool ( robocopy ) could have issues with the above SymLink / Junction if:

Code: Select all

Everyone DENY Read permission is not set on this folder.

Windows 7 infinite loop while using Robocopy
https://answers.microsoft.com/en-us/win ... 57e4d27232

You can view the junction mentioned above with:

Code: Select all

cd %localappdata%
dir /a

View Junction information

DIR slash a . Application Data Junction . 2021 04 13.png (7.78 KiB) Viewed 25895 times

Code: Select all

cd %localappdata%
cd "Application Data"
cd "Application Data"
cd "Application Data"
cd "Application Data"
cd "Application Data"

More "Application Data" beauty

CMD Application Data Junction . 2021 04 13.png (11.81 KiB) Viewed 25895 times

Best regards,
Gregor

[void]

Thank you for the detailed post Gregor,

I have put on my TODO list for the next build to add an ini setting to not follow reparse points.

I will look into detecting and ignoring circular references with FindFirstFile (scanning folders).

[void]

I have added an ini option to not follow reparse points in Everything 1.5.0.1254a.

To enable this setting:

• Completely exit out of Everything. (Right click the Everything system tray icon and click Exit)
• You may need to terminate Everything with the Task Manager if it is stuck indexing.
• Open your %APPDATA%\Everything\Everything.ini
• Change the following line:
  follow_reparse_points=1
  to:
  follow_reparse_points=0
• Save changes and restart Everything.

I still have on my TODO list to add detection for circular reparse points.

Everything 1.5.0.1254a also fixes an issue with Everything getting stuck sorting filenames.

[gregor.al]

Everything 1.5.0.1254a also fixes an issue with Everything getting stuck sorting filenames.

I have tried the new version and the sorting issue is indeed fixed. ( works on my machine )

I have added an ini option to not follow reparse points...

I haven't used this option, because I still want everything to follow reparse points ( just not recursively ).
Instead I have opted to manually exclude all folders that were causing the issue.

Observations:
Version 1.5.0.1254a has flat memory consumption behaviour.
In my environment, after the initial indexing burst, allocated memory is at 239 MB which is excellent.
I have left it running for an hour and it stays at 239 MB.

I can try additional things if needed and my environment allows it.

Best regards, Gregor