Slow "Querying" while compiling something

[dzenanz]

I love Everything and use it as my search too. But if I have something compiling and it uses 100% of the CPU, Everything's search slows down to a crawl. Also scrolling is slow, if not impossible. As I write computer code for a living, this happens somewhat frequently.

Compiler executable (cl.exe) has "Below normal" priority, while Everything has "Above normal". This is enforced by Process Hacker 2. But that does not help much. I am using Version 1.4.1.935 (x64). Is this a known problem? Is there a workaround?

[therube]

How many files does the compiler throw around in doing its' work?
Thinking that if it is very many, in a very short period of time, that number could be overwhelming Everything... ?
Probably some settings, unknown to me, that could be adjusted, to try to help compensate... ?

[dzenanz]

I didn't think of the number of files compiler touches. I was thinking "why doesn't Everything fight for CPU time better?"

While compiler does throw around many files by human standards (at least dozens, probably hundreds, possibly low thousands of files per second), it should not be too much by computer standards.

How would performance for this case be affected by disabling "Index date modified" and/or "Index recent changes"? Is either significant?

[NotNull]

I didn't think of the number of files compiler touches. I was thinking "why doesn't Everything fight for CPU time better?"

That will be more on how Windows handles that (= not very good). Example: yesterday my system had 100% CPU load due to an update, although I configured the responsible TrustedInstaller routine to run at the lowest priority. Result: unresponsive system.
JUst a thought: does CL.exe start other processes that are *not* running with lowest priority?

While compiler does throw around many files by human standards (at least dozens, probably hundreds, possibly low thousands of files per second), it should not be too much by computer standards.

I don't suspect that is the cause either. But to be sure, you can (temporary) disable "scanning for new files":
In the Everything search bar, type

/monitor_pause

(followed by ENTER)
To resume monitoring, type

/monitor_resume

(followed by ENTER)
(the files that were missed in that period will be picked up automatically)

How would performance for this case be affected by disabling "Index date modified" and/or "Index recent changes"? Is either significant?

From what I know about the inner workings of Everything (= not much), most CPU power is used when you have a complex filter active (or exclusions you defined for your indexes) and for sorting the results.
I doubt if index date/recent changes has much influence on CPU usage.

[dzenanz]

does CL.exe start other processes that are *not* running with lowest priority?

No. devenv.exe (Visual Studio process) launches MSBuild.exe, which launches cl.exe. cl is the leaf which does the work and consumes CPU time. But I also gave lower priority to MSBuild.

[void]

Do you observe the same issue with Everything 1.4.1.895?

I wonder if this is related to the newly added Sleep(0) which will make Everything give up its time slice every so often, perhaps too much.

[dzenanz]

I disabled "Index recent changes" earlier, it did not seem to have had much of an effect.

Now, I downloaded Everything 1.4.1.895 (it is not listed, I had to type the name), and I will let you know once I had a chance to observe it under load.

[dzenanz]

It does not look like version 1.4.1.895 made much of a difference.

[void]

Thanks for checking Everything 1.4.1.895.

Does Everything show any CPU usage in the Task Manager when compiling? -If so how much CPU Usage %?
Are you running into any low memory conditions?
What does Everything's page faults look like in the Task Manager? -It could be a zero soft page fault performance issue.
What type of CPU do you have?

Note:
Everything will show Querying... in status bar after a search takes longer than 250 milliseconds.

[therube]

(Process Hacker, double-click on Everything.exe, Performance [tab], will show a running graph of [Everything] CPU history. Similarly, Process Explorer.)

[dzenanz]

I have 32GB or RAM and 1GB swap file. If my computer runs out of memory, I get allocation errors not page faults.

During most recent incident, about half of RAM was free, page fault count for Everything barely budged, and CPU use was generally low, with some spikes going to about 50% of CPU time.

Screenshot 2019-05-21 15.25.05.png (55.67 KiB) Viewed 42063 times

[void]

Does increasing Everything's base priority help? try setting it to high?

What type of CPU do you have?

[dzenanz]

AMD Ryzen 7 PRO 1700 Eight-Core Processor (with hyper-threading enabled, so appears as 16 cores).

Giving Everything high CPU and I/O priority did not matter. It still sometimes queries for many seconds, in this instance for more than a minute.

Screenshot 2019-05-23 17.16.35.png (27.91 KiB) Viewed 42053 times

Screenshot 2019-05-23 17.15.03.png (761.02 KiB) Viewed 42053 times

[NotNull]

Did the /monitor_pause made any difference?

[void]

Could you please run Everything in debug mode:

• In Everything, type in the following search and press ENTER:
  /debug
• Perform a search while you are compiling to try and reproduce the issue.
• When Everything is stuck showing Querying... what is shown in the debug console?

[dzenanz]

/monitor_pause made Everything responsive, and /monitor_resume made it unresponsive again. The query I made was itk.*common.+s (if that matters). I sent the log to the support email address.

[NotNull]

Then there is at least some workaround (other then exiting Everything altogether): start your compiler with a script:

Code: Select all

"X:\path to\everything.exe" /monitor_pause
start /wait "" cl.exe <parameters>
"X:\path to\everything.exe" /monitor_resume

(But to be honest: I don't have a clue on how to start the compiler)

[void]

Thanks for the debug logs.

NotNull is on the right track with the disabling of the NTFS monitors.

search 'itk.*common.+s' filter '' sort 2
found 100 folders, size 65544, db search time taken: 0.024514 seconds
found 1000 files, size 65560, db search time taken: 0.252915 seconds

I can see the search query start and complete quickly.
Unfortunately, due to a design issue I can see the UI is not notified about the completed search immediately due to many changes to the disk.

I had intended these changes be processed only when Everything is idle.

Could you please try Everything 1.4.1.943
This version will make sure the UI is notified on new results before processing any changes to disk.

[collinchaffin]

I would love to see an additional feature added of the monitor pause/resume as a tray icon context menu item option - be it default or optional to allow a quick method to pause/resume without even having to open the main GUI or launch shortcuts.

Also, I find myself for this very reason when tons of file activity happens and in turn ES is actually then the culprit of taking the box to it knees, if I instead open task mgr and right click ES and force it's process priority to either below normal or low - that then even with monitoring enabled ES no longer makes the box unresponsive. Could this (or is it now) an option to always start ES with a lower priority?

Thanks!

[therube]

Can something like that be done already?
Batch file or whatever?

Seems you can't pass "slash" /settings from the command-line?
If you could, then maybe you could do something like:

> C:\> Everything.exe -set /monitor_pause -quit

and then to start it up,

> C:\> Everything.exe -set /monitor_resume -quit


And if not that, then maybe something with -config, pointing to a pre-configured "Everything_PAUSE.ini"?

[collinchaffin]

Yes but as I said, first of all this is always while already running, and not at new startup, and even if a shortcut could - it would be nice to be able to toggle this monitoring on and off via the tray icon. As for the process priority - that too COULD be performed using other methods AT STARTUP or I guess I could whip up a C# or Posh to change the existing process priority - but again would just be a nice enhancement to have to be able to set the ini option to always start at XXXX process priority.

[dzenanz]

With 1.4.1.943, it is weird now. The status bar count and scrollbar size update quickly, but the actual list doesn't always update together with it. Here is an example, which still shows the list for itk.*common.+ex after I have added p to the search string.

Screenshot 2019-05-30 09.51.42.png (183.89 KiB) Viewed 42010 times

And another example, which still shows the list for exp after I have changed the search string to exe

Screenshot 2019-05-30 09.53.15.png (199.5 KiB) Viewed 42010 times

[void]

Thanks for the screenshots.

A possible solution would be to force Everything to render the result list. However, this would still cause issues when rendering newly acquired extended file information (such as icons, sizes and dates).

Could you please try excluding your build directories:

• In Everything, from the Tools menu, click Options.
• Click the Exclude tab
• Click Add Folder.
• Select a build folder (eg: the folder containing custombuild.write.1.tlog seems to be the biggest culprit) and click OK.
• Repeat for any other build folders.
• Click OK.

The next major version of Everything will update in the background (ie: it will not cause the UI thread to lock up).

[dzenanz]

Excluding the build directories isn't really feasible. There are 22164 files named custombuild.write.1.tlog, and even if not counting multiple targets within a single project, I still have at least dozens and probably hundreds of build folders. Also, I sometimes search for existence of specific files in those build folders.

I will wait for the next major version of Everything (1.5.x?). Any estimate when it might arrive? In the meanwhile I will just live with this limitation.

Thanks for your help with this specific issue, and also for making and maintaining Everything!

[NotNull]

I will wait for the next major version of Everything (1.5.x?). Any estimate when it might arrive?

No release date known yet. Might take a while as there are quite a few things to do ...

[collinchaffin]

Any chance a future version might have a tray function to toggle monitoring on/off, and #2 is there currently an ini value i'm not aware of to tell ES to always start with a certain process priority? If not, could you consider that as well as it those two should not take too excessive code requirement (since the monitor on/off functions already are written just not tied to a gui menu) and process priority is almost no code other than reading the ini option and setting that value at startup.

As usual, thanks!

[void]

An Index menu might be an option in a future release which would have a toggle monitoring options.
I'll consider a tray context menu item too, thanks for the suggestion.

#2 is there currently an ini value i'm not aware of to tell ES to always start with a certain process priority?

No.
I'll consider an option to do this from es directly as a command line or ini option, thanks for the suggestion.

For now, please try:
START /high es.exe
https://ss64.com/nt/start.html

[NotNull]

I think collinchaffin's ES = Everything Search (=everything.exe).
And not the ES.exe command-line tool. It sent me in the wrong direction too.

[void]

START would also work the same with Everything.exe or any program.

Please try:
START /high Everything.exe

However, you'll need to make sure that Everything was not previously running, or force Everything to exit and restart with high priority:
START /wait Everything.exe -exit
START /high Everything.exe

[NotNull]

Everything has to be in "" to be regarded as the window title; otherwise it is seen as a command.
With that, it becomes (not tested):

Code: Select all

START "Everything" /wait Everything.exe -exit
START "Everything" /high Everything.exe

[void]

Everything has to be in "" to be regarded as the window title; otherwise it is seen as a command.

Updated, thanks.