Latest entries from Linux are invisible in Windows until Force Rebuild

[t42]

Hi,
The fact is that Everything is the one and only program for which I boot Windows.
Changes which are made in Linux on NTFS partitions are visible in Windows with Everything 1.4.x only after 'Force Rebuild'.
That is in Version 1.4.1.877 (x64), also in 1.4.0.713b and 1.4.1.809b.
It was not so in Version 1.3.4 :
Version 1.3.4.686 (x64): after start starts scanning all partitions & shows new files, which appeared in Linux session.
Version 1.3.4.679b (x64): after start shows the new files after short DB update without full scan.
Please advise Is that by design and is the 'Force Rebuild' my only option?

It is the same in Windows only situation. Just create some new file at it will be immediately identified by current instance of Everything, close this instance and open another one with its own db which was updated shortly before that, and new file will be missed.

In all version related setting are switched on, Everything runs under administrator, or as a service, tested in Windows Server 2012R2, Windows Server 2008R2 and Windows 8.1.

[NotNull]

Assuming you are using the following settings:
- Under folder indexing (Menu:Tools > Options > Indexes > Folders ) you added your Linux based SMB share (something like M: or \\Server\Share )
- Here you also enabled Attempt to monitor changes
- And you configured a sensible re-scan schedule

And still don't get updates after a re-scan took place?

What happens if you manually select the share and click the Rescan now button?


The "Attempt to monitor changes" monitors for directory changes made from the system where Everything is running and while Everything is running.
@void describes this mechanism much better here: viewtopic.php?t=6231

My first guess is: you are running Everything as an administrator and make your changes to the networked filesystem with your regular user account. Those two run in different desktops (Explorer shells). Chances are that monitoring doesn't work beyond the boundaries of the current shell, so (administrator) Everything.exe can't pick up changes made under your user account. In that case you'll have to waait for the next rescan to get Everything in sync again.


Long story short: Try running Everything with your user account instead of the Administrator account. For Folder Indexing you can even run Everything under a restricted account (no admin rights needed)

After reconfiguration, as a test: copy a file from your Windows machine to your Linux filesystem. It should be seen by Everything straightaway.

[t42]

Thank you for the very comprehensive answer and for the link to the What does "monitor changes" actually monitor discussion, which is a revelation. It made the problem clearer and easier to manage. Now I see that my question was based on a poor understanding ...
Also, probably there was some change between 1.3 and 1.4 versions that needed some adjustment from my side ...

[t42]

Still it is unclear if ntfs-3g driver modifies UsnJrnl while making FS changes. It seems that they are using it at least:
NTFS-3G Safe Read/Write NTFS Driver / ntfs-3g / [924232] /libntfs-3g/mst.c

Simplified test:
Open Linux and create file on NTFS partition.
Close Linux and start Windows, start Everything.

New file is shown in the search results shortly after start:
Everything-1.3.1.636b.x64
Everything-1.3.2.649b.x64
Everything-1.3.3.658b.x64
Everything-1.3.4.679b.x64
Everything-1.3.4.686.x64

New file is not shown after start in the search results
Everything-1.4.0.705b.x64
Everything-1.4.0.713b.x64
Everything-1.4.1.809b.x64
Everything-1.4.1.877.x64

All versions are run as an administrator under an administrator account. Same results with Everything service.
Enable USN Journal ON, Monitor changes ON, Include UsnJrnl in recent changes makes no difference.

[NotNull]

Everything can only connect to the USN jounral (and the $MFT) on locally attached storage.
On network storage, like your Linux Samba share or even Windows shares, you use the SMB protocol to connect from your Everything running Windows machine. SMB doesn't support direct access to remote MFT and USN journal.

Don't know if or why earlier versions of Everything seem to do a re-scan upon startup. I'm only using Everything for something like 4 months.
MAybe more experienced people can help you out here.

There is a setting folder_update_rescan_asap=1 which causes Everything to re-scan immediately after start if it missed a schedule. But default that's already set to 1.

[t42]

Everything can only connect to the USN jounral (and the $MFT) on locally attached storage.

That's true. But "create file on NTFS partition" in my comment above is about Linux's local storage, and about using afterwards Everything on the same set of discs locally, not with Everything's HTTP server. In this case we are using NTFS indexes not Folder indexes. My guess is that if Linux's ntfs-3g driver doesn't modify the USN, Everything may miss changes as there is no trigger for the version 1.4 to rescan but there is the one in the version 1.3.

[NotNull]

Everything can only connect to the USN jounral (and the $MFT) on locally attached storage.

That's true. But "create file on NTFS partition" in my comment above is about Linux's local storage, and about using afterwards Everything on the same set of discs locally, not with Everything's HTTP server. In this case we are using NTFS indexes not Folder indexes. My guess is that if Linux's ntfs-3g driver doesn't modify the USN, Everything may miss changes as there is no trigger for the version 1.4 to rescan but there is the one in the version 1.3.

Allright, I missed the dual-boot part ...
You could check if Linux actually uses and fills the USN journal by:
- Boot Linux
- create a new file DummyFile.txt on the NTFS filesystem
- Boot Windows
- Log in with administrative crdentials
- Check if this new file is in the USN journal:

Code: Select all

fsutil usn readdata "X:\RandomFolder\DummyFile.txt"

[t42]

>"You could check if Linux actually uses and fills the USN journal by ... fsutil usn readdata "X:\RandomFolder\DummyFile.txt"


It seems that I can't extract \$Extend\* files directly from Linux due to their presentation with 0 length and probably need WinPe environment for that purpose. So, thank you for your suggestion. The USN journal contains all relevant data about Linux-created file immediately after login in Windows. And still no luck from Everything-1.4.1.877.x64.

[NotNull]

It seems that I can't extract \$Extend\* files directly from Linux due to their presentation with 0 length

On Windows it is also presented as a 0 byte file. That's because the real data is in an ADS (altenative data stream).
On Windows, I copied the USN journal to the ' regular' filesystem to make it easier to handle.
Using CMD.exe I get:

Code: Select all

C:\temp>dir /a $*
 Volume in drive C has no label.
 Volume Serial Number is D020-C24C

 Directory of C:\temp

01/04/2016  01:43 PM                 0 $UsnJrnl
               1 File(s)              0 bytes
               0 Dir(s)  460,542,763,008 bytes free

C:\temp>dir /a /r $*
 Volume in drive C has no label.
 Volume Serial Number is D020-C24C

 Directory of C:\temp

01/04/2016  01:43 PM                 0 $UsnJrnl
                         8,788,094,952 $UsnJrnl:$J:$DATA
                                    32 $UsnJrnl:$Max:$DATA
               1 File(s)              0 bytes
               0 Dir(s)  460,542,500,864 bytes free

C:\temp>

So, the data of the USN journal is in the $UsnJrnl:$J stream.
Using Linux, the ntfs-3g driver can handle datastreams (otherwise there would be no new entries in the USN journal). Question is how you can access that. My guess is that you can use the same syntax (cat $UsnJrnl:$J | more , for example), but probably the ntfs-3g driver man-pages can tell you more.

Still, strange that Everything doesn't pick up on these changes in the journal ...
What are your settings for this volume under Menu:Tools > Options > Indexs > NTFS ?

[t42]

What are your settings for this volume under Menu:Tools > Options > Indexs > NTFS ?

Currently all relevant keys are on. I tried all possible combinations without any effect.

[void]

Could you please try running Everything in debug mode after making a change to this volume under linux:

• Make sure Everything does not start on system startup (Tools -> Options -> General -> Start on system startup).
• Make a change to the NTFS volume from linux.
• Reboot to Windows and run Everything manually with the -debug-log command line option.
• Wait for Everything to load and update the indexes.
• Please send your %TEMP%\Everything Debug Log.txt to support@voidtools.com

Not much has changed to the USN journal monitors between Everything 1.3 and 1.4.
The only difference between the two is Everything 1.4 does error checking, whereas Everything 1.3 does not.

Look for the following lines in the debug output:
read NTFS usn journal failed <?>
read usn journal: bad cb <?>
read usn journal: record overflow
read usn journal: bad file name offset <?>, length <?>
read usn journal: bad record length <?>

Debug logging help

[t42]

Thank you very much for the reply. I just sent email with a debug log attached. Also another Windows have been installed on the same hardware to exclude Windows from the 'dead zone' picture. A file was created on the NTFS volumes from each one.
*linux.txt - zero results
*windows.txt - all instances found

But there is the difference in the USNjrnl - USN line created in the jrnl from Linux is always zero!

Code: Select all

Usn              : 0x0000000000000000

Code: Select all

C:\Users\Administrator>fsutil usn readdata "h:\###\171022-1543=windows.txt

Major Version    : 0x3
Minor Version    : 0x0
FileRef#         : 0x00000000000000000003000000002e37
Parent FileRef#  : 0x00000000000000000003000000002e3d
Usn              : 0x00000000000020e0
Time Stamp       : 0x0000000000000000 00:00:00 1601-01-01
Reason           : 0x0
Source Info      : 0x0
Security Id      : 0x0
File Attributes  : 0x20
File Name Length : 0x2e
File Name Offset : 0x4c
FileName         : 171022-1543=windows.txt

C:\Users\Administrator>fsutil usn readdata "h:\###\171022-1543=linux.txt

Major Version    : 0x3
Minor Version    : 0x0
FileRef#         : 0x00000000000000000003000000002e45
Parent FileRef#  : 0x00000000000000000003000000002e3d
Usn              : 0x0000000000000000
Time Stamp       : 0x0000000000000000 00:00:00 1601-01-01
Reason           : 0x0
Source Info      : 0x0
Security Id      : 0x0
File Attributes  : 0x20
File Name Length : 0x2a
File Name Offset : 0x4c
FileName         : 171022-1543=linux.txt

Not much has changed to the USN journal monitors between Everything 1.3 and 1.4.

Maybe I shouldn't compare 1.3.x and 1.4x, sorry for that, because after manual start my Everything 1.3.4.686 each time starts a full scan of all partitions.

[void]

Thanks for the debug logs.

Usn : 0x0000000000000000

There is no USN entry for 171022-1543=linux.txt.

Is there a way to turn USN Journal logging on in Linux?

Everything 1.3 would work because of the forced rebuild.

You can manually mark an index as "out of date" in Everything with the following search command:
/reindex <drive>

For example, to force Everything to rebuild the indexes for your H: drive only:

• In Everything, type in the following search and press ENTER:
  /reindex h:

[t42]

Is there a way to turn USN Journal logging on in Linux?

It seems that there is no way to turn it on:
https://www.tuxera.com/community/ntfs-3g-manual/#6

Still the reindex method which you kindly suggested will resolve the problem.
Thank you for your advice. It is greatly appreciated!