File Size Not Registered On New File Creation

[therube]

i ran losslesscut 3.47.1, sandboxed (sandboxie), CONCAT'ing 5 files,
total size 700 MB
the concat'd output file sizes were not picked up by Everything (size column was blank)

11 other attempts, outside of sandbox, using other utilities; ffmpeg, avidemux, all displayed sizes

not aware particularly of sandboxed item size not turning up (of late)
a handful of other (well 1 hand + 1 finger, so 6) files (from some time ago, in the same output directory)
(a wee bit smaller, 46 MB, & unsure how created) all showed sizes as expected

not concerned about the lack of sizes (& i'm sure a Refresh would pick them up)
just pointing out (as i've not seen anything like this in some time)

"total size 0"

just noting in case you happen to find something of interest in the log

Code: Select all

found 0 folders with 0 threads in 0.000001 seconds
found 2 files with 4 threads in 0.106921 seconds
SET SORT 1
set sort 1 ascending 1 is valid 1
already sorted
finished sort, time taken 0.001871 seconds
total size 0, calculated in 0.000003 seconds

vs.

Code: Select all

found 8 files with 1 threads in 0.000031 seconds
SET SORT 1
set sort 1 ascending 1 is valid 1
already sorted
finished sort, time taken 0.002409 seconds
total size 48562716, calculated in 0.000002 seconds

https://www.videohelp.com/software/LosslessCut

[void]

Thanks for the bug report therube,

I couldn't reproduce the concatenating issue my end (losslesscut 3.47.1) on Windows 10 1903 x64
The concatenated output showed up fine in Everything with the correct size.



I wonder if this is related to the short filename used in the USN Journal issue.
Do you have any short filenames with the correct size in your index? (try search for ~)

Try right clicking the file in Windows Explorer, copying the size and then searching in Everything for a file with a matching size, eg:

size:1234567

Are you able to reproduce the issue? -If you haven't closed Everything yet, try terminating Everything and relaunching Everything in debug logging mode:

Everything.exe -debug-log

This will reprocess the USN Journal, which may include the USN changes to your concatenated file.



Maybe this has something to do with Sandboxie and the indexing of your output folder?
How is the "C:\Local\SANDBOX\CLEAN\drive\C\out" folder indexed? (eg: NTFS index of C:)

[therube]

How is the "C:\Local\SANDBOX\CLEAN\drive\C\out" folder indexed? (eg: NTFS index of C:)

Everything itself is run outside of the sandbox, so all that anything "sandboxie" is to Everything is simply a file/directory.
I.e., Everything sees: C:\Local\SANDBOX\CLEAN\drive\C\out where from within the sandbox itself Sandboxie only knows C:\out.

[void]

What version of Sandboxie? (Plus or Classic?) I would like to do some testing my end.

[therube]

Oh, definitely "classic". As in, SandboxieInstall-533-3.exe classic .
(Never trusted anything after that. And then once they [Sophos] closed up shop, never really followed along with the [open source] changes, so have never used the current versions.)

---

> Do you have any short filenames with the correct size in your index? (try search for ~)

All SFN show a file size.
(More accurately, any file name that contained a ~ also had a file size display.)


> Try right clicking the file in Windows Explorer, copying the size and then searching in Everything for a file with a matching size,
> eg: size:1234567

Actually, I could have done that (after running the CONCAT again) but didn't.
I'll see if I can't... (File size, newly created should end up the same.)


> Are you able to reproduce the issue?

No.
Ran the CONCAT again, 2x actually (today), & this time Everything picked the file size as expected.
(I did specifically run hash check against my Flash drive backed up files during this time, but still the size displayed.)


> -If you haven't closed Everything yet, try terminating Everything and relaunching Everything in debug logging mode:

I hadn't closed, but I also no longer had the CONCAT'd file [that did not show a size] (until I reran the CONCAT,
but at that point, today, file size was showing, so...).

---

I most likely had backups ongoing, originally, from my HDD to a slow USB Flash Drive (though the Flash drive was not monitored by Everything). And maybe that, along with all else that I was doing ? contributed to the size not turning up?

[void]

Thanks for the information therube,

Sandboxie x86 or x64?

All SFN show a file size.

Any short filenames with a size around 700MB that might be your concatenated file?

[therube]

Never even considered that, but Sandboxie is x64.
(Actual installer is x86, but I gather during the install it throws in 64-bit components.)

Any short filenames with a size around 700MB that might be your concatenated file?

(At the time) I didn't "loose" the file. Only the file size did not display (was NUL).

Or are you thinking that there could exist some 700MB file that I'm unaware of that might be this file?


(And with that, I'll throw more into the mix [which was actually going to be its' own topic, but then I kind of saw similarities, so...)

[therube]

i was hard deleting (through Windows, via Duplicate Cleaner) 12K files
on my USB Flash Drive (not indexed by Everything)
& it sure /appeared/, that at some point,
Everything (maybe some other programs too, not quite sure?), were "busy"

& about the time deletions stopped
so access returned to Everything

*busy, i simply couldn't access it by clicking its' taskbar icon
(the click was ineffectual)

ah, during that time, Duplicate Cleaner Pro 5
may very well have been updating (writing to) its' Log file:

...
Delete files: K:\FFS ----------------------------------- BAD, DTCAD23, only\ETC2\BASIS\OTHER+OLD\OLD\BASIS220\OLD\PR12SM
Delete files: K:\FFS ----------------------------------- BAD, DTCAD23, only\ETC2\BASIS\OTHER+OLD\OLD\BASIS220\OLD\PR12U
Delete files: K:\FFS ----------------------------------- BAD, DTCAD23, only\ETC2\BASIS\OTHER+OLD\OLD\BASIS220\OLD\PR13
Delete files: K:\FFS ----------------------------------- BAD, DTCAD23, only\ETC2\BASIS\OTHER+OLD\OLD\BASIS220\OLD\PR13D
Delete files: K:\FFS ----------------------------------- BAD, DTCAD23, only\ETC2\BASIS\OTHER+OLD\OLD\BASIS220\OLD\PR13F
Delete files: K:\FFS ----------------------------------- BAD, DTCAD23, only\ETC2\BASIS\OTHER+OLD\OLD\BASIS220\OLD\PR13P
Delete files: K:\FFS ----------------------------------- BAD, DTCAD23, only\ETC2\BASIS\OTHER+OLD\OLD\BASIS220\OLD\PR13S
Delete files: K:\FFS ----------------------------------- BAD, DTCAD23, only\ETC2\BASIS\OTHER+OLD\OLD\BASIS220\OLD\PR14
Delete files: K:\FFS ----------------------------------- BAD, DTCAD23, only\ETC2\BASIS\OTHER+OLD\OLD\BASIS220\OLD\PR15
Delete files: K:\FFS ----------------------------------- BAD, DTCAD23, only\ETC2\BASIS\OTHER+OLD\OLD\BASIS220\OLD\PR16M
Delete files: K:\FFS ----------------------------------- BAD, DTCAD23, only\ETC2\BASIS\OTHER+OLD\OLD\BASIS220\OLD\PR16MI
...

& if Everything were seeing (not the deletes, but) the change in the Log file... during that period...
(it took 13:42 from the OK to do the deletes, till the deletes finished. & along with writing to the log,
additionally a separate .sqlite .db was updated, so likely update of .shm .wal files too.
- probably ? that part happened after the deletion had completed, i'm thinking)

(all of my files show sizes, from 0 to ..., i.e., none have "nul" values.
wonder had i been something else "heavy" during this particular time, like CONCAT of video files,
sandboxed, if that might have then turned up a file with a NUL size?

& come to think of it fcp.exe (FastCopy command line) creates a log too...
[how stupid of me, to not think to use Everything to /find/ the log file - well that surely was quicker ]
& /that/ is stored on "C:", so seen by Everything, & if Everything were busy catching up with it's changes...

TotalRead = 26,782 MiB
TotalWrite = 26,782 MiB
TotalFiles = 12,726 (946)
TotalTime = 45:53
TransRate = 10.2 MB/s
FileRate = 4.62 files/s

i wonder if i might have been doing a fcp.exe during the same time i was doing CONCAT ?)


OK, so lets blow that idea out of the water
(at least attempting to duplicate the issue doing the same, failed)

though... i am unable to (now) interact with the Duplicate Cleaner program, or the (Windows) Delete
dialog, so... maybe, maybe the same sort of thing happened before
- only with a different set of programs (where Everything happened to be affected) ?
(Most likely, DC sends the files to delete to Windows, & waits... Actual deletion, even though these were hard deletes, is timely. Deletes finally finish. Return goes back to DC. DC writes the deleted file list to Log, & then updates .sqlite. All of this is relatively timely. Finally actual [user] control goes back to DC.)

OK, so for sure, the (DC) Log file is not updated until the deletes have actually finished,
so DC must be storing the list of deleted files in RAM up till that point.
(IOW, data is not streamed to the Log, so the Log file does not "grow" as files are deleted - only after the deletions have finished.
And I'm assuming ? that data is stored in RAM up until the time it is actually written to the Log file [& not stored in some temporary file on disk?].)

[therube]

Try right clicking the file in Windows Explorer, copying the size and then searching in Everything for a file with a matching size, eg:
size:1234567

Does not exist.


---

a slow USB Flash Drive (though the Flash drive was not monitored by Everything)

Absolute garbage.
UNRELIABLE. DO NOT PURCHASE.
Either... simply faulty (I've been through two of them), faulty engineering, or it is one of those "fake" drives.
https://www.newegg.com/team-model-tc175 ... 6820331073

[void]

I have been trying to reproduce the issue my end with no luck...

I think the issue has something to do with how Everything is caching file size for multiple USN events.

Having a large system delay might be causing Everything to keep this cache around for a long time for stale USN events.
I have been playing around with introducing large system delays to no effect.
The file size is always gathered correctly.

Everything does write a ton of information when verbose debug logging is enabled and gathering the file size fails.
Unfortunately, this means leaving verbose debug logging enabled for long periods of time to catch the issue..
Verbose debug logs can grow very large.

I'll keep investigating...



If anyone runs into the issue, could you please terminate Everything and rerun Everything in verbose debug mode:

• From the Task Manager (Ctrl + Shift + ESC)
• Terminate Everything.
• Relaunch Everything with:
  Everything.exe -verbose -debug-log
• Everything will reprocess the USN Journal and hopefully log the missing size reason.
• Please send your %TEMP%\Everything Debug Log.txt to support@voidtools.com

Privacy

[therube]

(though i've not run into this again... just some more thoughts... [& not looking for an answer, just throwing out thoughts])

---

as my USB flash drive "disintegrated" (for the 1000th time) [i've since stopped using it]
during that period of disintegration
Everything became non-responsive
- until Windows finally said, "good riddence, you're outta here!"
(at which time Everything came back)

-

when a drive has gone to sleep,
& with a slow to wake up,
during that time,
Everything can be in a "busy" state...

between something like that, & my "bad flash drive",
& writing logs &...
might all of that combined have contributed to above... ?

[void]

I little bit about how Everything works:

File size information is not stored in the USN Journal.

When a file is modified (data is written to a file):
A USN Journal event is created (USN_REASON_DATA_EXTEND)
This event is stored in your USN Journal.
No file size information is stored in the USN Journal.

Everything will see this USN event and start another thread to go get the size from disk.
Everything will get the size information for the file by File-ID (not filename).

The file could have been deleted before Everything reads the size.
No issue, we will delete the file from the index once we see the USN delete event.
We may see the file with an out of sync size for a split second before it is removed from the index.

The file could have been renamed or moved.
This doesn't matter because we find the file by File-ID.
File-ID doesn't change when a file is renamed or moved.

The file could have been modified, so we see the new size without processing the USN Journal event.
No issue, when we process the next USN event we will likely see the same size.

Size information shown in Everything can be out of sync.
Size information in Everything will always lag behind USN events.
As long as Everything keeps processing USN events we should keep the index up-to-date.



What is likely happening is the lookup for the file by File-ID is failing.
Lookup for a file by File-ID will often fail, eg: the file is deleted, maybe the update sequence is bad (read during a partial NTFS write)
None of these should matter as there should be pending USN journal events so Everything will keep trying to update your file size.
It's possible the File-ID look up failed and there was a missing USN journal event.



What Everything could do:
Combine with other methods to keep the file size information up to date. (ReadDirectoryChanges and ShellChangeNotify)
As you probably know, Everything currently only updates the file size when all handles are closed.