[Feature Request] XYplorer Tag Support

[dougbenham]

Tags for XYplorer are stored in

%APPDATA%\XYplorer\tag.dat

in the following format:

Code: Select all

XYplorer File Tags v5

Labels:
Red|000000|FC7268;Orange|000000|F6AB46;Yellow|000000|EFDC4A;Green|000000|B5D74A;Blue|000000|5DA4FE;Purple|000000|E29CDC;Grey|000000|B5B5B5

Extra Tags:
|2|0|0|||||
|0|0|0|||||
|0|0|0|||||
|0|0|0|||||
|0|0|0|||||
|0|0|0|||||
|0|0|0|||||
|0|0|0|||||
|0|0|0|||||
|0|0|0|||||
|0|0|0|||||
|0|0|0|||||
|0|0|0|||||
|0|0|0|||||
|0|0|0|||||
|0|0|0|||||

Storage: 0 (Absolute paths)

Data:
C:\MyFolder\File.txt|0|Tag1|||||||||||||||||
C:\MyFolder\File2.txt|0|Tag1, Tag2|||||||||||||||||

A temporary solution is to run this powershell to generate a compatible

property_user_values

csv file:

Code: Select all

$content = Get-Content "$env:APPDATA\XYplorer\tag.dat"
$data = $content.Where({ $_ -eq "Data:" }, "SkipUntil")
@("Filename,XY Tags") + ($data | Select-Object -Skip 1 | % {
    $split = $_ -split '\|'
    $filename = $split[0]
    $tags = $split[2] -split ', ' -join ';'
    return $filename + "," + $tags
}) | Out-File "$env:APPDATA\XYplorer\everything-properties.csv"

However, I'd love it if Everything could have native support for reading XYplorer tags (and perhaps other data like labels/colors). The feature could be turned on with a flag similar to

opus_meta_information

(maybe called

xyplorer_meta_information

).

[void]

I'll look into adding XYplorer tag support.

Thank you for the suggestion.

tag.dat documentation

[dougbenham]

Let me know if you want help, I'm an experienced C# and C++ developer

Or if you want a donation to prioritize this feature

[void]

Everything 1.5.0.1341a adds XYplorer tag support.

To load tags from your tag.dat file:

• Copy and paste the following into your Everything search box:
  /xyplorer_tag_dat=%APPDATA%\XYplorer\tag.dat
• Press ENTER in your Everything search box.
• If successful, xyplorer_tag_dat=%APPDATA%\XYplorer\tag.dat is shown in the status bar for a few seconds.
  where %APPDATA%\XYplorer\tag.dat is the location of your XYplorer tag.dat file.

Press F5 to refresh properties.
Select files and press Ctrl + F5 to refresh indexed properties.



Currently, only tag information is supported.
Please let me know if you would like to see support for other properties.

[dougbenham]

Wow!! Amazing work. I got it working. Is there anyway to make it not combine XYplorer tags with File Tags? I see a lot of my files have weird file tags set and I just want to ignore them and focus on tags I've set from XYplorer.

[void]

Everything will try to gather tags from the Windows Property System or built-in property handler if there's no XYplorer tag.

To show only properties from your XYplorer tag.dat, disable the Windows Property System and builtin property handlers:

• Copy and paste the following into your Everything search box:
  /property_builtin_handler=0
• Press ENTER in your Everything search box.
• If successful, property_builtin_handler=0 is shown in the status bar for a few seconds.
  ---
• Copy and paste the following into your Everything search box:
  /property_system=0
• Press ENTER in your Everything search box.
• If successful, property_system=0 is shown in the status bar for a few seconds.

Please note: this will disable all properties except XYplorer tags.
This might be acceptable for your usage?



property_builtin_handler
property_system

[dougbenham]

That is a decent solution but I probably won't use it because I do like other properties to be gathered and indexed. Anyways, thank you for your hard work.

[Marco Polo]

Everything 1.5.0.1341a adds XYplorer tag support.
Currently, only tag information is supported.
Please let me know if you would like to see support for other properties.

Support of other properties will be useful feature IMHO.


I had some experiments with XYplorer and found out some facts. Everything search treats "tag field" of XYplorer as singe string, not as a set of tags. And it also searches tag field for substring not for tag as a whole word.

E.g. tag field is "cat, mouse, bulldog, sheep". The search "bull" will succeed because of word "bulldog", but in some situations we want to see bulls without dogs. Though search with correct exclusion works fine

Code: Select all

tag:bull !tag:bulldog

You can search across tag boundaries and this search will succeed too (in double qoutes as it contains space)

Code: Select all

tag:"at, mou"

I think this behavior is similar to described in post about multiple tag search

In other words, multiple tag searching seems superficial in that it just defines a list of tags as one tag with semicolons in the tag itself. I know that's not the case but that's the feeling you get.

Is refresh F5 button or menu View / Refresh option deactivated after you update external tag.dat file so it has no impact to search result? I am sure changed tag.dat file should add one more file to search result but result grid does not change. Re-executing same search with ENTER in search field has no impact either. The only action to update search result according to new tag values is to change search expression and then undo changes. Is this a bug ? My ini file looks like this:

Code: Select all

property_system=0
property_system_eml=0
property_builtin_handler=0
opus_meta_information=0
xyplorer_tag_dat=c:\XYplorer\Data\tag.dat

[void]

Thank you for your feedback Marco Polo,

What XYplorer properties would you like to see?



Everything will treat the XYplorer tags as a single string.
I'll convert the ", " delimited lists to match Everything's ";" delimited lists.



Everything will use a partial search by default.
Please try ww:tag:bull to match whole words.

I'll look into adding a multi-item: search modifier to make it easier to match whole items in semicolon (;) delimited lists.



F5 will refresh the currently shown tags, but not perform a research.
You'll need to press F5, cut your entire search so the search box is cleared, then paste your search back.
I'll look into making Everything perform a research when pressing F5.

[void]

Everything 1.5.0.1344a adds a multistring: search modifier.

Example usage:
multistring:whole:tag:bull

(treats the tag as a semicolon (;) delimited list of strings and match a whole item named: bull)



Everything 1.5.0.1344a also improves handling of XYplorer tags and ", " delimited lists.



F5 will now also perform a requery.

[Marco Polo]

Everything 1.5.0.1344a also improves handling of XYplorer tags and ", " delimited lists.

Many thansk for this improvement.

I spent some time playing with XYplorer tag.dat file. The link above to XYplorer forum contains outdated information for file version 3. It has less number of extra fields (now 16 for version 19.70+) and less number of extra field types, though it is correct in general. Actual format saves file color value, then list of tags, 16 extra fields and comment.

Some more information about tag.dat file format from XYplorer manual. It can be found in CHM help with "tag.dat" search, but I feel it tricky to navigate here-and-there inside CHM with poor bokmarking features, so I desided just to read official PDF from site https://www.xyplorer.com/download/XYplorerHelp.pdf

Page 345 says:

Code: Select all

The Database: tag.dat
====================
All tag data are retained across sessions and stored in one file, tag.dat, located in the application data
folder. The file is kept in an open, simple, and human-friendly text format (UTF-16 Text File) so that you
can easily edit it manually using an editor or programmatically using any scriptable software.
File Format of the XYplorer Tag Database (version 5 -- from 19.70 onwards)
Line 1: Header and version info: XYplorer File Tags v5
Line 2: (empty)
Line 3: Labels:
XYplorer Help
346
Line 4: Label schemes, separated by ";". Each scheme: Name|TextColor|BackColor. Colors are in
RRGGBB.
Line 5: (empty)
Line 6: Extra Tags:
Line 7-22: Extra Tag Definitions: Caption|TypeID|0|0||||| . Some fields are reserved for future use.
Line 23: (empty)
Line 24: Storage: ...
Line 25: (empty)
Line 26: Data:
Line 27: Tagged item #1: Full filename|LabelID|Tags|Extra1|Extra2|Extra3|Extra4|Extra5|Extra6|
Extra7|Extra8|Extra9|Extra10|Extra11|Extra12|Extra13|Extra14|Extra15|Extra16|Comment
Line 28: Tagged item #2: ... etc. ...

Documentation says

Full filename

but page 66 says it can be

relative

.

Example for field of "list" type - TypeID=5 (popup list of predefined values, other types just have no predefined values):

Code: Select all

First Field Caption|5|0|0|many;predefined;field;values|||| - it is for extra1 field
Second Field Caption|4|0|0||||| - it is for extra2 field
Third Field Caption|0|0|0||||| - it is for extra3 field

Example of stored file metadata:

Code: Select all

file name| <color number> | [file,tag,list] |ex1|ex2|ex3|...||||||||||...|ex15|ex16|comment
file name|7|file,tag,list|many|2|field 3 - any text|...||||||||||...|ex15|ex16|comment

Color field is always set, it has 0 if it was not changed by user. Here file color is 7, field ex1=many, field ex2=2 (displays as 2 stars), field ex3 value is "field 3 - any text"

TypeID (it is only important for field data conversion inside XYplorer, IMHO)

Code: Select all

0 - text
1 - number
2 - date
3 - checkbox
4 - rating stars (1-5)
5 - list
6 - location (path)
7 - icon (???)
8 - image (path)
9 - script (field data is shown as text)

All field types store text, not binary data. Every char is acceptable except vertical line. New line char is replaced with "printable" (¶) character, so each row in tag.dat describes a file.

Comment is the last field and accepts vertical line too. Other fields do not store vertical line as said on page 347.

Extra field "type" has no impact to data, you can save any value regardless field type. E.g. save some string value to number or date filed is ok.

Checkbox is a strange type. It is "unset" only when value is empty string, any other string is "set" state. So values "false", "0" or just line feed / space still correspond to "set" state. There is no chanse to guess whether checkbox is unset or value was not assigned at all. But anyway checkbox field is able to contain any text value.

[Marco Polo]

Examples of Storage parameter impact

Code: Select all

Storage: 0 (Absolute paths)

Code: Select all

C:\mobile\android\F-Droid.apk|0|android,program||5|some text ...||||||||||||||
C:\mobile\sound\wakeup.ogg|0|sound,loud||4|||||||||||||||it is loud, VERY loud
R:\stagsi\Stagsi.sqlite|5|library,media,portable||3|||||||||||||||

Storage parameter (page 66)

Absolute paths: Not portable. [= Storage: 0]

It is default value for program. Saves full paths, no surprise. Human-readable description after value is written by XYplorer.

Code: Select all

Storage: 1 (Relative to application drive)

Code: Select all

?:\mobile\android\F-Droid.apk|0|...
?:\mobile\sound\wakeup.ogg|0|...
R:\stagsi\Stagsi.sqlite|5|...

Using the "?" syntax, useful for portable installations on sticks. [= Storage: 1]

Drive letter is changed to "?", files on other drives use absolute path.

Code: Select all

Storage: 2 (Relative to application)

Code: Select all

..\..\..\mobile\android\F-Droid.apk|0|...
..\..\..\mobile\sound\wakeup.ogg|0|...
R:\stagsi\Stagsi.sqlite|5|...

Useful for backups where XYplorer installations have the same position
relative to the tagged files. [= Storage: 2]

Use relative paths for files on the same drive as tag.dat (xyplorer.exe) and absolute paths for files on other drives.

Option 2 is probably good for usage with xyplorer itself, but tricky to guess real program path when file tag.dat is loaded OUTSIDE. Option 1 with relative to application drive is more error safe then option 2, IMHO. Probably option 2 is just to backup / restore tags data file.

Code: Select all

Storage: 3 (Relative to tags database)

Option 3 looks much better then option 2 for removable drives case.

Useful for in-place tags database files that are copied along with the
tagged files (a scripting command is planned for loading those databases). [= Storage: 3]

Code: Select all

Storage: 4 (Use volume serials)

Code: Select all

B2D60DFE:\mobile\android\F-Droid.apk|0|...
B2D60DFE:\mobile\sound\wakeup.ogg|0|...
78EE4C19:\stagsi\Stagsi.sqlite|5|...

The 8-digit hex volume serial is stored in place of the drive letter. Allows to
identify removable drives regardless of the drive letter assigned by the host system. [= Storage: 4]

Files on all drives use absolute paths with volume serials. Volume serials have the same format they are displayed by dir command:

Code: Select all

C:\> dir
 Volume in drive C has no label.
 Volume Serial Number is B2D6-0DFE

[void]

Thank you for the information and XYplorer PDF help Marco Polo,

Everything 1.5.0.1345a adds support for XYplorer comments and ratings.



Everything will use the first extra rating field in your tags.dat as the rating.



The Storage parameter is a little difficult to support as Everything doesn't know where the portable XYplorer application is located.
Everything will currently always treat the paths as relative to the tags.dat file location.

I'll will consider an option to specify the XYplorer application location and supporting the storage field in a future release.
For now, I recommend using full paths.

[Marco Polo]

Thank you Void.
Comment and star reading works fine.

The Storage parameter is a little difficult to support as Everything doesn't know where the portable XYplorer application is located.
Everything will currently always treat the paths as relative to the tags.dat file location.

I see no useful scenarios with that storage=2 option. Relative + absolute paths is good enough for me.

Is it possible to have a sort of «function» like read(M,N) to read XY extra field M to Everything custom property N without any conversion ? Those who prefer rating stars could use readstar(M,N). Setting configuration via ini file will be enough.

Motivation. Extra fields of XY are very flexible but it is difficult to guess what each user stores in those XY fields. Displaying fields as plain text without conversion and not fixing strict M/N numbers gives chance to load fields one needs while skipping others. I prefer "color" column that gives ability to assign file categories with single click (that column is is not a real extra field, because it is placed before tags column). But tastes differ and others may prefer column with dates or checkboxes.

There is lot of freedom for further usage. There are 16 extra fields today but there probably will be more some day. As i noticed in manual they started from 5 fields. So read(M,N) function will remain usable if some day number of extras will increase to 32.

[void]

I have put on my TODO list to add support for mapping the extra fields to an Everything property.

Thank you for the suggestion.

[dougbenham]

Everything 1.5.0.1341a adds XYplorer tag support.

To load tags from your tag.dat file:

• Copy and paste the following into your Everything search box:
  /xyplorer_tag_dat=%APPDATA%\XYplorer\tag.dat
• Press ENTER in your Everything search box.
• If successful, xyplorer_tag_dat=%APPDATA%\XYplorer\tag.dat is shown in the status bar for a few seconds.
  where %APPDATA%\XYplorer\tag.dat is the location of your XYplorer tag.dat file.

Press F5 to refresh properties.
Select files and press Ctrl + F5 to refresh indexed properties.



Currently, only tag information is supported.
Please let me know if you would like to see support for other properties.

Lets say I have 30k files. I have the Tags indexed in Everything.

Now in XYplorer I modify some tags on some files.

In Everything, since I have the Tags indexed, the only way to get those new tags loaded is by CTRL+F5 and fully re-indexing all those files. Sometimes re-indexing files can be a really slow process. 1k files might take a minute.

Am I missing something? Would there be any way that we can get a quick ingestion of all the data in our tag.dat and update appropriate indexes? (and skip full re-indexing other metadata that is slow to read)

[void]

Would an is-xyplorer-tag property be useful?

With this, you could perform the following search:

is-xyplorer-tag:

This would list all files that are referenced in your tag.dat
(updated at the time of search)
From there you can select all files and hit Ctrl + F5.

[dougbenham]

Sure that would be appreciated! Thank you.

[dougbenham]

Still I wish there was a way to only update the Tags property. I also have

Code: Select all

Video Format

as an indexed property and I think that slows down my indexing process quite a bit. Would love to be able to avoid heavy re-processing files' metadata and just keep the XYplorer tags synced up with Everything nicely.

[void]

Everything 1.5.0.1357a adds an is-xyplorer-tag: search function.

To search for files that are specified in your xyplorer tag.dat, include the following in your search:

is-xyplorer-tag:



Select all results and hit Ctrl + F5 to reindex properties for all xyplorer tags.

Unfortunately, Everything will need to reindex all properties for the selected files.
I have on my TODO list to add a property manager to reindex only specific properties.



is-xyplorer-tag: will rescan your xyplorer tag.dat at the time of search.

[dougbenham]

Thank you, greatly appreciate your time spent on this! The is-xyplorer-tag is massively helpful.

[void]

Just making a note here:

Everything 1366a and earlier is incorrectly treating byte 0xb6 as a new line (instead of ¶)

This is causing strange text to appear when this byte is used in UTF-8 sequences. (부)

This bug will be fixed in the next alpha update (1367a or later)
I'll make another post here when the update is available.