Getting a file list remotely on a Linux NAS

General discussion related to "Everything".
Post Reply
Smith
Posts: 2
Joined: Sat Jan 29, 2022 6:52 pm

Getting a file list remotely on a Linux NAS

Post by Smith »

I am doing this myself and figured i would share the method. It is indeed much faster than getting a file list over the network, if you do list generation as some sort of batch job or whatnot.

You need to somehow be able to run programs on your NAS. In the case of Netgear NAS devices you can just ssh in as root. The usual strong caveats about logging in as root apply (if careless it's very easy to disable the device, delete all your files etc in mere seconds)

The program in this case is perl which usually is installed on linux systems but any language would work. The program (make_efu.pl):

Code: Select all

#!/usr/bin/perl

$WINDOWS_TICK = "10000000";
$SEC_TO_UNIX_EPOCH = 11644473600;

$d = $ARGV[0];
$regexp1 = $ARGV[1];
$regexp2 = $ARGV[2];
if ($regexp1 ne "") {
        $regexp1 =~ s/\//\\\\/g; # windows file sep
        $regexp1 = qr/$regexp1/;
}

use File::Find;
print "Filename,Size,Date Modified,Date Created,Attributes\r\n"; # windows line endings
find(\&make_row, $d);

sub make_row {
        my $f = $File::Find::name;
        $f =~ s/\//\\/g; # windows file sep
        $size = (stat())[7];
        $mtime = $WINDOWS_TICK * ((stat(_))[9]  + $SEC_TO_UNIX_EPOCH);
        $ctime = $WINDOWS_TICK * ((stat(_))[10] + $SEC_TO_UNIX_EPOCH);
        if (-d _) {$attr = 0x10; } else {$attr = 0x80;}  # don't bother with unix writable/ win not readonly, just whether dir or file
        if (regexp ne "") {
                $f =~ s/$regexp1/$regexp2/;
        }
        print "\"$f\",$size,$mtime,$ctime,$attr\r\n";
}
How to use it:

Code: Select all

make_efu.pl <local directory> [<regex1> <regex2>] 

1 The program outputs the EFU formatted list to stdout so you have to capture that if you want it in a file.
2 It outputs msdos line endings and directory separation backslashes.
3 The conversion between linux and windows file times is handled.
4 The regex stuff is in case you want to translate each unix file name to whatever windows network context you have.

example (a netgear box):

root@nas2:/data/share#  screen -S screen_session_for_list_making
root@nas2:/data/share# ./make_efu.pl /data ^/data "\\\\NAS2" > NAS2.efu
^D
[screen is terminating]
root@nas2:/data/share# ls -lh NAS2.efu 
-rw-rw-rw-+ 1 root root 982M Jan 29 20:10 NAS2.efu
root@nas2:/data/share# ls -lh NAS2.efu 
-rw-rw-rw-+ 1 root root 983M Jan 29 20:10 NAS2.efu

etc etc
As per usual, sharing is caring but I am not responsible for any bugs, you run this completely at your own risk, only run it on trusted input parameters (eg whatever you do don't install this as a CGI script), the code is public domain, corrections and improvements are welcome.

Here is some ancillary info;

Code: Select all


What is the format of an EFU file list?
EFU files are comma-separated values (CSV) files.
A header is required with at least the Filename column specified.
File size is specified in bytes.
Dates are FILETIMEs (100-nanosecond intervals since January 1, 1601.)
Attributes can be zero or more of the Windows File Attributes.

* * *

FILETIME structure (minwinbase.h):
Contains a 64-bit value representing the number of 100-nanosecond intervals since January 1, 1601 (UTC).

Between 01-01-1601 and 01-01-1970, there are exactly 134774 days.
134774 * 24 (hours per day) * 3600 (seconds per hour) = 11644473600 s.

* * *

Windows file attributes

FILE_ATTRIBUTE_DIRECTORY
16 (0x10)
The handle that identifies a directory.
FILE_ATTRIBUTE_HIDDEN
2 (0x2)
The file or directory is hidden. It is not included in an ordinary directory listing.
FILE_ATTRIBUTE_NORMAL
128 (0x80)
A file that does not have other attributes set. This attribute is valid only when used alone.
FILE_ATTRIBUTE_READONLY
1 (0x1)
A file that is read-only.  Applications can read the file, but cannot
write to it or delete it.  This attribute is not honored on directories. 
For more information, see You cannot view or change the Read-only or the
System attributes of folders in Windows Server 2003, in Windows XP, in
Windows Vista or in Windows 7. 
FILE_ATTRIBUTE_SYSTEM
4 (0x4)
A file or directory that the operating system uses a part of, or uses exclusively.


* * *

Perl unix stat()

 0 dev      device number of filesystem
 1 ino      inode number
 2 mode     file mode  (type and permissions)
 3 nlink    number of (hard) links to the file
 4 uid      numeric user ID of file's owner
 5 gid      numeric group ID of file's owner
 6 rdev     the device identifier (special files only)
 7 size     total size of file, in bytes
 8 atime    last access time in seconds since the epoch
 9 mtime    last modify time in seconds since the epoch
10 ctime    inode change time in seconds since the epoch (*)
11 blksize  preferred I/O size in bytes for interacting with the
            file (may vary from file to file)
12 blocks   actual number of system-specific blocks allocated
            on disk (often, but not always, 512 bytes each)
Last edited by Smith on Sat Jan 29, 2022 9:19 pm, edited 1 time in total.
NotNull
Posts: 5461
Joined: Wed May 24, 2017 9:22 pm

Re: Getting a file list remotely on a Linux NAS

Post by NotNull »

Oh, wow! Thanks for sharing, @Smith!
Smith
Posts: 2
Joined: Sat Jan 29, 2022 6:52 pm

Re: Getting a file list remotely on a Linux NAS

Post by Smith »

NotNull wrote: Sat Jan 29, 2022 8:38 pm Oh, wow! Thanks for sharing, @Smith!
I of course immediately found a problem after posting it. File names in the list should be enclosed in quotes.

I edited to reflect this, the post hopefully shows up again later!
Post Reply