NEWS for the Linux download scripts, version 1.19.4-ESR

NEWS for the Linux download scripts, version 1.19.4-ESR

Postby hbuhrmester » 27.01.2021, 21:26

NEWS for the Linux download scripts, version 1.19.4-ESR

Release date: 2021-01-26
Development branch: esr-11.9
Supported version: WSUS Offline Update, Community Edition 11.9.7

The Linux download scripts, version 1.19.4-ESR try to keep up with WSUS Offline Update, Community Edition 11.9.7. These scripts are not compatible with the upcoming Community Edition 11.9.8, because of major changes in the calculation of dynamic updates.


Bugfixes

The Security-only Safety Guard did not work with GitLab

If security-only downloads are selected, then the full update rollups must be excluded from both download and installation. This requires a manual configuration, because incremental security-only updates and cumulative update rollups cannot be distinguished automatically.

The function seconly_safety_guard tries to make sure, that several configuration files in the directories client/static and client/exclude have been updated after the last official patch day; otherwise the download will be postponed.

Previously, the file modification date of these files would be compared to the current date. This worked well with Trac/svn. It does not work anymore with GitLab, because GitLab does not set the Last-Modified header for files, which are extracted from the version control system. Therefore, newly downloaded files will always have the file modification date set to the current date and time.

The Security-only Safety Guard now searches literally for the expected month name in the configuration files.


New Features

Optional "fast mode" for the calculation of the integrity database

WSUS Offline Update uses three hash function to create the integrity database: MD5, SHA-1 and SHA-256. This always looked over-engineered, and it is unnecessarily slow on old machines. An optional "fast mode" was introduced, which only calculates the SHA-1 hash. This option can be enabled in the file preferences.bash.

The resulting hashdeep files will be compatible with previous versions of the Linux scripts and with the Windows scripts, because hashdeep recognizes the file format automatically in its auditing mode. You don't need to delete any existing files in the directory ../client/md, because the hashdeep files will be deleted and recreated automatically on each download run.

Note: The Windows script DownloadUpdates.cmd can use the same fast mode with a small patch. This is only needed, if you like to compare the results on Windows and Linux with the script compare-integrity-database.bash:

  1. Change the hashdeep option "-c md5,sha1,sha256" to "-c sha1" everywhere.

  2. Find the line:

    Code: Select all
    for /F "usebackq tokens=3,5 delims=," %%i in ("%TEMP%\sha1-%1-%2.txt") do (


    and change it to:

    Code: Select all
    for /F "usebackq tokens=2,3 delims=," %%i in ("%TEMP%\sha1-%1-%2.txt") do (


    The hashdeep files are used for a quick integrity check: The expected SHA-1 hash is embedded into the filename of most security updates. It is compared to the calculated SHA-1 hash in the hashdeep file. If they don't match, then the downloaded file will be deleted.

    For this comparison, the script extracts fields 3 and 5 of the comma-separated hashdeep files in its default mode. It extracts fields 2 and 3 in the fast mode.

    Default field order of the hashdeep files:

    Code: Select all
    Field 1 = File size
    Field 2 = MD5 hash
    Field 3 = SHA-1 hash as calculated by hashdeep
    Field 4 = SHA-256 hash
    Field 5 = Relative pathname with embedded SHA-1 hash


    Field order in the fast mode:

    Code: Select all
    Field 1 = File size
    Field 2 = SHA-1 hash as calculated by hashdeep
    Field 3 = Relative pathname with embedded SHA-1 hash


    Warning: The patch for the Windows script DownloadUpdates.cmd was tested and should work so far, but if you mess this up, all existing downloads will be deleted due to "mismatching SHA-1 hashes".


Changed Features

Default languages are removed on the fly

By default, WSUS Offline Update downloads updates for the languages English and German.

In the Windows version, the default languages can be removed with the scripts RemoveGermanLanguageSupport.cmd and RemoveEnglishLanguageSupport.cmd. More languages can be added with the script AddCustomLanguageSupport.cmd.

The new Linux download scripts always tried to do that in a more consistent way:

  • First, the default languages are removed from several global files in the directory wsusoffline/static.

  • All languages, which are selected on the command-line, are then added back from the localized files in the same directory.

In previous versions, the removal of default languages would modify the global input files on disk. Now the default languages are removed on the fly, without changing the input files at all.

This functionality is implemented as a new function filter_default_languages in the file ./libraries/dos-files.bash. The script ./download-updates-tasks/30-remove-default-languages.bash is obsolete and will be removed.


hashdeep messages are duplicated with tee

The functions create_integrity_database and verify_integrity_database now use "tee" to duplicate messages from hashdeep and to write them to the logfile and the screen. This is possible, because the logfile is referenced with absolute paths since version 1.11 of the Linux download scripts.

Previously, a workaround was needed, because relative paths to the logfile would become invalid, if the working directory was changed with pushd/popd.


Network timeouts are increased

The values for timeout and waitretry are increased for both wget and aria2.


Sysinternals utilities are downloaded to the directory ../bin

The archives AutoLogon.zip, Sigcheck.zip and Streams.zip are downloaded to the directory ../bin, as in the Windows script DownloadUpdates.cmd.


Three more obsolete configuration files are removed

The following configuration files are no longer needed in the esr-11.9 version and will be removed:

Code: Select all
../static/StaticDownloadLink-mkisofs.txt
../static/StaticDownloadLinks-mkisofs.txt
../client/static/StaticUpdateIds-ie10-w61.txt



Renamed the functions filter_cr and todos_line_endings

The functions filter_cr and todos_line_endings are both meant to change line endings from and to DOS. They are now named more appropriately:

Code: Select all
filter_cr           ->  dos_to_unix
todos_line_endings  ->  unix_to_dos



Removed features

Removed support for Office 2010

This removes the following files:

Code: Select all
../static/StaticDownloadLinks-o2k10-*.txt
../client/static/StaticUpdateIds-o2k10.txt



Removed support for Windows 10 version 1709

The following files will be removed:

Code: Select all
../exclude/ExcludeList-w100-1709.txt
../client/static/StaticUpdateIds-w100-16299.txt
../client/static/StaticUpdateIds-w100-16299-x64.txt
../client/static/StaticUpdateIds-w100-16299-x86.txt
../client/static/StaticUpdateIds-wupre-w100-16299.txt
../client/static/StaticUpdateIds-servicing-w100-16299.txt
../client/static/StaticUpdateIds-w100-16299-dotnet.txt
../client/static/StaticUpdateIds-w100-16299-dotnet4-528049.txt




Downloads

The download links for this version are:


Link for the online check at VirusTotal. Note, that the URL includes the SHA-256 hash of the archive:


The hashes for the archive linux-scripts-1.19.4.tgz are:
Code: Select all
MD5      9eb76a18e3a5561add6d1b5a92d16835
SHA-1    618e6697a4cb40cdeb31f537f45eaa0992bf977a
SHA-256  54b1323ea4300db6d860dc1e3726e715b92b22ea56926be3df00868260438e6c




Further Reading

The first three posts in the original topic A complete rewrite of the Linux scripts are still being updated, to point to the latest version:

hbuhrmester
 
Posts: 506
Joined: 11.10.2013, 20:59

More about hashdeep options

Postby hbuhrmester » 27.01.2021, 23:59

More about hashdeep options

hashdeep files are csv-formatted files with an additional file header. The basic layout of these files can be defined when creating the files:

  • By default, hashdeep uses two hash functions, MD5 and SHA-256
  • In WSUS Offline Update, hashdeep uses three hash functions, MD5, SHA-1 and SHA-256, with the option "-c md5,sha1,sha256"
  • The suggested fast mode only uses SHA-1 hashes with the option "-c sha1"


The used format will be saved to the header of each hashdeep file. This header can be:

  • Default hashdeep format without specifying the option "-c"

    Code: Select all
    %%%% HASHDEEP-1.0
    %%%% size,md5,sha256,filename

  • Hashdeep format in WSUS Offline Update with the option "-c md5,sha1,sha256"

    Code: Select all
    %%%% HASHDEEP-1.0
    %%%% size,md5,sha1,sha256,filename

  • Hashdeep format with the fast mode option "-c sha1"

    Code: Select all
    %%%% HASHDEEP-1.0
    %%%% size,sha1,filename

This information will be used for the verification of the hashdeep files (or the downloads listed in these files). The option -c is only needed for the creation of the hashdeep files, and only to create different hashes than the default MD5 and SHA-256. There is nothing to add or change for the verification step.


Only the last column, the filename, is a bit ambiguous:

  • By default, hashdeep reports the full path.
  • With the option -l, hashdeep uses relative paths to the current working directory.
  • With the option -b, hashdeep only reports the filename.

So, if the options -l or -b are used, they must be used consistently for both creation and verification of the hashdeep files.



The mentioned Linux script compare-integrity-database.bash compares the directories client/md on both Windows and Linux:

  • It strips the first five lines of each file, because the working directory and the exact command-line will be different on Windows and Linux.
  • It sorts all files consistently by the byte order, because hashdeep doesn't use a deterministic sort order.
  • Then it calls diff to compare the results.

This is enough for a deep comparison of all downloads on Windows and Linux. The hard work of listing the files and calculating the file hashes has already been done by hashdeep. But of course, then the basic layout of the hashdeep files must be the same on both sides.

This is what I meant with:
This [the Windows patch] is only needed, if you like to compare the results on Windows and Linux with the script compare-integrity-database.bash



There are several options, which are neither explained in the manual nor on the help page.

The problem with the help page is, that it is limited to exactly 22 lines. This is the idea of a "page" for command-line applications: The help page fits into a typical terminal window of 24x80 characters, and there are two lines left to enter the next command.

Code: Select all
$ hashdeep -h
hashdeep version 4.4 by Jesse Kornblum and Simson Garfinkel.
$ hashdeep [OPTION]... [FILES]...
-c <alg1,[alg2]> - Compute hashes only. Defaults are MD5 and SHA-256
                   legal values: md5,sha1,sha256,tiger,whirlpool,
-p <size> - piecewise mode. Files are broken into blocks for hashing
-r        - recursive mode. All subdirectories are traversed
-d        - output in DFXML (Digital Forensics XML)
-k <file> - add a file of known hashes
-a        - audit mode. Validates FILES against known hashes. Requires -k
-m        - matching mode. Requires -k
-x        - negative matching mode. Requires -k
-w        - in -m mode, displays which known file was matched
-M and -X act like -m and -x, but display hashes of matching files
-e        - compute estimated time remaining for each file
-s        - silent mode. Suppress all error messages
-b        - prints only the bare name of files; all path information is omitted
-l        - print relative paths for filenames
-i/-I     - only process files smaller than the given threshold
-o        - only process certain types of files. See README/manpage
-v        - verbose mode. Use again to be more verbose
-d        - output in DFXML; -W FILE - write to FILE.
-j <num>  - use num threads (default 1)



But these are not necessarily all options. Repeating the option -h two or three times will increasingly display more info (much like the option -v):

Code: Select all
$ hashdeep -h -h -h
hashdeep version 4.4 by Jesse Kornblum and Simson Garfinkel.
$ hashdeep [OPTION]... [FILES]...
-c <alg1,[alg2]> - Compute hashes only. Defaults are MD5 and SHA-256
                   legal values: md5,sha1,sha256,tiger,whirlpool,
-p <size> - piecewise mode. Files are broken into blocks for hashing
-r        - recursive mode. All subdirectories are traversed
-d        - output in DFXML (Digital Forensics XML)
-k <file> - add a file of known hashes
-a        - audit mode. Validates FILES against known hashes. Requires -k
-m        - matching mode. Requires -k
-x        - negative matching mode. Requires -k
-w        - in -m mode, displays which known file was matched
-M and -X act like -m and -x, but display hashes of matching files
-e        - compute estimated time remaining for each file
-s        - silent mode. Suppress all error messages
-b        - prints only the bare name of files; all path information is omitted
-l        - print relative paths for filenames
-i/-I     - only process files smaller than the given threshold
-o        - only process certain types of files. See README/manpage
-v        - verbose mode. Use again to be more verbose
-d        - output in DFXML; -W FILE - write to FILE.
-j <num>  - use num threads (default 1)
-f <file> - Use file as a list of files to process.
-V        - display version number and exit
-0        - use a NUL (\0) for newline.
-u        - escape Unicode
-E        - Use case insensitive matching for filenames in audit mode
-B        - verbose mode; repeat for more verbosity
-C        - OS X only --- use Common Crypto hash functions
-Fb       - I/O mode buffered; -Fu unbuffered; -Fm memory-mapped
-o[bcpflsde] - Expert mode. only process certain types of files:
               b=block dev; c=character dev; p=named pipe
               f=regular file; l=symlink; s=socket; d=door e=Windows PE
-D <num>  - set debug level
sizeof(off_t)= 8
HAVE_PTHREAD
HAVE_PTHREAD_H



The option "-W FILE - write to FILE" is often overlooked. Specifying an output file in hashdeep is better than using an output redirection. An output redirection will always create an empty file, even if hashdeep doesn't create any output nor set an error code. For some time, this was one of the most common errors in WSUS Offline Update, leading to silly workarounds:

hbuhrmester
 
Posts: 506
Joined: 11.10.2013, 20:59

Re: NEWS for the Linux download scripts, version 1.19.4-ESR

Postby aker » 04.02.2021, 00:12

The "-W"-option does not work here. Or I didn't understand, how to use it.
If I run (from .\wsusoffline\cmd) "..\client\bin\hashdeep64.exe -c md5,sha1,sha256 -b -r ..\client\w63\glb -W ..\client\md\test.txt" I get a list of hashes printed to the console output conatining the lines
Code: Select all
D:\wsusoffline\tmp\test\cmd\-W: No such file or directory
D:\wsusoffline\tmp\test\client\md\test.txt: No such file or directory


Also I cannot see it documented anywhere (not in your post nor in running "hashdeep -h -h -h" on Windows).
Wer Rechtschreibfehler findet, darf sie behalten oder an den Meistbietenden versteigern. / Everybody finding a misspelling is allowed to keep or sell it.
aker


WSUS Offline Update „Community Edition“ 12.5.1 / 11.9.9esr
https://gitlab.com/wsusoffline
aker
 
Posts: 3881
Joined: 02.03.2011, 15:32

Re: NEWS for the Linux download scripts, version 1.19.4-ESR

Postby hbuhrmester » 04.02.2021, 02:09

That's why it's often overlooked, but it's there:

Code: Select all
-d        - output in DFXML; -W FILE - write to FILE.



The order of options and parameters may be important: Options like -c -b -W should come before parameters (files and directories to be hashed). The point is, that hashdeep accepts a list of files and directories, and anything after the -r may be treated as a file or directory name.

That's why the help page briefly says:

Code: Select all
$ hashdeep [OPTION]... [FILES]...



The manual is more elaborate. If in doubt, options should be tried in the order listed here:

Code: Select all
SYNOPSIS
    hashdeep -V | -h
    hashdeep  [-c  <alg1>[,<alg2>]] [-k <file>] [-i <size>] [-f <file>] [-o
    <fbcplsde>] [-amxwMXreEspblvv] [-F<bum>] [-j <num>] [FILES]



This command seems to work:

Code: Select all
hashdeep -c md5,sha1,sha256 -b -W ../client/md/test.txt -r ../client/dotnet


and creates the file:

Code: Select all
%%%% HASHDEEP-1.0
%%%% size,md5,sha1,sha256,filename
## Invoked from: /home/hb/Projekte/wsusoffline-esr-11.9.8-b27/bin
## $ hashdeep -c md5,sha1,sha256 -b -W ../client/md/test.txt -r ../client/dotnet
##
117380440,aebcb9fcafa2becf8bb30458a7e1f0a2,8dd233698d5eb4609b86c2ac917279fe39e0ef4c,9b1f71cd1b86bb6ee6303f7be6fbbe71807a51bb913844c85fc235d5978f3a0f,ndp48-x86-x64-allos-enu.exe
242743296,d481cda2625d9dd2731a00f482484d86,3dce66bae0dd71284ac7a971baed07030a186918,0582515bde321e072f8673e829e175ed2e7a53e803127c50253af76528e66bc1,dotnetfx35.exe
5883536,a525f258d1ae90e016e61c74f480ec99,4f257180e4f0c4b34872bc5e91d9375865008f01,d90b9bf05612a813f3c4bbe5e311cfebd4cb7ab2b5d0f022fffd7e843893a217,ndp48-x86-x64-allos-deu.exe
34072608,14b0077e3932471223ee0dbdaf5fdd13,91a16a2e3d7e0aae83e948a5890d94eeece49421,3724993f7877d50872428f3de6384606b0dba24bfc87388579814301f5a7caa6,dotnetfx35langpack_x86de.exe
46092832,347e03d70126d10efa94eae00704b1c4,008b43a25d4faa7a21cdebb43db74d08216e6344,c21abd24cc2e7d3507857a6feeebce98146203b35a2644d6920fee784328f568,dotnetfx35langpack_x64de.exe
65444688,88bc05e20114a4506f40c36911de92fa,3049a85843eaf65e89e2336d5fe6e85e416797be,b21d33135e67e3486b154b11f7961d8e1cfd7a603267fb60febb4a6feab5cf87,NDP46-KB3045557-x86-x64-AllOS-ENU.exe
6696464,6b016b3b9198cb5889857835acad0bc8,b3adada709612792688e92f4defcae31aca4d0fa,499dc560513b88147fb5c5fb33121e84a127f58bca6f6b64a20aff40835f2900,NDP46-KB3045557-x86-x64-AllOS-DEU.exe



On the other hand, if the input directory is empty, then hashdeep still creates an empty output file. I thought, this could be prevented with the option -W.
hbuhrmester
 
Posts: 506
Joined: 11.10.2013, 20:59

Re: NEWS for the Linux download scripts, version 1.19.4-ESR

Postby aker » 15.02.2021, 14:02

@hbuhrmester

Vorschlag: Wir ändern die Reihenfolge der Hashes der „kompletten“ Hash-Erstellung von „md5,sha1,sha256“ auf „sha1,md5,sha256“. Dann ist keine Modifikation der DownloadUldates.cmd mehr nötig, bis ich etwas geeignetes für den Fast Mode zusammengebastelt habe.

Viele Grüße
Wer Rechtschreibfehler findet, darf sie behalten oder an den Meistbietenden versteigern. / Everybody finding a misspelling is allowed to keep or sell it.
aker


WSUS Offline Update „Community Edition“ 12.5.1 / 11.9.9esr
https://gitlab.com/wsusoffline
aker
 
Posts: 3881
Joined: 02.03.2011, 15:32

Re: NEWS for the Linux download scripts, version 1.19.4-ESR

Postby hbuhrmester » 15.02.2021, 16:36

Das ändert nichts, weil die Download-Skripte für den Integritätstest immer zwei Spalten aus den Hashdeep-Dateien extrahieren müssen: die Spalte mit den berechneten SHA-1-Hashen und die letzte Spalte mit den Dateinamen. Die Dateinamen enthalten die erwarteten SHA-1-Hashe, und diese müssen mit den berechneten Werten übereinstimmen.

Im Standardmodus müssen deshalb die Spalten 3 und 5 extrahiert werden, im fast mode die Spalten 2 und 3.

Felder im Standardmodus
  1. Dateigröße
  2. berechneter MD5-Hash
  3. berechneter SHA-1-Hash
  4. berechneter SHA-256-Hash
  5. Dateiname mit eingebettetem SHA-1-Hash als Referenzwert

Felder im Fast Mode
  1. Dateigröße
  2. berechneter SHA-1-Hash
  3. Dateiname mit eingebettetem SHA-1-Hash als Referenzwert


Ein Patch für DownloadUpdates.cmd würde also zwei Änderungen enthalten:

  1. Die Option -c md5,sha1,sha256 wird geändert in -c sha1
  2. Die Zeile:

    Code: Select all
    for /F "usebackq tokens=3,5 delims=," %%i in ("%TEMP%\sha1-%1-%2.txt") do (


    wird geändert in:

    Code: Select all
    rem *** default mode ***
    rem token 3 -> %%i  berechneter SHA-1 Hash
    rem token 5 -> %%j  Dateiname mit eingebettetem SHA-1-Hash
    rem *** fast mode ***
    rem token 2 -> %%i  berechneter SHA-1 Hash
    rem token 3 -> %%j  Dateiname mit eingebettetem SHA-1-Hash

    for /F "usebackq tokens=2,3 delims=," %%i in ("%TEMP%\sha1-%1-%2.txt") do (


Wenn Du jetzt zwei Spalten vertauscht, ändert das ja nichts daran, dass sich der Index der letzten Spalte auf jeden Fall ändert von 5 auf 3.


Aber der Fast Mode ist optional, und Du musst im Skript DownloadUpdates.cmd gar nichts ändern. hashdeep selber verwendet die Option -c nur beim Erstellen der Dateien. Beim Verifizieren der Dateien erkennt Hashdeep die Reihenfolge der Felder am Dateiheader. Das wollte ich im Artikel More about hashdeep options weiter oben klären.

Das Skript DownloadUpdates.cmd kann deshalb Hashdeep-Dateien verifizieren, die im Fast-Mode erstellt wurden, auch wenn es selber diesen optionalen Modus nicht anbietet.

Der einzige Punkt war, dass Hashdeep-Dateien, die im Fast-Mode erstellt wurden, nicht einfach wieder gelöscht werden sollen. Das wird dadurch erreicht, dass jetzt beide Muster getestet werden:

Code: Select all
rem *** delete old-style hashes ***
if exist ..\client\md\nul (
  for /f "delims=" %%f in ('dir /b ..\client\md\hashes-*.txt 2^>nul') do (
    %SystemRoot%\System32\findstr.exe /L /C:"-c md5,sha1,sha256 -b" /C:"-c sha1 -b" "..\client\md\%%f" >nul 2>&1
    if errorlevel 1 (
      del /Q "..\client\md\%%f" >nul 2>&1
    )
  )
)


Das ist, soweit ich weiß, in beiden beta-Versionen enthalten, und das genügt dann auch.
hbuhrmester
 
Posts: 506
Joined: 11.10.2013, 20:59


Return to Linux

Who is online

Users browsing this forum: No registered users and 3 guests