A complete rewrite of the Linux scripts

[hbuhrmester]

Introducing a complete rewrite of the Linux download scripts

I like to introduce a complete rewrite of the Linux download scripts for the project WSUS Offline Update. These scripts offer many improvements over the legacy script DownloadUpdates.sh:

• Separation of a frontend and backend script
  
  The script update-generator.bash is used to interactively select the update, language and download options. The script download-updates.bash fetches the selected updates without any user interaction. This separation makes the structure of both files more straightforward.
  
  
• Highly modular approach
  
  Both scripts are further split into libraries, common tasks, setup tasks and download tasks. Each script does one task only in the most straightforward manner. This resembles the flow of control and makes the scripts easily expandable and more maintainable.
  
  
• Unified language settings
  
  There is no distinction between default languages, custom languages and update languages.
  Users can specify one language on the command line, and then they will get downloads for the specified language only, and nothing more.
  
  
• Verification of downloaded files
  
  SHA-1 hashes are embedded into the filename of all security updates, as a number of 40 hexadecimal digits. These are compared to the checksums, which are calculated by hashdeep.
  The verification of digital file signatures with Sysinternals Sigcheck running under wine was tried, but it doesn't really work without the necessary root certificates.
  
  
• Compatibility
  
  The download script uses the same algorithms for calculating superseded and dynamic updates as the Windows script DownloadUpdates.cmd. The compliance with the Windows scripts can be tested with the scripts compare-integrity-database.bash and compare-update-tables.bash.
  
  
• Desktop integration
  
  Obsolete updates are not deleted immediately, but moved into the trash. GNOME and most other GTK+ based desktop environments use GVFS to handle the trash. The package trash-cli can be used with other desktop environments or window managers. trash-cli should also work without any graphical environment.
  
  
• Self updates of WSUS Offline Update
  
  Both the setup and the download script check for new versions of WSUS Offline Update. They also handle updates of the configuration files in the static and exclude directories.
  
  
• Same day rules
  
  Same day rules are used to prevent the repeated evaluation of the same tasks in adjacent runs of the download script.
  
  
• Documentation
  
  There is even a complete documentation.

Current version

The current version is 1.4, which was released on 2018-04-21. It is compatible with WSUS Offline Update 11.2.2 and later.

An archive, the corresponding hashes file, and the results of a virus scan at VirusTotal can be downloaded at:

http://downloads.hartmut-buhrmester.de/sh-1.4.tgz
http://downloads.hartmut-buhrmester.de/ ... hashes.txt
http://downloads.hartmut-buhrmester.de/ ... report.pdf
http://downloads.hartmut-buhrmester.de/ ... llpage.png

The hashes for the archive sh-1.4.tgz are:

Code: Select all

MD5:     c4ddf148487616d4a3314cc1db9efd03
SHA-256: b1c010ee06f182288a1ff64bfe7c28541eafc7420f5f3b1dde0a2cd7f25cde1a


Please see the installation guides in English ( viewtopic.php?f=9&t=6180#p21449 ) and German ( viewtopic.php?f=9&t=6180#p21450 ) in the following posts for the setup.


Changes

• 2017-01-09 Download links edited to point to version 1.0-beta-2
• 2017-03-30 Download links edited to point to version 1.0-beta-3
• 2017-06-24 Download links edited to point to version 1.0-beta-4
• 2017-08-26 Download links edited to point to version 1.0-beta-5
• 2018-01-19 Download links edited to point to version 1.0
• 2018-01-21 Changed the file ending of the archive from .tar.gz to .tgz
• 2018-02-06 Download links edited to point to version 1.1
• 2018-04-17 Download links edited to point to version 1.2
• 2018-04-19 Download links updated to point to version 1.3
• 2018-04-21 Download links updated to point to version 1.4

[Rush]

can anybody explane me how i used it under ubuntu linux in german please ?

thanks

[hbuhrmester]

Installation guide for the Linux download scripts, version 1.1

WSUS Offline Update already includes the new Linux download scripts. You don't need to install the Linux scripts separately, as it was necessary for the first beta versions, but you should review the needed packages from your Linux distribution.


Install the required and recommended packages

For Debian and Debian-derived distributions, you need to distinguish between the packages md5deep and hashdeep.

The upstream developers moved their project from SourceForge to GitHub, and they renamed their project from md5deep to hashdeep:

http://md5deep.sourceforge.net/
https://github.com/jessek/hashdeep/

Debian followed this move and renamed the package md5deep to hashdeep, starting with Debian 8 Jessie-Backports in summer 2015. The general rule for Debian and Debian-derived distributions then is: Install the package md5deep, if the distribution was released before 2015. Install the package hashdeep for all recent distributions.

• For Debian 7 Wheezy:
  

  Code: Select all

  su -
aptitude install cabextract md5deep wget xmlstarlet trash-cli


  
  
• For Debian 8 Jessie-Backports and newer:
  

  Code: Select all

  su -
aptitude install cabextract hashdeep wget xmlstarlet trash-cli


  
  
• For Ubuntu 14.04LTS Trusty:
  

  Code: Select all

  sudo apt-get install cabextract md5deep wget xmlstarlet trash-cli


  
  
• For Ubuntu 16.04LTS Xenial and newer:
  

  Code: Select all

  sudo apt-get install cabextract hashdeep wget xmlstarlet trash-cli


Other distributions, which are not Debian-based, seem to stay with the package name md5deep.

Note, that both packages md5deep and hashdeep install a series of related applications: hashdeep, md5deep, sha1deep, sha256deep, tigerdeep, and whirlpooldeep. Throughout WSUS Offline Update, you always need the application hashdeep, regardless of the package name.


Download and unpack the wsusoffline archive

Download the newest wsusoffline archive from the download page http://download.wsusoffline.net/ and unpack it. Note, that the zip archive comes with an accompanying hashes files. You can use it to verify the download with:

Code: Select all

hashdeep -a -v -v -l -k wsusoffline1111_hashes.txt wsusoffline1111.zip

The new Linux scripts are included in the "sh" subdirectory. Due to the packaging on Windows, the scripts are not yet executable. Run the included script fix-file-permissions.bash once as:

Code: Select all

bash fix-file-permissions.bash

to make the scripts update-generator.bash, download-updates.bash, get-all-updates.bash, and some others executable.

You can then use the script update-generator.bash to interactively select one update, language and the available options to download. This script doesn't allow multiple selections for update and language, though.

You can also use the script get-all-updates.bash as a template: This script downloads all updates with all available options for the default languages German and English. But it is also meant for customization – you can simply comment out or delete all items you don't need.


Notes

The new Linux scripts don't work alone – they need the configuration files from the wsusoffline installation. Also, the Linux download scripts can only replace the Windows download scripts, e.g. DownloadUpdates.cmd. To install the updates, you surely need the files in the client directory, e.g. the application UpdateInstaller.exe.

Therefore, you should not download the Linux scripts separately, as it was necessary for the first beta versions. Just get the latest wsusoffline archive and find the Linux scripts in the "sh" subdirectory.

If you need to copy or move the wsusoffline directory, please make sure to keep the modification date of all files. You could use "cp --archive" or "cp --preserve" instead of just "cp". This is important for all files throughout WSUS Offline Update.

You can find the complete documentation is in the subdirectory documentation.


Changes
As is custom in some download forums, the first post with the introduction and the two Quick Installation Guides in English and German will be regularly updated for the latest available version. The rest of the discussion just stays in chronological order.

• 2017-01-09 Download links edited to point to version 1.0-beta-2
• 2017-03-30 Download links edited to point to version 1.0-beta-3
• 2017-04-13 Included the section to download the wsusoffline archive first
• 2017-06-24 Installation guide updated to version 1.0-beta-4
• 2017-08-26 Installation guide updated to version 1.0-beta-5
• 2018-01-19 Installation guide updated to version 1.0
• 2018-02-06 The version number was updated to 1.1, but there are no changes in the content

[hbuhrmester]

Installationsanleitung für die Linux-Download-Skripte, Version 1.1

WSUS Offline Update enthält bereits die neuen Linux-Skripte. Sie brauchen die Linux-Skripte deshalb nicht mehr separat zu installieren, wie es für die ersten beta-Versionen nötig war. Sie sollten aber die benötigten Pakete aus ihrer Linux-Distribution überprüfen.


Installieren Sie die benötigten und empfohlenen Pakete

Bei Debian und von Debian abgeleiteten Distributionen müssen Sie zwischen den Paketen md5deep und hashdeep unterscheiden.

Die Entwickler sind mit ihrem Projekt von SourceForge nach GitHub umgezogen, und sie haben ihr Projekt von md5deep in hashdeep umbenannt:

http://md5deep.sourceforge.net/
https://github.com/jessek/hashdeep/

Debian hat diesen Schritt nachvollzogen und das Paket md5deep in hashdeep umbenannt. Dieser Wechsel erfolgte im Sommer 2015 mit den Debian 8 Jessie-Backports. Die allgemeine Regel für Debian und von Debian abgeleitete Distributionen lautet deshalb: Installieren Sie das Paket md5deep, wenn die Distribution älter ist als 2015. Installieren Sie das Paket hashdeep in allen aktuellen Distributionen.

• Für Debian 7 Wheezy:
  

  Code: Select all

  su -
aptitude install cabextract md5deep wget xmlstarlet trash-cli


  
  
• Für Debian 8 Jessie-Backports und neuer:
  

  Code: Select all

  su -
aptitude install cabextract hashdeep wget xmlstarlet trash-cli


  
  
• Für Ubuntu 14.04LTS Trusty:
  

  Code: Select all

  sudo apt-get install cabextract md5deep wget xmlstarlet trash-cli


  
  
• Für Ubuntu 16.04LTS Xenial und neuer:
  

  Code: Select all

  sudo apt-get install cabextract hashdeep wget xmlstarlet trash-cli


Andere Distributionen, die nicht auf Debian basieren, scheinen den Paketnamen md5deep beizubehalten.

Beachten Sie, dass beide Pakete md5deep und hashdeep eine Reihe von ähnlichen Anwendungen installieren: hashdeep, md5deep, sha1deep, sha256deep, tigerdeep und whirlpooldeep. In WSUS Offline Update müssen Sie immer die Anwendung hashdeep verwenden, unabhängig vom Paketnamen.


Laden Sie das wsusoffline-Archiv herunter und entpacken Sie es

Laden Sie das neueste wsusoffline-Archiv von der Downloadseite http://download.wsusoffline.net/ herunter und entpacken Sie es. Beachten Sie, dass das Zip-Archiv von einer Hashes-Datei begleitet wird. Sie können das Archiv damit überprüfen:

Code: Select all

hashdeep -a -v -v -l -k wsusoffline1111_hashes.txt wsusoffline1111.zip

Die neuen Linux-Skripte sind im Verzeichnis "sh" enthalten. Da das wsusoffline-Archiv unter Windows erstellt wurde, sind die Skripte noch nicht ausführbar. Rufen Sie das Skript fix-file-permissions.bash einmal auf mit:

Code: Select all

bash fix-file-permissions.bash

um die Skripte update-generator.bash, download-updates.bash, get-all-updates.bash und einige andere ausführbar zu machen.

Sie können dann das Skript update-generator.bash aufrufen, um interaktiv ein Update, eine Sprache und die verfügbaren Optionen auszuwählen. Dieses Skript erlaubt allerdings keine Mehrfachauswahl für Update und Sprache.

Sie können auch das Skript get-all-updates.bash als Vorlage verwenden: Dieses Skript lädt alle Updates mit allen verfügbaren Optionen für die Standardsprachen Deutsch und Englisch herunter. Es kann beliebig angepasst werden – kommentieren Sie einfach alle Punkte aus, die sie nicht benötigen.


Anmerkungen

Die Linux-Skripte funktionieren nicht alleine – sie benötigen die Konfigurationsdateien der wsusoffline-Installation. Außerdem können die Linux-Download-Skripte nur die Windows-Download-Skripte ersetzen, also zum Beispiel DownloadUpdates.cmd. Um die Downloads zu installieren, werden auch die Dateien im Verzeichnis client benötigt, zum Beispiel der UpdateInstaller.exe.

Sie sollten die Linux-Skripte deshalb nicht mehr separat herunterladen, wie es für die ersten beta-Versionen noch nötig war. Laden Sie einfach das aktuellste wsusoffline-Archiv herunter und benutzen Sie die Skripte im Verzeichnis "sh".

Wenn Sie das Verzeichnis wsusoffline kopieren oder verschieben möchten, achten Sie bitte darauf, das Änderungsdatum aller Dateien beizubehalten. Anstelle "cp" können Sie "cp --archive" oder "cp --preserve" verwenden. Dies ist für die korrekte Funktion von WSUS Offline Update notwendig.

Sie finden die komplette Dokumentation im Verzeichnis documentation.


Änderungen
Wie in manchen Download-Foren üblich, werden der erste Beitrag mit der Einleitung und die Installationsanleitungen in Deutsch und Englisch immer an die aktuelle Version angepasst. Der Rest der Diskussion bleibt in der chronologischen Reihenfolge.

• 2017-01-09 URLs aktualisiert auf die Version 1.0-beta-2
• 2017-03-30 URLs aktualisiert auf die Version 1.0-beta-3
• 2017-06-24 Installationsanleitung überarbeitet für die Version 1.0-beta-4
• 2017-08-26 Installationsanleitung überarbeitet für die Version 1.0-beta-5
• 2018-01-19 Installationsanleitung überarbeitet für die Version 1.0
• 2018-02-06 Die Versionsnummer wurde auf 1.1 aktualisiert, aber der Inhalt ist unverändert

[hbuhrmester]

New version 1.0-beta-2

A new version 1.0-beta-2 of the improved Linux download scripts has been released on 2017-01-09. This version brings two major improvements:

1. The file client/autostart.ini will be rewritten to show an icon of the UpdateInstaller.exe and the built date of the medium. This file only works in Windows, and only, if it is in the root directory of a mounted ISO image, a real CD/DVD or a disk partition.
   
   
2. A configuration variable $prefer_seconly is introduced to prefer security-only update rollups over the full quality update rollups for Windows 7 and Windows Server 2008 R2, Windows Server 2012, Windows 8.1 and Windows Server 2012 R2.
   
   This variable is supposed to be a permanent setting. It is defined and set to disabled in the script download-updates.bash. It should be edited in the file preferences.bash.

The downloads are available at:

http://downloads.hartmut-buhrmester.de/ ... a-2.tar.gz
http://downloads.hartmut-buhrmester.de/ ... beta-2.txt
http://downloads.hartmut-buhrmester.de/ ... beta-2.pdf

Note: The Introduction ( viewtopic.php?f=9&t=6180#p21327 ), Quick Installation Guide ( viewtopic.php?f=9&t=6180#p21449 ) and Kurzinstallationsanleitung ( viewtopic.php?f=9&t=6180#p21450 ) have been edited to point to the new download locations.

[boco]

Maybe the Linux project should better be forked and run as a separate community project. There are more than a few differences between those OS (line endings, separators, case sensitivity etc.).

[crashmaster]

hbuhrmester,

First - Thank you for the excellent rewrite. We've been using it for a couple of weeks, it's fantastic and extremely well done with excellent documentation. It should completely replace the old shell scripts, or as another poster suggested, perhaps forked as there are several improvements and enhancements that we really like.

Second - We noticed that after the script finds a WSUS Offline upgrade available, it asks if you want to upgrade and it defaults to "N". If the user selects "Y", it upgrades wsus offline update and re-runs the comparison. This worked excellent, however is there any particular reason we could not adjust this to be "Y" or "Yes" instead? Or perhaps a new parameter with a boolean option so that we could set so that we update it automatically. We ask this because we are rarely at the console when we run your scripts - they are scheduled via cron. We'd rather it be auto upgraded if practical.

Third - Procedure to upgrade your sh-new / linux scripts themselves. I see there's a beta 2 version available. Perhaps I'm missing it in the preferences, or perhaps it's obvious - but is there an autoupdate mechanism/documentation for your scripts, or should we follow the install instructions and reinstall and just monitor this forum for changes?

Thank you!

[WSUSUpdateAdmin]

Moin!

Das wird auf jeden Fall kommen.
Ich möchte nur "falcon" nicht vorgreifen bzw. in die Parade fahren, der sich ja dankenswerterweise bereit erklärt hat, die Pflege der Linux-Skripte zu übernehmen (vgl. viewtopic.php?f=9&t=5955).

Vielen Dank & viele Grüße,
Torsten

[hbuhrmester]

Release Notes for Version 1.0-beta-3

Release date: 2017-03-30
Intended compatibility: WSUS Offline Update version 10.9.1 - 10.9.2

This is a maintenance release to keep up with WSUS Offline Update 10.9.1 and 10.9.2. It offers the following changes:

• Some more Service Packs are excluded, if the option -includesp is not used.
  
  This was suggested by negg in viewtopic.php?f=3&t=6143 .
  
  If the option -includesp for Service Packs is missing, then the ExcludeList-SPs.txt is applied to both statically defined and dynamically determined updates. Previously, it was only applied to statically defined updates. The result is, that some more updates are excluded, if the option -includesp is not used.
  
  In the Windows version, this corresponds to the option Include Service Packs in UpdateGenerator.exe.
  
  
• Empty download directories are now deleted, along with the corresponding hashes file.
  
  If the option -includesp is missing, then some download directories may end up empty, for example:
  
  

  Code: Select all

  wsusoffline/client/o2k7/deu
wsusoffline/client/o2k10/deu
wsusoffline/client/o2k13/deu


  
  
  Currently, these directories contain Service Packs only. If Service Packs are excluded, then these directories should be empty.
  
  The correct handling of empty directories is to delete the directories and the corresponding hashdeep files. This is the same approach as in the Windows script DownloadUpdates.cmd.
  
  Note, however, that the cleanup function in the Linux scripts does not delete existing Service Packs, if they are still referenced from the static directory. Instead, they are only reported as valid static files. In this case, they must be removed manually and then they won't get downloaded again.
  
  This was introduced to keep localized downloads, if different languages are downloaded in turn, but the same mechanism also protects existing downloads in a few similar cases.
  
  
• A bug in the handling of empty directories was found and fixed.
  
  grep is used to extract information from text files. In most cases, only the standard output of grep is needed. grep also sets a result code, but this information is not really needed, if the standard output is used. A result code of "1" means, that there are no matching lines, and that the standard output will be empty. This information is redundant and not an error by itself.
  
  But a result code of "1" causes an error, if the shell option errexit or a trap on ERR is used. Then the result code must be masked like:
  
  

  Code: Select all

  grep ... || true

  
  
  
• The script 40-check-for-self-updates.bash was split into two smaller scripts, to handle its tasks separately.
  
  In the previous version 1.0-beta-2, the script 40-check-for-self-updates.bash handled both version updates for WSUS Offline Update and the update of the configuration files in the static and exclude directories.
  
  These tasks are now handled by two smaller scripts:
  
  

  Code: Select all

  50-check-wsusoffline-version.bash
70-update-configuration-files.bash


  
  
  The Windows version uses different scripts for these tasks as well: The application UpdateGenerator.exe initiates a version check by calling CheckOUVersion.cmd. A self update is done by the script UpdateOU.new, which will be renamed to UpdateOU.cmd. The script DownloadUpdates.cmd handles the updates of the configuration files in the static and exclude directories.
  
  
• An online check for new versions of the Linux scripts is introduced with the new script 60-check-script-version.bash.
  
  This feature was requested by crashmaster in viewtopic.php?f=9&t=6180#p21683 .
  
  The approach is quite similar to the version check for WSUS Offline Update: The file installed-version.txt is included in the archive, and a second file available-version.txt will be downloaded from the Internet. If these files differ, then a new version of the Linux scripts is available. The file available-version.txt has the necessary information to download and install the new version.
  
  But maybe this option should be tested some more. Therefore, the script 60-check-script-version.bash has been moved to the directory available-tasks. To enable it, it must be moved to the directory common-tasks.
  
  
• A new configuration variable "unattended_updates" is introduced to install new versions of WSUS Offline Update or the Linux scripts automatically.
  
  This was also a suggestion by crashmaster.
  
  By default, the scripts 50-check-wsusoffline-version.bash and 60-check-script-version.bash will not install new versions of WSUS Offline Update or the Linux scripts without confirmation. Both scripts report new versions and then ask for confirmation to install them. After 30 seconds this question defaults to "no".
  
  Then these scripts won't be blocked and wait forever, but if nobody is watching, new versions won't get installed.
  
  The new configuration variable unattended_updates changes this behavior: new versions are reported, and the scripts still ask for confirmation, but after 30 seconds this question defaults to "yes" and then new versions are installed anyway. This may be better suited for cron jobs and similar automated tasks.
  
  The variable unattended_updates is defined and set to "disabled" in the scripts download-updates.bash and update-generator.bash. It should be changed to "enabled" in the file preferences.bash.
  
  
• A new file fix-file-permissions.bash was added to make the Linux scripts executable again, should they loose their file permissions.
  
  This is not necessary by now and not, if the Linux scripts are extracted from the original tar.gz archive. It may become necessary, if Linux scripts are included in a zip archive, which was created on Windows. Then Linux scripts will loose their file permissions and not be executable any more.
  
  The script fix-file-permissions.bash should correct that; but since it is also affected, it must be run once within the installation directory with:
  
  

  Code: Select all

  bash fix-file-permissions.bash

  
  
  zip archives created on Linux will preserve Linux file permissions; for example, the zip files available at GitHub work without such precautions.

Note: The original post and the quick installation guides in German and English, earlier in this thread, have been modified to point to version 1.0-beta-3.

[WSUSUpdateAdmin]

Moin!

Ohne die Linux-Download-Skripte selbst zu benutzen und auch ohne die Rückmeldung von "falcon" gehe ich einfach mal davon aus, dass die hier vorgestellte Neuimplementierung eine deutliche Verbesserung darstellt und habe sie deswegen jetzt eingecheckt.

Besten Dank an Hartmut!

Viele Grüße
Torsten