A complete rewrite of the Linux scripts

A complete rewrite of the Linux scripts

Postby hbuhrmester » 14.12.2016, 13:53

Introducing a complete rewrite of the Linux download scripts

I like to introduce a complete rewrite of the Linux download scripts for the project WSUS Offline Update. These scripts offer many improvements over the legacy script DownloadUpdates.sh:

  • Separation of a frontend and backend script

    The script update-generator.bash is used to interactively select the update, language and download options. The script download-updates.bash fetches the selected updates without any user interaction. This separation makes the structure of both files more straightforward.

  • Highly modular approach

    Both scripts are further split into libraries, common tasks, setup tasks and download tasks. Each script does one task only in the most straightforward manner. This resembles the flow of control and makes the scripts easily expandable and more maintainable.

  • Unified language settings

    There is no distinction between default languages, custom languages and update languages.
    Users can specify one language on the command line, and then they will get downloads for the specified language only, and nothing more.

  • Verification of downloaded files

    SHA-1 hashes are embedded into the filename of all security updates, as a number of 40 hexadecimal digits. These are compared to the checksums, which are calculated by hashdeep.
    The verification of digital file signatures with Sysinternals Sigcheck running under wine was tried, but it doesn't really work without the necessary root certificates.

  • Compatibility

    The download script uses the same algorithms for calculating superseded and dynamic updates as the Windows script DownloadUpdates.cmd. The compliance with the Windows scripts can be tested with the scripts compare-integrity-database.bash and compare-update-tables.bash.

  • Desktop integration

    Obsolete updates are not deleted immediately, but moved into the trash. GNOME and most other GTK+ based desktop environments use GVFS to handle the trash. The package trash-cli can be used with other desktop environments or window managers. trash-cli should also work without any graphical environment.

  • Self updates of WSUS Offline Update

    Both the setup and the download script check for new versions of WSUS Offline Update. They also handle updates of the configuration files in the static and exclude directories.

  • Same day rules

    Same day rules are used to prevent the repeated evaluation of the same tasks in adjacent runs of the download script.

  • Documentation

    There is even a complete documentation.


Download and installation

The current version is 1.0-beta-3, which was released on 2017-03-30. It is compatible with WSUS Offline Update 10.9.1 - 10.9.2.

An archive and the corresponding hashes file can be downloaded at:

http://downloads.hartmut-buhrmester.de/ ... a-3.tar.gz
http://downloads.hartmut-buhrmester.de/ ... beta-3.txt
http://downloads.hartmut-buhrmester.de/ ... beta-3.pdf

Verify the archive with:

Code: Select all
hashdeep -a -v -v -l -k hashes-sh-new-1.0-beta-3.txt sh-new-1.0-beta-3.tar.gz


Then unpack it into the wsusoffline directory. It should create a directory "sh-new-1.0-beta-3" along the existing directory "sh".



Changes
  • 2017-01-09 Download links edited to point to version 1.0-beta-2
  • 2017-03-30 Download links edited to point to version 1.0-beta-3
Last edited by hbuhrmester on 30.03.2017, 19:14, edited 2 times in total.
Reason: Download links updated for version 1.0-beta-3
hbuhrmester
 
Posts: 207
Joined: 11.10.2013, 20:59

Re: A complete rewrite of the Linux scripts

Postby Rush » 20.12.2016, 20:47

can anybody explane me how i used it under ubuntu linux in german please :) ?

thanks :)
Rush
 
Posts: 48
Joined: 09.08.2015, 19:05

Quick installation guide

Postby hbuhrmester » 22.12.2016, 15:38

Quick installation guide

The new Linux download scripts still need the configuration files of the WSUS Offline Update installation, to calculate static and dynamic update lists. These are the files in the directories static, exclude, client/static, client/exclude, and xslt.

Also, the Linux download scripts can only replace the download part. To install the updates, you surely need the UpdateInstaller.exe and all other files in the client directory.

With the current versions of WSUS Offline Update (10.9.2) and the Linux scripts (1.0-beta-3), it is necessary to download and unpack the wsusoffline archive first. This may change with the next versions, as discussed later in this thread ( viewtopic.php?f=9&t=6180&start=10#p22583 ).

Install the required and recommended packages

For Debian and Debian-derived distributions, you may have to differentiate between the packages md5deep and hashdeep.

The upstream developers moved their project from SourceForge to GitHub, and they renamed the project from md5deep to hashdeep:

http://md5deep.sourceforge.net/
https://github.com/jessek/hashdeep/

Debian followed this move and renamed the package md5deep to hashdeep, starting with Debian 8 Jessie-Backports in summer 2015. The general rule for Debian and Debian-derived distributions then is: Install the package md5deep, if the distribution was released before 2015. Install the package hashdeep for all recent distributions.

  • For Debian 7 Wheezy:
    Code: Select all
    su -
    aptitude install cabextract md5deep wget xmlstarlet trash-cli

  • For Debian 8 Jessie-Backports and later:
    Code: Select all
    su -
    aptitude install cabextract hashdeep wget xmlstarlet trash-cli

  • For Ubuntu 14.04LTS Trusty:
    Code: Select all
    sudo apt-get install cabextract md5deep wget xmlstarlet trash-cli

  • For Ubuntu 16.04LTS Xenial and later:
    Code: Select all
    sudo apt-get install cabextract hashdeep wget xmlstarlet trash-cli

Other distributions, which are not Debian-based, seem to stay with the package name md5deep.

Note, that both packages md5deep and hashdeep install a series of related applications: hashdeep, md5deep, sha1deep, sha256deep, tigerdeep, and whirlpooldeep. For the next steps, you always need the application hashdeep, regardless of the package name.


Download and unpack the wsusoffline archive

  • Refer to the download page http://download.wsusoffline.net/ , to find the most recent version of WSUS Offline Update. For version 10.9.2, you would download the following archive and hashes file:
    Code: Select all
    wget http://download.wsusoffline.net/wsusoffline1092.zip
    wget http://download.wsusoffline.net/wsusoffline1092_hashes.txt

  • Verify the archive with:
    Code: Select all
    hashdeep -a -v -v -l -k wsusoffline1092_hashes.txt wsusoffline1092.zip

  • Unpack the archive:
    Code: Select all
    unzip wsusoffline1092.zip

    This will create a new directory wsusoffline.

  • Change to the directory wsusoffline:
    Code: Select all
    cd wsusoffline

Download and install the archive for the new Linux scripts

  • Download the archiv and the hashes file to the directory wsusoffline:
    Code: Select all
    wget http://downloads.hartmut-buhrmester.de/sh-new-1.0-beta-3.tar.gz
    wget http://downloads.hartmut-buhrmester.de/hashes-sh-new-1.0-beta-3.txt

  • Verify the integrity of the archive:
    Code: Select all
    hashdeep -a -v -v -l -k hashes-sh-new-1.0-beta-3.txt sh-new-1.0-beta-3.tar.gz

  • Unpack the archive in the directory wsusoffline:
    Code: Select all
    tar xvzf sh-new-1.0-beta-3.tar.gz

    This will create a new directory sh-new-1.0-beta-3.

  • Change to the directory sh-new-1.0-beta-3 and run the script update-generator.bash, to interactively select the updates to fetch:
    Code: Select all
    cd sh-new-1.0-beta-3
    ./update-generator.bash

Note: the complete documentation is in the subdirectory documentation.


Changes
As is custom in some download forums, the first post with the introduction and the two Quick Installation Guides in English and German will be regularly updated for the latest available version. The rest of the discussion just stays in chronological order.

  • 2017-01-09 Download links edited to point to version 1.0-beta-2
  • 2017-03-30 Download links edited to point to version 1.0-beta-3
  • 2017-04-13 Included a new section to download the wsusoffline archive first
Last edited by hbuhrmester on 15.04.2017, 06:51, edited 7 times in total.
Reason: Download links updated for version 1.0-beta-3
hbuhrmester
 
Posts: 207
Joined: 11.10.2013, 20:59

Kurzinstallationsanleitung

Postby hbuhrmester » 22.12.2016, 15:39

Kurzinstallationsanleitung

Installiere die benötigten und empfohlenen Pakete...

für Debian:
Code: Select all
su -
aptitude install cabextract hashdeep wget xmlstarlet trash-cli


für Ubuntu:
Code: Select all
sudo apt-get install cabextract hashdeep wget xmlstarlet trash-cli


Lade das Archiv und die Prüfsummendatei in das Verzeichnis wsusoffline herunter. Dies ist das Verzeichnis, in dem sich das Windows-Programm UpdateGenerator.exe befindet.
Code: Select all
wget http://downloads.hartmut-buhrmester.de/sh-new-1.0-beta-3.tar.gz
wget http://downloads.hartmut-buhrmester.de/hashes-sh-new-1.0-beta-3.txt


Überprüfe die Integrität des Archivs:
Code: Select all
hashdeep -a -v -v -l -k hashes-sh-new-1.0-beta-3.txt sh-new-1.0-beta-3.tar.gz


Entpacke das Archiv im Verzeichnis wsusoffline:
Code: Select all
tar xvzf sh-new-1.0-beta-3.tar.gz


Wechsle in das Verzeichnis sh-new-1.0-beta-3 und starte das Skript update-generator.bash, um die gewünschten Updates interaktiv auszuwählen:
Code: Select all
cd sh-new-1.0-beta-3
./update-generator.bash


Hinweis: Die vollständige Dokumentation befindet sich im Unterverzeichnis documentation.


Änderungen
  • 09.01.2017 Anpassung der Download-Links an die Version 1.0-beta-2
  • 30.03.2017 Anpassung der Download-Links an die Version 1.0-beta-3
Last edited by hbuhrmester on 30.03.2017, 19:56, edited 3 times in total.
Reason: Download links updated for version 1.0-beta-3
hbuhrmester
 
Posts: 207
Joined: 11.10.2013, 20:59

New version 1.0-beta-2

Postby hbuhrmester » 09.01.2017, 13:08

New version 1.0-beta-2

A new version 1.0-beta-2 of the improved Linux download scripts has been released on 2017-01-09. This version brings two major improvements:

  1. The file client/autostart.ini will be rewritten to show an icon of the UpdateInstaller.exe and the built date of the medium. This file only works in Windows, and only, if it is in the root directory of a mounted ISO image, a real CD/DVD or a disk partition.

  2. A configuration variable $prefer_seconly is introduced to prefer security-only update rollups over the full quality update rollups for Windows 7 and Windows Server 2008 R2, Windows Server 2012, Windows 8.1 and Windows Server 2012 R2.

    This variable is supposed to be a permanent setting. It is defined and set to disabled in the script download-updates.bash. It should be edited in the file preferences.bash.

The downloads are available at:

http://downloads.hartmut-buhrmester.de/ ... a-2.tar.gz
http://downloads.hartmut-buhrmester.de/ ... beta-2.txt
http://downloads.hartmut-buhrmester.de/ ... beta-2.pdf

Note: The Introduction ( viewtopic.php?f=9&t=6180#p21327 ), Quick Installation Guide ( viewtopic.php?f=9&t=6180#p21449 ) and Kurzinstallationsanleitung ( viewtopic.php?f=9&t=6180#p21450 ) have been edited to point to the new download locations.
hbuhrmester
 
Posts: 207
Joined: 11.10.2013, 20:59

Re: A complete rewrite of the Linux scripts

Postby boco » 09.01.2017, 16:24

Maybe the Linux project should better be forked and run as a separate community project. There are more than a few differences between those OS (line endings, separators, case sensitivity etc.).
Microsoft update catalog: http://catalog.update.microsoft.com/v7/site/
Windows Install media creator: https://support.microsoft.com/en-us/help/15088/windows-create-installation-media
boco
 
Posts: 1742
Joined: 24.11.2009, 17:00
Location: Germany

Re: A complete rewrite of the Linux scripts

Postby crashmaster » 11.01.2017, 19:58

hbuhrmester,

First - Thank you for the excellent rewrite. We've been using it for a couple of weeks, it's fantastic and extremely well done with excellent documentation. It should completely replace the old shell scripts, or as another poster suggested, perhaps forked as there are several improvements and enhancements that we really like.

Second - We noticed that after the script finds a WSUS Offline upgrade available, it asks if you want to upgrade and it defaults to "N". If the user selects "Y", it upgrades wsus offline update and re-runs the comparison. This worked excellent, however is there any particular reason we could not adjust this to be "Y" or "Yes" instead? Or perhaps a new parameter with a boolean option so that we could set so that we update it automatically. We ask this because we are rarely at the console when we run your scripts - they are scheduled via cron. We'd rather it be auto upgraded if practical.

Third - Procedure to upgrade your sh-new / linux scripts themselves. I see there's a beta 2 version available. Perhaps I'm missing it in the preferences, or perhaps it's obvious - but is there an autoupdate mechanism/documentation for your scripts, or should we follow the install instructions and reinstall and just monitor this forum for changes?

Thank you!
crashmaster
 
Posts: 1
Joined: 11.01.2017, 18:32

Re: A complete rewrite of the Linux scripts

Postby WSUSUpdateAdmin » 23.01.2017, 14:22

Moin!

Das wird auf jeden Fall kommen.
Ich möchte nur "falcon" nicht vorgreifen bzw. in die Parade fahren, der sich ja dankenswerterweise bereit erklärt hat, die Pflege der Linux-Skripte zu übernehmen (vgl. viewtopic.php?f=9&t=5955).

Vielen Dank & viele Grüße,
Torsten
WSUSUpdateAdmin
Administrator
 
Posts: 1988
Joined: 07.07.2009, 14:38

Release Notes for Version 1.0-beta-3

Postby hbuhrmester » 30.03.2017, 19:29

Release Notes for Version 1.0-beta-3

Release date: 2017-03-30
Intended compatibility: WSUS Offline Update version 10.9.1 - 10.9.2

This is a maintenance release to keep up with WSUS Offline Update 10.9.1 and 10.9.2. It offers the following changes:

  • Some more Service Packs are excluded, if the option -includesp is not used.

    This was suggested by negg in viewtopic.php?f=3&t=6143 .

    If the option -includesp for Service Packs is missing, then the ExcludeList-SPs.txt is applied to both statically defined and dynamically determined updates. Previously, it was only applied to statically defined updates. The result is, that some more updates are excluded, if the option -includesp is not used.

    In the Windows version, this corresponds to the option Include Service Packs in UpdateGenerator.exe.

  • Empty download directories are now deleted, along with the corresponding hashes file.

    If the option -includesp is missing, then some download directories may end up empty, for example:

    Code: Select all
    wsusoffline/client/o2k7/deu
    wsusoffline/client/o2k10/deu
    wsusoffline/client/o2k13/deu


    Currently, these directories contain Service Packs only. If Service Packs are excluded, then these directories should be empty.

    The correct handling of empty directories is to delete the directories and the corresponding hashdeep files. This is the same approach as in the Windows script DownloadUpdates.cmd.

    Note, however, that the cleanup function in the Linux scripts does not delete existing Service Packs, if they are still referenced from the static directory. Instead, they are only reported as valid static files. In this case, they must be removed manually and then they won't get downloaded again.

    This was introduced to keep localized downloads, if different languages are downloaded in turn, but the same mechanism also protects existing downloads in a few similar cases.

  • A bug in the handling of empty directories was found and fixed.

    grep is used to extract information from text files. In most cases, only the standard output of grep is needed. grep also sets a result code, but this information is not really needed, if the standard output is used. A result code of "1" means, that there are no matching lines, and that the standard output will be empty. This information is redundant and not an error by itself.

    But a result code of "1" causes an error, if the shell option errexit or a trap on ERR is used. Then the result code must be masked like:

    Code: Select all
    grep ... || true


  • The script 40-check-for-self-updates.bash was split into two smaller scripts, to handle its tasks separately.

    In the previous version 1.0-beta-2, the script 40-check-for-self-updates.bash handled both version updates for WSUS Offline Update and the update of the configuration files in the static and exclude directories.

    These tasks are now handled by two smaller scripts:

    Code: Select all
    50-check-wsusoffline-version.bash
    70-update-configuration-files.bash


    The Windows version uses different scripts for these tasks as well: The application UpdateGenerator.exe initiates a version check by calling CheckOUVersion.cmd. A self update is done by the script UpdateOU.new, which will be renamed to UpdateOU.cmd. The script DownloadUpdates.cmd handles the updates of the configuration files in the static and exclude directories.

  • An online check for new versions of the Linux scripts is introduced with the new script 60-check-script-version.bash.

    This feature was requested by crashmaster in viewtopic.php?f=9&t=6180#p21683 .

    The approach is quite similar to the version check for WSUS Offline Update: The file installed-version.txt is included in the archive, and a second file available-version.txt will be downloaded from the Internet. If these files differ, then a new version of the Linux scripts is available. The file available-version.txt has the necessary information to download and install the new version.

    But maybe this option should be tested some more. Therefore, the script 60-check-script-version.bash has been moved to the directory available-tasks. To enable it, it must be moved to the directory common-tasks.

  • A new configuration variable "unattended_updates" is introduced to install new versions of WSUS Offline Update or the Linux scripts automatically.

    This was also a suggestion by crashmaster.

    By default, the scripts 50-check-wsusoffline-version.bash and 60-check-script-version.bash will not install new versions of WSUS Offline Update or the Linux scripts without confirmation. Both scripts report new versions and then ask for confirmation to install them. After 30 seconds this question defaults to "no".

    Then these scripts won't be blocked and wait forever, but if nobody is watching, new versions won't get installed.

    The new configuration variable unattended_updates changes this behavior: new versions are reported, and the scripts still ask for confirmation, but after 30 seconds this question defaults to "yes" and then new versions are installed anyway. This may be better suited for cron jobs and similar automated tasks.

    The variable unattended_updates is defined and set to "disabled" in the scripts download-updates.bash and update-generator.bash. It should be changed to "enabled" in the file preferences.bash.

  • A new file fix-file-permissions.bash was added to make the Linux scripts executable again, should they loose their file permissions.

    This is not necessary by now and not, if the Linux scripts are extracted from the original tar.gz archive. It may become necessary, if Linux scripts are included in a zip archive, which was created on Windows. Then Linux scripts will loose their file permissions and not be executable any more.

    The script fix-file-permissions.bash should correct that; but since it is also affected, it must be run once within the installation directory with:

    Code: Select all
    bash fix-file-permissions.bash


    zip archives created on Linux will preserve Linux file permissions; for example, the zip files available at GitHub work without such precautions.


Note: The original post and the quick installation guides in German and English, earlier in this thread, have been modified to point to version 1.0-beta-3.
hbuhrmester
 
Posts: 207
Joined: 11.10.2013, 20:59

Re: A complete rewrite of the Linux scripts

Postby WSUSUpdateAdmin » 04.04.2017, 10:36

Moin!

Ohne die Linux-Download-Skripte selbst zu benutzen und auch ohne die Rückmeldung von "falcon" gehe ich einfach mal davon aus, dass die hier vorgestellte Neuimplementierung eine deutliche Verbesserung darstellt und habe sie deswegen jetzt eingecheckt.

Besten Dank an Hartmut! :D

Viele Grüße
Torsten
WSUSUpdateAdmin
Administrator
 
Posts: 1988
Joined: 07.07.2009, 14:38

Next

Return to Linux

Who is online

Users browsing this forum: No registered users and 2 guests