NEWS for version 1.11 (2019-04-04)

NEWS for version 1.11 (2019-04-04)

Postby hbuhrmester » 04.04.2019, 14:12

NEWS for version 1.11

Release date: 2019-04-04
Intended compatibility: WSUS Offline Update 11.6.1 and later

New features

The script update-generator.bash remembers the last used settings

The script update-generator.bash writes the current settings to a settings file update-generator.ini, if the external utility “dialog” is used. On the next run, the settings will be reloaded and the previously selected options are checked again.

Note, that this does not work with the internal command “select” of the bash.

Revised method for calculating superseded updates

An optional, new method for the calculation of superseded updates removes one possible cause for missing updates: It automatically corrects the list of superseded updates for updates, which are only superseded by full quality update rollups, but not by security-only updates. This may prevent some rare problems with missing updates.

The current implementation for calculating superseded updates in both Windows and Linux is depicted in the forum article:


One problem with this implementation is, that superseded updates may be missing, if the superseding updates are excluded from download.

This is an old problem, which was observed a few times:

It was first found with Windows XP: The embedded Windows XP POSReady was supported longer than the regular desktop versions. Updates for the embedded version can supersede older updates for the desktop versions. The newer updates for the embedded version are not downloaded by WSUS Offline Update, because they cannot be installed on the desktop versions. But the older updates for the desktop versions are still treated as superseded and not downloaded either. This was solved by adding the missing updates to the file ExcludeList-superseded-exclude.txt, which will reschedule them for download.

When monthly quality update rollups and security-only updates were first introduced for Window 7, 8 and 8.1, the quality update rollups superseded the security-only updates. WSUS Offline Update needed two steps to support security-only updates:

  • The quality update rollups had to be excluded from download and installation.
  • The security-only updates had to be re-enabled for download.

At this point, I first suggested a new method, which could reschedule superseded updates for download, if the superseding updates are excluded from download. The problem with this method was, that it needed an initial block list of excluded downloads to start with, and this list didn't exist yet. So it was only a partial solution.


The initial release of the Linux download scripts, version 1.0-beta-1, actually included an implementation of the new method in the subdirectory available-tasks.

The differentiation of quality update rollups and security-only updates was finally solved in WSUS Offline Update by creating new configuration files in the client/static and client/exclude directories.

In the meantime, Microsoft changed the way, how quality update rollups and security-only updates depend on each other, after just one month:

“UPDATED 12/5/2016: Starting in December 2016, monthly rollups will not supersede security only updates. The November 2016 monthly rollup will also be updated to not supersede security only updates.”

https://techcommunity.microsoft.com/t5/Windows-Blog-Archive/More-on-Windows-7-and-Windows-8-1-servicing-changes/ba-p/166783


A more subtle problem is, that the quality update rollups sometimes seem to include older updates, while the security-only updates only include new updates for the current month. Then these older updates will be missing, if security-only updates are selected. Again, this was solved by adding missing updates to a new configuration file ExcludeList-superseded-exclude-seconly.txt.

The problems with the files ExcludeList-superseded-exclude.txt and ExcludeList-superseded-exclude-seconly.txt is, that they can only be updated, after some updates have been found missing.

When I retested the new method, it could detect two missing updates, before they were reported as missing:


So, this method may still be useful, and the script 60-main-updates.bash now includes an optional implementation. It uses the file HideList-seconly.txt as an initial block list, to automatically correct the list of superseded updates for updates, which are only superseded by the monthly quality update rollups, but not by security-only updates.

To select the new file ExcludeList-Linux-superseded-seconly-revised.txt for the calculation of dynamic updates, both options prefer_seconly and revised_method must be set to “enabled” in the preferences file.

New script open-support-pages.bash

This script opens the Microsoft support pages for a series of kb numbers.

It tries a series of Linux “open handlers”, to open the URLs with the preferred application of the desktop environment. Suitable open handlers are:

Code: Select all
Open handler      Package name        Desktop environment

gio open          libglib2.0-bin      GNOME 3.30 in Debian 10 Buster/testing
gvfs-open         gvfs-bin            GNOME 3.22 in Debian 9 Stretch/stable
gnome-open        libgnome2-bin       GNOME 2
kde-open5         kde-cli-tools       KDE 5 (untested)
kde-open          kde-runtime         KDE 4 (untested)
exo-open          exo-utils           Xfce
xdg-open          xdg-utils           others

(The package names are for Debian and related distributions.)


In addition to these open handlers, sensible-browser, firefox-esr and firefox are also tried, because this script only needs to handle http or https URLs. The script /usr/bin/sensible-browser is part of the update-alternatives system in Debian. It uses gnome-www-browser, x-www-browser or www-browser, depending on the context.

If none of the above can be found, then the script recommends the installation of xdg-open as a general open handler, which is not tied to a particular desktop environment.

Note: Neither gvfs-open nor Firefox can handle multiple URLs on the command line. Calling xdg-open repeatedly would mean, that the application should be launched multiple times, but this will fail with Firefox. To have Firefox open the URLs in multiple tabs, it must be launched first, before running this script.

Bug fixes

Workaround for broken cabextract in Debian 10 Buster/testing, thanks to mani and slycordinator

As of 2019-03-26, the package cabextract is still broken in Debian 10 Buster/testing, long after two relevant bug reports have been marked as fixed and closed.


But so far, the fixed packages only arrived in Debian Sid/unstable. They are not yet available in Debian Buster/testing.

So I added some more tests and a workaround, which does not use the cabextract option -F. Then the file wsusscn2.cab must be completely unpacked, which may take slightly longer.


User visible changes

Better support for terminal colors

Version 1.4 of the Linux download scripts introduced text formatting of the output, using bold text and terminal colors. Such text formatting should only be used, if the output is written to a terminal emulator window or to a Linux virtual console. It should not be used, if the output is redirected to a file or piped to another application, for example if the script is running as a cron job.

The library messages.bash uses two tests, to make sure that text formatting can used safely:

  • First, the script tests, if standard output and error output are attached to a terminal, using the test -t of POSIX shells.
  • Then all escape sequences are determined with the utility tput, rather than hard-coding them. tput checks again, if text formatting is safe to use.

But tput is overly restrictive with the use of terminal colors: It only uses colors, if the environment variable TERM is set to “xterm-256color”, “rxvt-256color” or “rxvt-unicode-256color”. Otherwise, only bold text is used.

Many terminal emulators simply set TERM to “xterm”, and they don't provide any means to change this environment variable. Still, all tested terminal emulators support colors, including xterm itself. Then it should be safe to change TERM from xterm or xterm-color to xterm-256color, to get the expected results.

The same adjustments could be done with rxvt, but Debian already provides different builds with different settings for rxvt. The urxvt from the package rxvt-unicode-256color sets the environment variable TERM to “rxvt-unicode-256color”, which is recognized as a color-capable terminal by tput.

Notes:

There are other ways to set or change the environment variable TERM. This may benefit other applications as well. For example, some themes for the Midnight Commander also require 256 colors.

Within a shell, environment variables can be defined before the script or application to run:

Code: Select all
~$ TERM=xterm-256color ./update-generator.bash


Some terminal emulators like the MATE Terminal allow to run a custom command instead of the standard shell. This can be used to set environment variables with:

Code: Select all
/usr/bin/env TERM=xterm-256color bash


The environment variable TERM could also be set in files like ~/.profile or ~/.bashrc, but this may give unexpected results: Using TERM=xterm in the Linux console will mess up the output of the external utility “dialog”, because the box drawing characters are different.

Support for ExcludeListForce-all.txt

The custom file wsusoffline/exclude/custom/ExcludeListForce-all.txt was already applied to static and dynamic updates for Windows, Office and .NET Frameworks. It is now applied to the .NET Framework installation files as well.


Support for ExcludeList-superseded-exclude-seconly.txt

The custom file wsusoffline/exclude/custom/ExcludeList-superseded-exclude-seconly.txt will be used for the calculation of superseded updates, if security-only updates are selected.


Virus definition files are tested with cabextract

The four virus definition files are basically self-extracting cabinet files. They can be at least partially tested with cabextract -t.

The scripts compare-integrity-database.bash and compare-update-tables.bash are configured with command-line arguments

These scripts are used for development: They compare the directories md and ofc on Windows and Linux. Previously, these directories had to be edited directly in the scripts, but they are now passed as command-line arguments.

Internal changes

Reordered the sections at the top of the scripts update-generator.bash and download-updates.bash

The new order in these files is:

  1. Shell options
  2. Environment variables
  3. Configuration: script version and release date
  4. Global variables: script name and home directory, other directories and log file
  5. Preferences: default values for the settings in the optional preferences file
  6. Traps
  7. Functions
  8. Commands

The directories cache, log and timestamps are stored with absolute paths

The directories cache, log and timestamps used to be defined with the relative paths ../cache, ../log and ../timestamps. This does not work well with external utilities like hashdeep and curl, which often require changes to the current working directory and thereby make relative references invalid.

After revealing the current working directory with readlink, these directories are now stored with absolute paths.

The function seconly_safety_guard calculates the patch days for two months

The function seconly_safety_guard tries to make sure, that some configuration files in the directories client/exclude and client/static have been updated after each patch day, before downloading security-only updates. Without this configuration, WSUS Offline Update would default to download and install the full quality update rollups, or it might even download and install both sets of updates.

The first implementation only calculated the patch day of the current month, and compared the modification date of the configuration files to this date.

Now the function calculates the official patch days for the last two months, and selects the right one for comparison:

  • The last patch day is the second Tuesday of the current month, if today is on that day or later.
  • Otherwise, the last patch day is the second Tuesday of the last month.

These calculations also use an integer value for the day of the week, instead of comparing the weekday names literally.

The function log_message duplicates messages to the terminal and the log file unchanged

The function log_message does not prefix the message with the current date anymore. It is now used to duplicate the output of hashdeep to the terminal and to the log file.

The function apply_exclude_lists now skips empty lines in input files

The function apply_exclude_lists now reads the input files line-by-line. Empty lines are ignored. This should prevent errors with files, which only consist of one empty line.

New function name_to_description and changed function language_name_to_locale

The new function name_to_description checks, that the specified update, language or option name exists in a table and returns the description. This replaces some awkward constructs with grep.

The function language_name_to_locale was rewritten in the same way to return the locale for a language name, e.g. deu → de, enu → en.

The file sh/exclude/ExcludeListUSB-w60.txt was simplified

The different filters vcredist* were replaced with a single filter *_x64*.


Added Cygwin to known systems and unzip to needed packages, thanks to slycordinator

Cygwin is basically a Linux environment on Windows. All needed GNU utilities and other packages should be available. trash-cli can be installed from the Python repository with "pip install trash-cli".

unzip is needed for the self-update of the WSUS Offline Update installation and to unpack the Sysinternals utilities Autologon and Sigcheck. unzip is often installed with graphical archive managers like Xarchiver, but it may be missing in a basic command-line system.


Documentation

The new file compatibility.txt lists the distributions, on which the Linux scripts were at least briefly tested, and the corresponding Bash versions.

Some typos were corrected.

The E-Mail address was replaced.


Downloads

The download links for this version are:

http://downloads.hartmut-buhrmester.de/sh-1.11.tgz
http://downloads.hartmut-buhrmester.de/ ... hashes.txt
http://downloads.hartmut-buhrmester.de/ ... report.pdf
http://downloads.hartmut-buhrmester.de/ ... llpage.png


The hashes for the archive sh-1.11.tgz are:
Code: Select all
MD5:     0fea2bc2c57edcfa1f2208dafa0d0fa5
SHA-256: 556380039cf5ab62ec45a2f23e4eb9dc2a0cc149e24524d8c17f9c097c2af64c



Further Reading

The first three posts in the original topic A complete rewrite of the Linux scripts are still being updated, to point to the latest version:

Last edited by hbuhrmester on 05.04.2019, 10:02, edited 1 time in total.
Reason: added credits for cabextract workaround in Debian Buster
hbuhrmester
 
Posts: 361
Joined: 11.10.2013, 20:59

Return to Linux

Who is online

Users browsing this forum: No registered users and 2 guests