A complete rewrite of the Linux scripts

Re: A complete rewrite of the Linux scripts

Postby mani » 04.04.2017, 16:15

Dear Hartmut
First of all thanks very much for the great work you have done. I was running the beta2 version for around two weeks and now the beta3 for the second day without real problems.

I run a copy of your “get-all-updates.bash” as “get-my-updates.bash” with my changes as a cron job (which runs 30 to 35 minutes). My minor problem is that only every second day it will check my updates, because the time stamps difference is exactly 24 hours. I changed line 87 in the timestamps.bash to:
local -i time_interval="${2:-84000}" # use less than 24 hours as default to allow Crown runs every day (84000=23h 20 min)
I believe you are planning with “time_interval_input_files=86400” an easier possibility………

I check also the result of the crown job with logcheck as an email, with stripping all “common” lines. Due to sorting from logcheck, an easy possibility to see which command causes some abnormalities and/or which files are downloaded, I had to add the command line information to the 20-start-logging.bash (log_message "command: $command_line"), because your info-block has the same information, but without date and time and get sorted in a block at the end of the email.

This stripes over 8000 lines from the daily download.log down to:
System Events
2017-04-04 02:01:09 - command: ./download-updates.bash w61 deu -includesp -includecpp -includedotnet -includewddefs -includemsse
2017-04-04 02:02:55 (639 KB/s) - '../client/wddefs/x86-glb/mpas-fe.exe' saved [44963096/44963096]
2017-04-04 02:06:51 (639 KB/s) - '../client/msse/x86-glb/mpam-fe.exe' saved [147751184/147751184]
2017-04-04 02:08:24 - command: ./download-updates.bash w61 enu -includesp -includecpp -includedotnet -includewddefs -includemsse
2017-04-04 02:09:28 - command: ./download-updates.bash w61-x64 deu -includesp -includecpp -includedotnet -includewddefs -includemsse
2017-04-04 02:10:47 (639 KB/s) - '../client/wddefs/x64-glb/mpas-feX64.exe' saved [45731088/45731088]
2017-04-04 02:14:39 (639 KB/s) - '../client/msse/x64-glb/mpam-fex64.exe' saved [148521232/148521232]
2017-04-04 02:17:28 - command: ./download-updates.bash w61-x64 enu -includesp -includecpp -includedotnet -includewddefs -includemsse
2017-04-04 02:20:00 - command: ./download-updates.bash w100 deu -includesp -includecpp -includedotnet -includewddefs8
2017-04-04 02:21:32 - command: ./download-updates.bash w100 enu -includesp -includecpp -includedotnet -includewddefs8
2017-04-04 02:21:40 - command: ./download-updates.bash w100-x64 deu -includesp -includecpp -includedotnet -includewddefs8
2017-04-04 02:25:11 - command: ./download-updates.bash w100-x64 enu -includesp -includecpp -includedotnet -includewddefs8
2017-04-04 02:25:20 - command: ./download-updates.bash o2k13 deu -includesp
2017-04-04 02:30:22 - command: ./download-updates.bash o2k13 enu –includesp

if somebody is interested in the rules-file, feel free to ask (it is still a fast and dirty version)

Posts: 1
Joined: 04.04.2017, 15:50

Re: A complete rewrite of the Linux scripts

Postby TimmW » 12.04.2017, 14:17

First of all much thanks for the great work that has been done. As a frequent but not regularly user of the WSUSoffline Tool I was curious to see that there is finally a new attempt for a linux version of the tool doing the downloading stuff, perfect. This looks like a welcome solution to maintain an up-to-date source of windows patches and service packs on a linux hosted file server with a lot of windows clients in our small office. So, I saw it today and directly thought to give it a try. I did the following on a Ubuntu virtual machine (ubuntu 14.04LTS 64bit) hosted on my Win7 pc:
Code: Select all
sudo apt-get install cabextract hashdeep wget xmlstarlet trash-cli

This failed as hashdeep seems not availaible in the ubuntu repositories!! I stumbled over md5deep and executed
Code: Select all
sudo apt-get install cabextract md5deep wget xmlstarlet trash-cli

Code: Select all
mkdir wsusoffline
Code: Select all
cd wsusoffline/

3. executed
Code: Select all
wget http://downloads.hartmut-buhrmester.de/sh-new-1.0-beta-3.tar.gz
Code: Select all
wget http://downloads.hartmut-buhrmester.de/hashes-sh-new-1.0-beta-3.txt

4. tried then to verify the downloads via
Code: Select all
md5deep -a -v -v -l -k hashes-sh-new-1.0-beta-3.txt sh-new-1.0-beta-3.tar.gz
which returned 4.2 !? Ignoring this and hoping/expecting no probs from the correctly downloaded file ...

5. extracted the archive with
Code: Select all
tar xvzf sh-new-1.0-beta-3.tar.gz

6. changed to the new directory
Code: Select all
cd sh-new-1.0-beta-3

Now comes the Problem: when executing
it always exits with an error, here is the dump from the shell:
Code: Select all
grep: ../cmd/DownloadUpdates.cmd: No such file or directory

Info: Starting update-generator.bash 1.0-beta-3 (2017-03-30)
Info: Running on WSUS Offline Update version not-available

Info: Checking needed applications...
Info: Checking recommended applications...
Info: Found Linux trash handler: gvfs-trash

Info: Setting download options for GNU Wget...
Info: Wake up sleeping DSL modems and routers...
PING www.wsusoffline.net ( 56(84) bytes of data.

--- www.wsusoffline.net ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3008ms
rtt min/avg/max/mdev = 29.128/35.424/46.641/6.784 ms

Info: Testing the internet connection...
Info: Connection test succeeded

Info: Searching for new versions of WSUS Offline Update...
cat: ../static/StaticDownloadLink-this.txt: No such file or directory
Failure: unhandled error 1
Backtrace: error_handler cat_dos compare_wsusoffline_versions source run_scripts update_generator main
Caller 0: 58 cat_dos ./libraries/dos-files.bash
Caller 1: 85 compare_wsusoffline_versions ./common-tasks/50-check-wsusoffline-version.bash
Caller 2: 331 source ./common-tasks/50-check-wsusoffline-version.bash
Caller 3: 314 run_scripts ./update-generator.bash
Caller 4: 334 update_generator ./update-generator.bash
Caller 5: 344 main ./update-generator.bash
Failure: unhandled error 1
Backtrace: error_handler compare_wsusoffline_versions source run_scripts update_generator main
Caller 0: 85 compare_wsusoffline_versions ./common-tasks/50-check-wsusoffline-version.bash
Caller 1: 331 source ./common-tasks/50-check-wsusoffline-version.bash
Caller 2: 314 run_scripts ./update-generator.bash
Caller 3: 334 update_generator ./update-generator.bash
Caller 4: 344 main ./update-generator.bash
Exiting with error code 1 ...

I tried already:
    copied the script 60-check-script-version.bash and moved it to the directory common-tasks
    edited preferences-template.bash and set check_for_self_updates="disabled"
None of the above changed anything in the behaviour or the error code produced, what am I missing ??

Your help and advice is highly appreciated, much thanks,
Posts: 3
Joined: 12.04.2017, 13:42

Re: A complete rewrite of the Linux scripts

Postby psj » 12.04.2017, 14:51

In your description there is no mention of the files from the windows version of
WSUS Offline Update. Are you aware that the new linux scripts still require
the presence of the files from the windows version?

Posts: 36
Joined: 11.06.2010, 17:58

Re: A complete rewrite of the Linux scripts

Postby TimmW » 12.04.2017, 15:30

Sorry for being so dumb,
I was not aware of the necessity of the complete original zip archive for the wiondows version. After redoing it (first get the zip archive in the current version, unpack it) performing teh steps as described above now works fine. Much thanks for your quick clue and alerting me to do the right thing. Now I am getting the "selection menu" and can procede in testing.

Thanks again and best regards,
Posts: 3
Joined: 12.04.2017, 13:42

Re: A complete rewrite of the Linux scripts

Postby hbuhrmester » 12.04.2017, 20:13

The Linux download scripts still need the configuration files from the WSUS Offline Update installation. These are the files in the directories static, exclude, client/static, and client/exclude. The Linux scripts also need the XSLT transformation files from the directory xslt, to calculate superseded and dynamic updates.

The Windows script DownloadUpdates.cmd is checked once, to get the exact version of WSUS Offline Update. This is a simple grep and does not execute anything. It is implemented in the function set_wou_version as:

Code: Select all
function set_wou_version ()
    if  wou_version="$(grep_dos -F -- "set WSUSOFFLINE_VERSION=" ../cmd/DownloadUpdates.cmd)"; then
        wou_version="${wou_version/set WSUSOFFLINE_VERSION=/}"
    return 0

Also, the Linux download scripts can only replace the download part. To install anything, you definitely need the UpdateInstaller.exe and all the other files in the client subdirectory.

Actually, this is mentioned in the Quick Installation Guide. It says: "Download the archive and the hashes file to the directory wsusoffline. This is the directory, where the Windows utility UpdateGenerator.exe resides."

But maybe it should be pointed out more clearly, that the archive for WSUS Offline Update needs to be downloaded first.

Regarding md5deep/hashdeep

The package name md5deep was changed to hashdeep in Debian 8 Jessie-Backports https://packages.debian.org/jessie-backports/hashdeep . The installed binary /usr/bin/hashdeep has a change date of Aug 11 2015.

Ubuntu 14.04LTS does not have this package, because it was released in April 2014, before that switch.

A package search for md5deep shows, that md5deep is a real package only on Ubuntu Precise (12.04LTS) and Trusty (14.04LTS). In all newer versions it is a "transitional dummy package for hashdeep".

http://packages.ubuntu.com/search?keywo ... ection=all

Accordingly, the package hashdeep is available on Ubuntu Xenial (16.04LTS) and newer.

http://packages.ubuntu.com/search?keywo ... ection=all

So, the situation is quite similar to Debian, as could be expected. In recent versions of both Debian and Ubuntu, the package hashdeep should be used, with md5deep offered as a dummy package for an easy upgrade.

4. tried then to verify the downloads via

Code: Select all
md5deep -a -v -v -l -k hashes-sh-new-1.0-beta-3.txt sh-new-1.0-beta-3.tar.gz

which returned 4.2 !? Ignoring this and hoping/expecting no probs from the correctly downloaded file ...

At this point you should have tried hashdeep, which is also installed by the package md5deep. Actually, these binaries are all copies of the same file, but they behave differently, depending on how they are called. In Debian 7 Wheezy I got the following display:

Code: Select all
~$ ls -l /usr/bin/*deep
-rwxr-xr-x 1 root root 212204 Jun 14  2012 /usr/bin/hashdeep
-rwxr-xr-x 1 root root 212204 Jun 14  2012 /usr/bin/md5deep
-rwxr-xr-x 1 root root 212204 Jun 14  2012 /usr/bin/sha1deep
-rwxr-xr-x 1 root root 212204 Jun 14  2012 /usr/bin/sha256deep
-rwxr-xr-x 1 root root 212204 Jun 14  2012 /usr/bin/tigerdeep
-rwxr-xr-x 1 root root 212204 Jun 14  2012 /usr/bin/whirlpooldeep
~$ md5sum /usr/bin/*deep
b1029a5a5feb815134503e9db23f80f7  /usr/bin/hashdeep
b1029a5a5feb815134503e9db23f80f7  /usr/bin/md5deep
b1029a5a5feb815134503e9db23f80f7  /usr/bin/sha1deep
b1029a5a5feb815134503e9db23f80f7  /usr/bin/sha256deep
b1029a5a5feb815134503e9db23f80f7  /usr/bin/tigerdeep
b1029a5a5feb815134503e9db23f80f7  /usr/bin/whirlpooldeep

But, although the binaries look the same, they calculate different hashes and have different options.

In recent Debian versions, hashdeep is installed by the package hashdeep, and the other files are provided by symbolic links.

* debian/hashdeep.links:
- Added to enable multi-call for md5deep and others.

But this topic always seems to be confusing. I guess, I'll add some more hints to the Quick Installation Guide.
Posts: 210
Joined: 11.10.2013, 20:59

Re: A complete rewrite of the Linux scripts

Postby TimmW » 13.04.2017, 08:05

Dear hbuhrmester,

much thanks for your thorough clarification and all the effort you are putting in. I guess it might not be a bad idea to point out a bit more the fact that the WSUS Offline Update needs to be downloaded first. Though it is clearly writen as you highlighted to put the files to that directory - honestly I read this but I did not think too much about it and it did not ring a bell.

Keep on, best regards,
Posts: 3
Joined: 12.04.2017, 13:42

Re: A complete rewrite of the Linux scripts

Postby hbuhrmester » 13.04.2017, 10:11

With the current versions of WSUS Offline Update (10.9.2) and the Linux scripts (1.0-beta-3), it is necessary to download and unpack the wsusoffline archive first. This will change with the next versions:

WSUSUpdateAdmin has already replaced the old Linux script DownloadUpdates.sh with the new Linux scripts in changeset 866, so they will be included in the next release of WSUS Offline Update:

The new Linux scripts regularly check for new versions of WSUS Offline Update, and can download and install them. The script 50-check-wsusoffline-version.bash just assumes, that there is a file ../static/StaticDownloadLink-this.txt, which indicates the currently installed version. With small changes around this part, the same script could also do an initial download of the wsusoffline archive.

It actually works with the version 1.0-beta-3, but you would need to create a dummy file ../static/StaticDownloadLink-this.txt first:
Code: Select all
mkdir -p wsusoffline/static
cd wsusoffline/
echo "unavailable" > static/StaticDownloadLink-this.txt

Then download the archive and hashes file for the Linux scripts and unpack them, as described in the Quick Installation Guide:
Code: Select all
wget http://downloads.hartmut-buhrmester.de/sh-new-1.0-beta-3.tar.gz
wget http://downloads.hartmut-buhrmester.de/hashes-sh-new-1.0-beta-3.txt
hashdeep -a -v -v -l -k hashes-sh-new-1.0-beta-3.txt sh-new-1.0-beta-3.tar.gz
tar xvzf sh-new-1.0-beta-3.tar.gz
cd sh-new-1.0-beta-3

After some tests, the scripts will search for new versions of WSUS Offline Update. If none is installed, you will get the message:
Code: Select all
Info: Searching for new versions of WSUS Offline Update...
Info: A new version of WSUS Offline Update is available:
- Installed version: unavailable
- Available version: wsusoffline1092
Info: Do you want to install the new version now?
Note: This question automatically selects "No" after 30 seconds, to skip
the pending self-update and let the script continue. This is also the
default answer, if you simply hit return.
[y|N]: y

Just type "y" to confirm the update and it will download and install the wsusoffline archive itself.
Posts: 210
Joined: 11.10.2013, 20:59

Re: A complete rewrite of the Linux scripts

Postby n8marti » 24.04.2017, 13:27

This beta script is working well for me. However, I'm wondering if there is a way to pass multiple options to the download-updates script? I read in the Manual that multiple languages need to be downloaded in turn, but what about updates for multiple versions of Windows? I am supporting a team with very poor internet and 6 different versions of Windows: 32-bit & 64-bit versions of Win 7, 8, and 10. Unfortunately, I have no control over what OSes they run. I would like to be able to choose multiple versions of Windows simultaneously from the update-generator script, or at least find out if I can pass multiple version options to the command line. Thanks.
Posts: 1
Joined: 24.04.2017, 13:16

Re: A complete rewrite of the Linux scripts

Postby hbuhrmester » 25.04.2017, 09:33

n8marti wrote:I would like to be able to choose multiple versions of Windows simultaneously from the update-generator script, or at least find out if I can pass multiple version options to the command line. Thanks.

Such multiple options are not supported, at least for now. I wanted to keep these scripts simple, and I didn't want to make everything different than the Windows version. The Windows script DownloadUpdates.cmd also handles only one Windows version per run.

If you have different Windows versions to support, you could compile a small script of your own to run these downloads. The included script get-all-updates.bash may serve as a template. I use this script to compare the results on Windows and Linux with a standard download set, but it is also meant for customization.

This does not really affect the performance over slow Internet connections: There are some common tasks for all Windows versions, like wsus, win, and the optional downloads cpp, dotnet, msse, and wdefs. But these tasks are evaluated only once each day. The new Linux script uses timestamps, to keep track of these tasks: If one task has already been done in the past 24 hours, the complete processing will be skipped.

If one task could not be completed successfully, its timestamp will not be updated, and then it is rerun immediately. This could happen with the virus definition files sometimes, although I made the download of these files more robust.

I might someday implement a list for the languages, though, because this could prevent some unneeded processing. But then there should also be a user interface to select multiple languages. I once created a mockup for the old Linux script to show, how this could look like with dialog: viewtopic.php?f=9&t=4061

The old Linux script DownloadUpdates.sh had some integrated lists for all Windows versions or all Office versions, but it was poorly implemented: The script DownloadUpdates.sh would recursively call itself for every update. This didn't achieve anything, as it didn't make the script run faster. But it broke repeatedly, because the script sometimes failed to find its own installation directory, for example:
viewtopic.php?f=9&t=4469 , viewtopic.php?f=9&t=5298 , viewtopic.php?f=9&t=5314 , viewtopic.php?f=9&t=5346
Posts: 210
Joined: 11.10.2013, 20:59

Release Notes for Version 1.0-beta-4

Postby hbuhrmester » 24.06.2017, 15:13

Release Notes for Version 1.0-beta-4

Release date: 2017-06-23
Intended compatibility: WSUS Offline Update version 10.9.2 and newer

This version offers the following changes:

  • The obsolete script DownloadUpdates.sh and related files will be deleted, if they are found in the same directory as the new Linux scripts

    The new Linux scripts are included in the WSUS Offline Update archive since changeset 866 http://trac.wsusoffline.net/changeset/866 . They are installed into the directory wsusoffline/sh. But then the old scripts in the same directory, if present, should be removed.

  • The Linux scripts can do an initial installation of WSUS Offline Update

    The Linux scripts need the configuration files of a WSUS Offline Update installation, to calculate static and dynamic downloads. Also, the new Linux download scripts can only replace the download part of WSUS Offline Update, namely the application UpdateGenerator.exe and the script DownloadUpdates.cmd. The client directory with the application UpdateInstaller.exe will be needed for the installation of the updates on Windows.

    The Linux scripts already included a script to search for new versions of WSUS Offline Update and to download and install them on demand. The same script can now do an initial installation of the wsusoffline archive, if none is installed.

    To use this approach, you should create an enclosing directory "wsusoffline" first, which will contain both the new Linux scripts and the contents of the wsusoffline archive. Change to the directory "wsusoffline" and download and unpack the Linux scripts to this directory. Change to the new directory "wsusoffline/sh-new-1.0-beta-4" and run the script update-generator.bash. It will download and unpack the wsusoffline archive outside of the directory sh-new-1.0-beta-4, but inside the enclosing directory wsusoffline.

    This may not be the usual way to unpack an archive, but the Linux scripts need to be in the wsusoffline directory, not the other way round. Don't complain, if you litter your home directory with lots of new files and directories.

    The next release of WSUS Offline Update will already include the new Linux download scripts, and then this approach will not be necessary anymore, but it was a small addition to an existing file, and so it was implemented anyway.

  • The integrity of the WSUS catalog file wsusscn2.cab is tested with cabextract

    The test with cabextract -t ensures, that all files in the archive can be extracted. This should detect damaged archives, which are sometimes found in the Microsoft content delivery network after the official patch day on the second Tuesday each month. These files can be successfully downloaded, without any error indicated by wget. But the verification of the digital file signature with Sysinternals Sigcheck may still fail. Usually, these problems disappear by itself after a few days.

    Testing a damaged file with cabextract typically finds errors at the end of the file. An example result is shown below. The warning about "extra bytes at end of file" can be ignored, though.

    Code: Select all
    wsusscn2.cab: WARNING; possible 16168 extra bytes at end of file.
    Testing cabinet: wsusscn2.cab
      index.xml  OK                                2508e14227888917d2769db31ed8e1c6
      package.cab  OK                              0f7dd82ec3d6b0761cb29eb2a00675a5
      package2.cab  OK                             6698aec96565b40e4fe1b82fbb4aa2e9
      package3.cab  OK                             0eef4f5fcf236fa539da5618b1ee8325
      package4.cab  OK                             40d90c96a53883ba404399442b33f0c4
      package5.cab  OK                             740689ddd85cdd0b1183b85f5794e11f
      package6.cab  OK                             b3ca8da969ee2f96fbe3c722f41c2749
      package7.cab  OK                             ba1bbfd1d238abe3c423a8677ad3f7dd
      package8.cab  OK                             fe93c09cdc220c6b7d864007084ae930
      package9.cab  OK                             721490a2f936d2a8ca83e11c2a08a216
      package10.cab  OK                            e54853f76de0b866d0413e0243e847fd
      package11.cab  OK                            bafb3946d3d26f97ec02157ead53ff63
      package12.cab  OK                            745ec78ad5ea69737224de451679f072
      package13.cab  OK                            df367ee491b7ec9079e167123ac0d4a4
      package14.cab  OK                            bf882964ed4557b8ac1546596446693f
      package15.cab  OK                            053a043b17277621a8ad423b3cef3e28
      package16.cab  OK                            d17019aa3a2d8bf8b0cde407320b3b5d
      package17.cab  OK                            a5e9e40f3606c21b1c3c01d3bcd74cea
      package18.cab  OK                            606375e2fd92132b0b2e335075a561fd
      package19.cab  OK                            40d20af34d0e6a3751cd572ae3296bae
      package20.cab  OK                            73b533405d21623809a94687ff82f0f3
      package21.cab  OK                            bc8e73d331e4066e5a23449244839339
      package22.cab  OK                            7b2b15c6171fe7784b606590ce0eb97c
      package23.cab  OK                            8210ff967d7f721fd5bbb03b5b056ad0
      package24.cab  OK                            43d464867ba4a6a2e63964992bb1822d
      package25.cab  OK                            9adf7bc2ccd5c39241192fa87eac058b
      package26.cab  OK                            de0d2b053a7adb53d56e7a1bea2b454b
      package27.cab  OK                            1963f03520d33eca52fa496dfb328c5f
      package28.cab  OK                            b5a3cd6436728a47d8049f23d904d174
      package29.cab  OK                            ce06c65e46a02cb54d0dfb07abb562b6
      package30.cab  OK                            f2f1f8cd8f9e9e47f78b2566fdb4785c
      package31.cab  OK                            eafe03f304ae9a850b2986d703448a49
      package32.cab  OK                            5b2b6c1bdfc9f192d33c67555014d176
      package33.cab  OK                            0865cd229c9092b83390ee4b80cfd3da
      package34.cab  OK                            b7788e1215d281cc069670f5d45985cc
      package35.cab  OK                            b82fe757de95aced8e41ca79db08f460
      package36.cab  OK                            70928874dec242b8daf821f4bab7d56d
      package37.cab  OK                            7e63ed0ebc979b35fcce7ef3566ba36f
      package38.cab  OK                            3065ec8b2f3678b9accb7ef26694bd9a
      package39.cab  OK                            8531a95049406c30f310e49aad8d42ba
      package40.cab  OK                            7ebb921d75c50e6f4f826741e8126a09
      package41.cab  OK                            de7261f8d70c6d4d516eeb4dd9eacc61
      package42.cab  OK                            8a0ec2fe99573402935d457de4d873f2
      package43.cab  OK                            29547f67da39b8b9f9bb8eac2e89e6d3
      package44.cab  OK                            dfaeb5d16a31fdd72f83089d30e7fd80
      package45.cab  OK                            ace35ffcaafe25059d76dc10fcc59315
      package46.cab  OK                            cd5dad0f47c4bcc68d6a05bb480f9709
      package47.cab  OK                            fa16cf9109ecec04d0e7db825fef2573
      package48.cab  OK                            c6e1595c979c8c21f6c1dd2a2b5ce389
      package49.cab  failed (checksum error)
      package50.cab  failed (checksum error)
      package51.cab  failed (checksum error)
      package52.cab  failed (checksum error)
      package53.cab  failed (checksum error)
      package54.cab  failed (checksum error)
      package55.cab  failed (checksum error)
      package56.cab  failed (checksum error)
      package57.cab  failed (checksum error)
      package58.cab  failed (checksum error)
      package59.cab  failed (checksum error)
      package60.cab  failed (checksum error)
      package61.cab  failed (checksum error)
      package62.cab  failed (checksum error)
      package63.cab  failed (checksum error)

    All done, errors in processing 15 file(s)

    The download part of WSUS Offline Update only uses the second file in this archive, package.cab, and this file seems to be okay. This could explain, why even a damaged archive wsusscn2.cab seems to work fine for the download part of WSUS Offline Update: The file package.cab (and then package.xml) can still be extracted and used for the calculation of superseded and dynamic downloads. But the installation will probably fail, if the file wsusscn2.cab is damaged.

    Security updates can be verified by comparing the SHA-1 hashes, which are embedded into the filenames, with the values, which are calculated by hashdeep.

  • The download script caches the file package.xml

    This was once suggested by "Schrabs" in viewtopic.php?f=5&t=5290&start=10#p17632 .

    The file package.xml is only extracted from the cabinet file wsusscn2.cab, if this file changes. Otherwise, a cached copy of the file package.xml in the new directory wsusoffline/cache will be used.

    The file package.xml contains just one long line without any line breaks. This is the most compact form of XML files and similar formats like JSON. In this form, it can be parsed by applications, but it cannot be displayed in a text editor nor searched with grep. For convenience, the script also creates a pretty-printed copy of the file with the name package-formated.xml. This is mostly for development, though.

  • The same_day function uses three different time intervals for different tasks

    The same_day function is used to prevent a repeated evaluation of the same tasks in adjacent runs of the download script. For example, the directory client/win contains common downloads for all Windows versions. Similarly, the directory client/ofc contains common files for all Office versions. If different Windows or Office versions are downloaded in turn, then the downloads for win and ofc should only be processed once.

    The same_day function uses timestamp files, to keep track of the downloads, which have already been done. The file modification date of the timestamp file is then compared to the current date.

    The first implementation of the same_day function used the output of the command "date" in the form "2017-05-09", which is known as ISO 8601 ( https://xkcd.com/1179/ ). If the file modification date of the timestamp file and the current date were on the same day, then the same_day function returned "true" and the download task would be skipped.

    The next implementation in version 1.0-beta-2 actually calculated the time difference in seconds. The same_day function returned true, if the difference between the file modification date and the current date was less than 24 hours.

    This approach has now been improved: the same_day function uses three different time intervals for different tasks:

    1. The four virus definition files change every two hours. It may be useful, to check these files more often, for example every four hours.

    2. Configuration files are checked once daily as before. This includes new versions of WSUS Offline Update and the Linux scripts, updates of the configuration files for WSUS Offline Update, and the WSUS catalog files wsusscn2.cab.

      If these configuration files change, then most or all of the remaining updates will be rescheduled by deleting their timestamp files.

    3. The remaining updates all depend on the configuration files. If the configuration files don't change, then these updates cannot change either. These are the updates for Windows, Office, .Net frameworks, and Visual C++ runtime libraries.

      The time interval for these dependent updates is set to a safe value of two days.

      Actually, these updates could be postponed forever. They will be rescheduled immediately, if one of the configuration files changes. This would result in an event-driven evaluation of the updates, rather then recalculating everything everyday.

    As "mani" pointed out in viewtopic.php?f=9&t=6180&start=10#p22502 , the time interval for "one day" should not be exactly 24 hours, but slightly less.

    The interval length must take into account the time needed to process the task: check the consistency of existing downloads, calculate static and dynamic download links, fetch all files (if they don't exist yet), and calculate new hashes for the download directory. The timestamp will be updated after successfully completing the task.

    An initial download, for example of the Windows 10 updates, may take several hours, depending on the speed of the Internet connection. Therefore, "four hours" are calculated as 3:20 hours, "one day" is now 21 hours, and "two days" are 45 hours. Of course, these are only guesses, which can be adjusted as needed in the file timestamps.bash.

  • Languages can be linked to a comma-separated list on the command line

    One goal in the development of the Linux scripts was to use unified language settings: The second parameter should always be a real language like "deu" or "enu", and then this language would be used, wherever a language selection is needed. This worked well for a single language, but if several languages were needed, they had to be downloaded in turn.

    In this version, language parameters can be joined to a comma-separated list. Instead of:

    Code: Select all
    ./download-updates.bash w60 deu
    ./download-updates.bash w60 enu

    you can simply write:

    Code: Select all
    ./download-updates.bash w60 deu,enu

    This makes the processing of Windows Vista, Windows 7, .NET Frameworks and Microsoft Security Essentials much faster, because it avoids unneeded iterations of these tasks.

    The downloads for Microsoft Office are still calculated separately for each language, and they won't gain the same acceleration.

    The general approach to language settings stays the same: there is no distinction between default languages, custom languages and update languages. Only languages, which are given on the command-line, will be included.

    Unfortunately, the setup script update-generator.bash doesn't support multiple selections yet. It uses the built-in command "select" of the bash to create simple dialogs, but these dialogs don't allow multiple selections. This may be implemented later with "dialog", as once suggested for the old Linux scripts ( viewtopic.php?f=9&t=4061 ).

    See the example script get-all-updates.bash for more examples. It is meant as a template for customization.

  • Windows Vista was removed from selection dialogs

    Windows Vista is no longer supported by Microsoft, and so the support for Vista was removed in WSUS Offline Update changeset 869 ( http://trac.wsusoffline.net/changeset/869 ).

    Windows Server 2008 is still supported, in both 32-bit and 64-bit versions.

    Therefore, the download part of WSUS Offline Update didn't really change much: The selections for w60 and w60-x64 are still there, but they now refer to Windows Server 2008 only.

    Most changes will be in the installation part of WSUS Offline Update: The application UpdateInstaller.exe distinguishes between all supported Windows versions, and it won't probably support Vista any longer.

    If you like to get the latest available updates for Windows Vista, you can combine this version of the Linux scripts with WSUS Offline Update 10.9.2 and just select "Windows Server 2008" from the selection dialog. Then you should save the wsusoffline installation, or at least the client directory, to a tar archive, to preserve the current state.

    For a short time, the available updates for Windows Vista and Server 2008 may stay the same. Windows Server 2008 will continue to get new updates, but these may not be installable on Vista anymore. New updates may still replace (supersede) old updates, and then some updates, which could be installed on Windows Vista, might be missing.

    This would be a similar situation as with Windows XP: The embedded version of Windows XP still gets updates, but they can't be installed in a regular Windows XP Home Edition or Windows XP Professional. Still, these updates for Windows XP embedded replace the updates for the regular versions, so that these updates are now missing. They are still available on the server, but WSUS Offline Update with its calculation of superseded updates may not download them anymore.

  • unzip is copied to the client directory

    unzip.exe is copied from the directory ../bin/ to ../client/bin/, because it is needed for the installation of two updates for Windows 7. This was pointed out by "wuttztfz" in viewtopic.php?f=9&t=6627 .

  • Added support for .Net Framework 4.7

    This change was introduced in changeset 872 ( http://trac.wsusoffline.net/changeset/872 ). It will appear in the next regular release of WSUS Offline Update.

Installation Guides updated

Please see the updated installation guides in English ( viewtopic.php?f=9&t=6180#p21449 ) and German ( viewtopic.php?f=9&t=6180#p21450 ).
Posts: 210
Joined: 11.10.2013, 20:59


Return to Linux

Who is online

Users browsing this forum: No registered users and 3 guests