A complete rewrite of the Linux scripts

Re: A complete rewrite of the Linux scripts

Postby mani » 04.04.2017, 16:15

Dear Hartmut
First of all thanks very much for the great work you have done. I was running the beta2 version for around two weeks and now the beta3 for the second day without real problems.

I run a copy of your “get-all-updates.bash” as “get-my-updates.bash” with my changes as a cron job (which runs 30 to 35 minutes). My minor problem is that only every second day it will check my updates, because the time stamps difference is exactly 24 hours. I changed line 87 in the timestamps.bash to:
local -i time_interval="${2:-84000}" # use less than 24 hours as default to allow Crown runs every day (84000=23h 20 min)
I believe you are planning with “time_interval_input_files=86400” an easier possibility………

I check also the result of the crown job with logcheck as an email, with stripping all “common” lines. Due to sorting from logcheck, an easy possibility to see which command causes some abnormalities and/or which files are downloaded, I had to add the command line information to the 20-start-logging.bash (log_message "command: $command_line"), because your info-block has the same information, but without date and time and get sorted in a block at the end of the email.

This stripes over 8000 lines from the daily download.log down to:
System Events
=-=-=-=-=-=-=
***************
2017-04-04 02:01:09 - command: ./download-updates.bash w61 deu -includesp -includecpp -includedotnet -includewddefs -includemsse
2017-04-04 02:02:55 (639 KB/s) - '../client/wddefs/x86-glb/mpas-fe.exe' saved [44963096/44963096]
2017-04-04 02:06:51 (639 KB/s) - '../client/msse/x86-glb/mpam-fe.exe' saved [147751184/147751184]
2017-04-04 02:08:24 - command: ./download-updates.bash w61 enu -includesp -includecpp -includedotnet -includewddefs -includemsse
2017-04-04 02:09:28 - command: ./download-updates.bash w61-x64 deu -includesp -includecpp -includedotnet -includewddefs -includemsse
2017-04-04 02:10:47 (639 KB/s) - '../client/wddefs/x64-glb/mpas-feX64.exe' saved [45731088/45731088]
2017-04-04 02:14:39 (639 KB/s) - '../client/msse/x64-glb/mpam-fex64.exe' saved [148521232/148521232]
2017-04-04 02:17:28 - command: ./download-updates.bash w61-x64 enu -includesp -includecpp -includedotnet -includewddefs -includemsse
2017-04-04 02:20:00 - command: ./download-updates.bash w100 deu -includesp -includecpp -includedotnet -includewddefs8
2017-04-04 02:21:32 - command: ./download-updates.bash w100 enu -includesp -includecpp -includedotnet -includewddefs8
2017-04-04 02:21:40 - command: ./download-updates.bash w100-x64 deu -includesp -includecpp -includedotnet -includewddefs8
2017-04-04 02:25:11 - command: ./download-updates.bash w100-x64 enu -includesp -includecpp -includedotnet -includewddefs8
2017-04-04 02:25:20 - command: ./download-updates.bash o2k13 deu -includesp
2017-04-04 02:30:22 - command: ./download-updates.bash o2k13 enu –includesp

if somebody is interested in the rules-file, feel free to ask (it is still a fast and dirty version)

Mani
mani
 
Posts: 1
Joined: 04.04.2017, 15:50

Re: A complete rewrite of the Linux scripts

Postby TimmW » 12.04.2017, 14:17

Hi,
First of all much thanks for the great work that has been done. As a frequent but not regularly user of the WSUSoffline Tool I was curious to see that there is finally a new attempt for a linux version of the tool doing the downloading stuff, perfect. This looks like a welcome solution to maintain an up-to-date source of windows patches and service packs on a linux hosted file server with a lot of windows clients in our small office. So, I saw it today and directly thought to give it a try. I did the following on a Ubuntu virtual machine (ubuntu 14.04LTS 64bit) hosted on my Win7 pc:
1.
Code: Select all
sudo apt-get install cabextract hashdeep wget xmlstarlet trash-cli

This failed as hashdeep seems not availaible in the ubuntu repositories!! I stumbled over md5deep and executed
Code: Select all
sudo apt-get install cabextract md5deep wget xmlstarlet trash-cli

2.
Code: Select all
mkdir wsusoffline
and
Code: Select all
cd wsusoffline/

3. executed
Code: Select all
wget http://downloads.hartmut-buhrmester.de/sh-new-1.0-beta-3.tar.gz
and
Code: Select all
wget http://downloads.hartmut-buhrmester.de/hashes-sh-new-1.0-beta-3.txt

4. tried then to verify the downloads via
Code: Select all
md5deep -a -v -v -l -k hashes-sh-new-1.0-beta-3.txt sh-new-1.0-beta-3.tar.gz
which returned 4.2 !? Ignoring this and hoping/expecting no probs from the correctly downloaded file ...

5. extracted the archive with
Code: Select all
tar xvzf sh-new-1.0-beta-3.tar.gz

6. changed to the new directory
Code: Select all
cd sh-new-1.0-beta-3


Now comes the Problem: when executing
./update-generator.bash
it always exits with an error, here is the dump from the shell:
Code: Select all
grep: ../cmd/DownloadUpdates.cmd: No such file or directory

Info: Starting update-generator.bash 1.0-beta-3 (2017-03-30)
Info: Running on WSUS Offline Update version not-available

Info: Checking needed applications...
Info: Checking recommended applications...
Info: Found Linux trash handler: gvfs-trash

Info: Setting download options for GNU Wget...
Info: Wake up sleeping DSL modems and routers...
PING www.wsusoffline.net (81.3.27.18) 56(84) bytes of data.

--- www.wsusoffline.net ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3008ms
rtt min/avg/max/mdev = 29.128/35.424/46.641/6.784 ms

Info: Testing the internet connection...
Info: Connection test succeeded

Info: Searching for new versions of WSUS Offline Update...
cat: ../static/StaticDownloadLink-this.txt: No such file or directory
Failure: unhandled error 1
Backtrace: error_handler cat_dos compare_wsusoffline_versions source run_scripts update_generator main
Caller 0: 58 cat_dos ./libraries/dos-files.bash
Caller 1: 85 compare_wsusoffline_versions ./common-tasks/50-check-wsusoffline-version.bash
Caller 2: 331 source ./common-tasks/50-check-wsusoffline-version.bash
Caller 3: 314 run_scripts ./update-generator.bash
Caller 4: 334 update_generator ./update-generator.bash
Caller 5: 344 main ./update-generator.bash
Failure: unhandled error 1
Backtrace: error_handler compare_wsusoffline_versions source run_scripts update_generator main
Caller 0: 85 compare_wsusoffline_versions ./common-tasks/50-check-wsusoffline-version.bash
Caller 1: 331 source ./common-tasks/50-check-wsusoffline-version.bash
Caller 2: 314 run_scripts ./update-generator.bash
Caller 3: 334 update_generator ./update-generator.bash
Caller 4: 344 main ./update-generator.bash
Exiting with error code 1 ...


I tried already:
    copied the script 60-check-script-version.bash and moved it to the directory common-tasks
    edited preferences-template.bash and set check_for_self_updates="disabled"
None of the above changed anything in the behaviour or the error code produced, what am I missing ??

Your help and advice is highly appreciated, much thanks,
Timm
TimmW
 
Posts: 3
Joined: 12.04.2017, 13:42

Re: A complete rewrite of the Linux scripts

Postby psj » 12.04.2017, 14:51

In your description there is no mention of the files from the windows version of
WSUS Offline Update. Are you aware that the new linux scripts still require
the presence of the files from the windows version?

psj.
psj
 
Posts: 36
Joined: 11.06.2010, 17:58

Re: A complete rewrite of the Linux scripts

Postby TimmW » 12.04.2017, 15:30

Sorry for being so dumb,
I was not aware of the necessity of the complete original zip archive for the wiondows version. After redoing it (first get the zip archive in the current version, unpack it) performing teh steps as described above now works fine. Much thanks for your quick clue and alerting me to do the right thing. Now I am getting the "selection menu" and can procede in testing.

Thanks again and best regards,
Timm
TimmW
 
Posts: 3
Joined: 12.04.2017, 13:42

Re: A complete rewrite of the Linux scripts

Postby hbuhrmester » 12.04.2017, 20:13

The Linux download scripts still need the configuration files from the WSUS Offline Update installation. These are the files in the directories static, exclude, client/static, and client/exclude. The Linux scripts also need the XSLT transformation files from the directory xslt, to calculate superseded and dynamic updates.

The Windows script DownloadUpdates.cmd is checked once, to get the exact version of WSUS Offline Update. This is a simple grep and does not execute anything. It is implemented in the function set_wou_version as:

Code: Select all
function set_wou_version ()
{
    if  wou_version="$(grep_dos -F -- "set WSUSOFFLINE_VERSION=" ../cmd/DownloadUpdates.cmd)"; then
        wou_version="${wou_version/set WSUSOFFLINE_VERSION=/}"
    else
        wou_version="not-available"
    fi
    return 0
}

Also, the Linux download scripts can only replace the download part. To install anything, you definitely need the UpdateInstaller.exe and all the other files in the client subdirectory.

Actually, this is mentioned in the Quick Installation Guide. It says: "Download the archive and the hashes file to the directory wsusoffline. This is the directory, where the Windows utility UpdateGenerator.exe resides."

But maybe it should be pointed out more clearly, that the archive for WSUS Offline Update needs to be downloaded first.


Regarding md5deep/hashdeep

The package name md5deep was changed to hashdeep in Debian 8 Jessie-Backports https://packages.debian.org/jessie-backports/hashdeep . The installed binary /usr/bin/hashdeep has a change date of Aug 11 2015.

Ubuntu 14.04LTS does not have this package, because it was released in April 2014, before that switch.

A package search for md5deep shows, that md5deep is a real package only on Ubuntu Precise (12.04LTS) and Trusty (14.04LTS). In all newer versions it is a "transitional dummy package for hashdeep".

http://packages.ubuntu.com/search?keywo ... ection=all

Accordingly, the package hashdeep is available on Ubuntu Xenial (16.04LTS) and newer.

http://packages.ubuntu.com/search?keywo ... ection=all

So, the situation is quite similar to Debian, as could be expected. In recent versions of both Debian and Ubuntu, the package hashdeep should be used, with md5deep offered as a dummy package for an easy upgrade.


4. tried then to verify the downloads via

Code: Select all
md5deep -a -v -v -l -k hashes-sh-new-1.0-beta-3.txt sh-new-1.0-beta-3.tar.gz


which returned 4.2 !? Ignoring this and hoping/expecting no probs from the correctly downloaded file ...


At this point you should have tried hashdeep, which is also installed by the package md5deep. Actually, these binaries are all copies of the same file, but they behave differently, depending on how they are called. In Debian 7 Wheezy I got the following display:

Code: Select all
~$ ls -l /usr/bin/*deep
-rwxr-xr-x 1 root root 212204 Jun 14  2012 /usr/bin/hashdeep
-rwxr-xr-x 1 root root 212204 Jun 14  2012 /usr/bin/md5deep
-rwxr-xr-x 1 root root 212204 Jun 14  2012 /usr/bin/sha1deep
-rwxr-xr-x 1 root root 212204 Jun 14  2012 /usr/bin/sha256deep
-rwxr-xr-x 1 root root 212204 Jun 14  2012 /usr/bin/tigerdeep
-rwxr-xr-x 1 root root 212204 Jun 14  2012 /usr/bin/whirlpooldeep
~$ md5sum /usr/bin/*deep
b1029a5a5feb815134503e9db23f80f7  /usr/bin/hashdeep
b1029a5a5feb815134503e9db23f80f7  /usr/bin/md5deep
b1029a5a5feb815134503e9db23f80f7  /usr/bin/sha1deep
b1029a5a5feb815134503e9db23f80f7  /usr/bin/sha256deep
b1029a5a5feb815134503e9db23f80f7  /usr/bin/tigerdeep
b1029a5a5feb815134503e9db23f80f7  /usr/bin/whirlpooldeep


But, although the binaries look the same, they calculate different hashes and have different options.

In recent Debian versions, hashdeep is installed by the package hashdeep, and the other files are provided by symbolic links.

* debian/hashdeep.links:
- Added to enable multi-call for md5deep and others.


But this topic always seems to be confusing. I guess, I'll add some more hints to the Quick Installation Guide.
hbuhrmester
 
Posts: 199
Joined: 11.10.2013, 20:59

Re: A complete rewrite of the Linux scripts

Postby TimmW » 13.04.2017, 08:05

Dear hbuhrmester,

much thanks for your thorough clarification and all the effort you are putting in. I guess it might not be a bad idea to point out a bit more the fact that the WSUS Offline Update needs to be downloaded first. Though it is clearly writen as you highlighted to put the files to that directory - honestly I read this but I did not think too much about it and it did not ring a bell.

Keep on, best regards,
Timm
TimmW
 
Posts: 3
Joined: 12.04.2017, 13:42

Re: A complete rewrite of the Linux scripts

Postby hbuhrmester » 13.04.2017, 10:11

With the current versions of WSUS Offline Update (10.9.2) and the Linux scripts (1.0-beta-3), it is necessary to download and unpack the wsusoffline archive first. This will change with the next versions:

WSUSUpdateAdmin has already replaced the old Linux script DownloadUpdates.sh with the new Linux scripts in changeset 866, so they will be included in the next release of WSUS Offline Update:


The new Linux scripts regularly check for new versions of WSUS Offline Update, and can download and install them. The script 50-check-wsusoffline-version.bash just assumes, that there is a file ../static/StaticDownloadLink-this.txt, which indicates the currently installed version. With small changes around this part, the same script could also do an initial download of the wsusoffline archive.

It actually works with the version 1.0-beta-3, but you would need to create a dummy file ../static/StaticDownloadLink-this.txt first:
Code: Select all
mkdir -p wsusoffline/static
cd wsusoffline/
echo "unavailable" > static/StaticDownloadLink-this.txt


Then download the archive and hashes file for the Linux scripts and unpack them, as described in the Quick Installation Guide:
Code: Select all
wget http://downloads.hartmut-buhrmester.de/sh-new-1.0-beta-3.tar.gz
wget http://downloads.hartmut-buhrmester.de/hashes-sh-new-1.0-beta-3.txt
hashdeep -a -v -v -l -k hashes-sh-new-1.0-beta-3.txt sh-new-1.0-beta-3.tar.gz
tar xvzf sh-new-1.0-beta-3.tar.gz
cd sh-new-1.0-beta-3
./update-generator.bash


After some tests, the scripts will search for new versions of WSUS Offline Update. If none is installed, you will get the message:
Code: Select all
Info: Searching for new versions of WSUS Offline Update...
Info: A new version of WSUS Offline Update is available:
- Installed version: unavailable
- Available version: wsusoffline1092
Info: Do you want to install the new version now?
---------------------------------------------------------------------------
Note: This question automatically selects "No" after 30 seconds, to skip
the pending self-update and let the script continue. This is also the
default answer, if you simply hit return.
---------------------------------------------------------------------------
[y|N]: y


Just type "y" to confirm the update and it will download and install the wsusoffline archive itself.
hbuhrmester
 
Posts: 199
Joined: 11.10.2013, 20:59

Re: A complete rewrite of the Linux scripts

Postby n8marti » 24.04.2017, 13:27

This beta script is working well for me. However, I'm wondering if there is a way to pass multiple options to the download-updates script? I read in the Manual that multiple languages need to be downloaded in turn, but what about updates for multiple versions of Windows? I am supporting a team with very poor internet and 6 different versions of Windows: 32-bit & 64-bit versions of Win 7, 8, and 10. Unfortunately, I have no control over what OSes they run. I would like to be able to choose multiple versions of Windows simultaneously from the update-generator script, or at least find out if I can pass multiple version options to the command line. Thanks.
n8marti
 
Posts: 1
Joined: 24.04.2017, 13:16

Re: A complete rewrite of the Linux scripts

Postby hbuhrmester » 25.04.2017, 09:33

n8marti wrote:I would like to be able to choose multiple versions of Windows simultaneously from the update-generator script, or at least find out if I can pass multiple version options to the command line. Thanks.


Such multiple options are not supported, at least for now. I wanted to keep these scripts simple, and I didn't want to make everything different than the Windows version. The Windows script DownloadUpdates.cmd also handles only one Windows version per run.

If you have different Windows versions to support, you could compile a small script of your own to run these downloads. The included script get-all-updates.bash may serve as a template. I use this script to compare the results on Windows and Linux with a standard download set, but it is also meant for customization.

This does not really affect the performance over slow Internet connections: There are some common tasks for all Windows versions, like wsus, win, and the optional downloads cpp, dotnet, msse, and wdefs. But these tasks are evaluated only once each day. The new Linux script uses timestamps, to keep track of these tasks: If one task has already been done in the past 24 hours, the complete processing will be skipped.

If one task could not be completed successfully, its timestamp will not be updated, and then it is rerun immediately. This could happen with the virus definition files sometimes, although I made the download of these files more robust.

I might someday implement a list for the languages, though, because this could prevent some unneeded processing. But then there should also be a user interface to select multiple languages. I once created a mockup for the old Linux script to show, how this could look like with dialog: viewtopic.php?f=9&t=4061

The old Linux script DownloadUpdates.sh had some integrated lists for all Windows versions or all Office versions, but it was poorly implemented: The script DownloadUpdates.sh would recursively call itself for every update. This didn't achieve anything, as it didn't make the script run faster. But it broke repeatedly, because the script sometimes failed to find its own installation directory, for example:
viewtopic.php?f=9&t=4469 , viewtopic.php?f=9&t=5298 , viewtopic.php?f=9&t=5314 , viewtopic.php?f=9&t=5346
hbuhrmester
 
Posts: 199
Joined: 11.10.2013, 20:59

Previous

Return to Linux

Who is online

Users browsing this forum: No registered users and 1 guest