NEWS for Linux download scripts, version 2.4 beta-1

NEWS for Linux download scripts, version 2.4 beta-1

Postby hbuhrmester » 11.05.2021, 21:44

NEWS for the Linux download scripts, version 2.4 (beta-1)

Release date: 2021-05-11
Development branch: master
Supported version: WSUS Offline Update, Community Edition 12.5 (b66)

This is a first beta version of the Linux download scripts, version 2.4, to keep up with the changes in WSUS Offline Update, Community Edition 12.5 (b66).

This version changes the way, how superseded updates and dynamic updates for both Windows and Office are calculated: All dynamic updates are now extracted by the ProductId, separately for each Windows and Office version. Therefore, the download directory ../client/ofc is no longer used and may be deleted safely. The list of superseded updates is more complete, so that fewer updates for Windows Server 2012 and Windows 8.1 / Server 2012 R2 are downloaded.


Bugfixes

The Security-only Safety Guard did not work with GitLab

If security-only downloads are selected, then the full update rollups must be excluded from both download and installation. This requires a manual configuration, because incremental security-only updates and cumulative update rollups cannot be distinguished automatically.

The function seconly_safety_guard tries to make sure, that several configuration files in the directories client/static and client/exclude have been updated after the last official patch day; otherwise the download will be postponed.

Previously, the file modification date of these files would be compared to the current date. This worked well with Trac/svn. It does not work anymore with GitLab, because GitLab does not set the Last-Modified header for files, which are extracted from the version control system. Therefore, newly downloaded files will always have the file modification date set to the current date and time.

The Security-only Safety Guard now searches literally for the expected month name in the configuration files.


New features

Added two scripts for development

The new script reset-wsusoffline.bash deletes all automatically created files: timestamps, hashdeep files, cached files, superseded updates and the ETag database for GitLab. It keeps the manually created file preferences.bash, to preserve custom settings.

The file syntax-check.bash does a quick syntax check of all scripts, using both bash and shellcheck.

Both scripts are meant for development.


Changed features

New method for the calculation of dynamic updates

Dynamic updates for both Windows and Office are extracted by the ProductId, separately for each product. Dynamic Office updates are now downloaded to the directories ../client/o2k13 and ../client/o2k16. The directory ../client/ofc is no longer used and may be deleted safely.

New method for the calculation of superseded updates

The calculation of superseded updates was changed in two ways:

aker introduced a patch to recognize updates, which are only superseded in certain contexts. This may reduce the number of "exceptions", which are maintained in the files ExcludeList-superseded-exclude.txt and ExcludeList-superseded-exclude-seconly.txt.

The list of superseded updates is also more complete, because the restriction to two file paths was removed from the file extract-update-cab-exe-ids-and-locations.xsl. This means, that more superseded updates will be recognized and fewer updates for Windows Server 2012 and Windows 8.1 / Server 2012 R2 will be downloaded. This is basically the Windows 10 patch, that I suggested about a year ago:


Static download files for .NET Frameworks are simplified

The files:

Code: Select all
StaticDownloadLinks-dotnet-x86-deu.txt
StaticDownloadLinks-dotnet-x64-deu.txt


are combined to:

Code: Select all
StaticDownloadLinks-dotnet-deu.txt


Hashdeep files are calculated with the bare mode option

The hashdeep relative path mode option -l was replaced with the bare mode option -b. This removes all path information from the filenames of the hashed files. The calculation of the integrity database can be much simplified:

  • The directory changes with pushd/popd are not necessary anymore.
  • Since the bare mode removes all directory information, there is no distinction between a Windows path and Linux path.
  • The counting of the input files was removed. This was mostly done for the dotnet directory, because single *.exe files were used as input. If these were missing, then the input would be empty, and hashdeep might try to read from standard input instead. The script could hang at this point. After the removal of the dotnet subdirectories, all directories are scanned recursively, and this prevents the possible bug.

Hashdeep files created with the relative path mode must be deleted once, because they would cause all updates to be reported as "moved".

The file hashes-wddefs.txt was replaced with two separate files for the subdirectories x86-glb and x64-glb.

Improved support for service packs

The option -includesp is supported by the scripts copy-to-target.bash, create-iso-image.bash and download-updates.bash. Service packs are read from the files:

Code: Select all
../exclude/custom/ExcludeList-SPs.txt
../client/static/StaticUpdateIds-w63-upd1.txt
../client/static/StaticUpdateIds-w63-upd2.txt


This way, users can provide their own service packs with a custom ExcludeList-SPs.txt.

The support for the option -includesp also means, that some downloads may be missing, if this option is not used. However, the option was named -includesp rather than -excludesp for consistency with the other optional downloads.

Internal changes

The file ../client/exclude/ExcludeList.txt can be updated from GitLab.

The files StaticUpdateIds-w63-upd1.txt and StaticUpdateIds-w63-upd2.txt are used for the calculation of the ExcludeList-Linux-superseded-seconly.txt.


Removed features

Removed the "revised" method for calculating superseded updates, if security-only updates are selected

One of Microsoft's original goals for cumulative monthly update rollups was to include former updates:

Over time, Windows will also proactively add patches to the Monthly Rollup that have been released in the past. Our goal is eventually to include all of the patches we have shipped in the past since the last baseline, so that the Monthly Rollup becomes fully cumulative and you need only to install the latest single rollup to be up to date.



This refers to the cumulative monthly update rollups only, not to the incremental security-only updates. Therefore, sometimes updates will superseded by the monthly update rollups, but not by the security-only updates. These updates would be added to the file ExcludeList-superseded-exclude-seconly.txt. However, this is only done, if these updates are found to be missing during installation.

The revised method for calculating superseded updates tried to recognize this situation automatically:

  1. The file HideList-seconly.txt contains a list of all known update rollups. The kb numbers are traced back to the FileIds, and then to the parent bundle records.
  2. The RevisionIds of the parent bundle records are removed from the list of superseding updates: If update rollups are excluded from download, then they cannot possibly supersede other updates.
  3. The list of superseded updates is then calculated as before. The file ExcludeList-Linux-superseded-seconly-revised.txt will be similar to the file ExcludeList-Linux-superseded-seconly.txt, but with an additional correction for the update rollups from the file HideList-seconly.txt.

This method actually worked – in two cases it could predict missing updates, before they were reported as missing:


But, as soon as all missing updates are added to the files ExcludeList-superseded-exclude.txt or ExcludeList-superseded-exclude-seconly.txt, this method would not make any difference anymore.

The revised method was introduced in the Linux download scripts, version 1.11 (2019-04-04) . It is removed now, because I don't know, how it fits into the new implementation of calculating superseded updates, or if it is still useful at all.


TODO

Download of Microsoft Edge (Chromium)



Downloads

The download links for this version are:


Link for the online check at VirusTotal. Note, that the URL includes the SHA-256 hash of the archive:


The hashes for the archive linux-scripts-2.4-b1.tgz are:
Code: Select all
MD5       40f3529c17f4ed5ecb4138907d82e7dd
SHA-1     3b8aae4811bd6635b3e54690f0c79f8752d7f089
SHA-256   a0e3d0065f8f69501701d6829cf5a7abac71b0851ec5e87409b42c620b2d9779




Further Reading

The first three posts in the original topic A complete rewrite of the Linux scripts are still being updated, to point to the latest version:

hbuhrmester
 
Posts: 506
Joined: 11.10.2013, 20:59

Re: NEWS for Linux download scripts, version 2.4 beta-1

Postby aker » 11.05.2021, 23:45

Just an idea:
We could try to get the rollup UpdateIds/FileIds based on their KB-number (which will fail on dotNET updates) or add the UpdateId(s) to HideList-SecOnly…
Wer Rechtschreibfehler findet, darf sie behalten oder an den Meistbietenden versteigern. / Everybody finding a misspelling is allowed to keep or sell it.
aker


WSUS Offline Update „Community Edition“ 12.5.1 / 11.9.9esr
https://gitlab.com/wsusoffline
aker
 
Posts: 3880
Joined: 02.03.2011, 15:32

Re: NEWS for Linux download scripts, version 2.4 beta-1

Postby aker » 16.05.2021, 16:16

I plan to change HideList-SecOnly.txt to contain the UpdateID instead of the KB-number. I'd like to use the UpdateID as WUA reports it on client-side, too.
As I do not have the time to test this, could you check, which updates will not be superseded, if we ignore the following UpdateIds when calcuating the superseded updates:
Code: Select all
w60
950b28b9-1281-4bce-8434-d595e60eca1f (2021-05 rollup w2k8 x86)
e749c40e-4db0-4482-92b6-3711f9cbb0d7 (2021-05 rollup w2k8 x64)

w61
6280222c-cce0-4b00-81f7-698ea2b445de (2021-05 rollup w61 x86)
72608cae-122f-4108-8991-f07d674a6c85 (2021-05 rollup w61 x64)
b6cc4f2b-93d2-4ac0-aa1d-b4f9245baee4 (2021-05 rollup w61 embedded x86)
9f08a240-8de3-441d-8af7-bdf692e657ea (2021-05 rollup w61 embedded x64)
fe9222fd-9660-4d61-9cde-5887177b6e5b (2021-05 rollup w2k8r2 x64)

w62
7028e00e-2edb-41ad-ab6c-a1da934bba41 (2021-05 rollup w62 embedded x86)
07765547-13b4-4bad-978c-7bc3a3213f71 (2021-05 rollup w62 embedded x64)
101a6390-dd1a-4835-8e49-9fe167f78a52 (2021-05 rollup w2k12 x64)

w63
9c655212-54c2-41a4-b0dd-2bce266fe67d (2021-05 rollup w63 x86)
62b4282b-266d-4f40-8ca3-d97ffadc7033 (2021-05 rollup w63 x64)
a4d880cb-c4d5-4974-b55b-015f5388201f (2021-05 rollup w2k12r2 x64)
Wer Rechtschreibfehler findet, darf sie behalten oder an den Meistbietenden versteigern. / Everybody finding a misspelling is allowed to keep or sell it.
aker


WSUS Offline Update „Community Edition“ 12.5.1 / 11.9.9esr
https://gitlab.com/wsusoffline
aker
 
Posts: 3880
Joined: 02.03.2011, 15:32

Re: NEWS for Linux download scripts, version 2.4 beta-1

Postby aker » 30.05.2021, 13:38

Ich pushe das Thema "superseded Updates" noch einmal, da ich dies in der nächsten Version angehen möchte.
Ich bin der Meinung, dass man Updates, welche geblacklistet sind, anhand Ihrer UpdateId aus der Superseded-Rechnung ausschließen sollte, weiß aber noch nicht, wie man das intelligent hinbekommt.

Die Diskussion dazu möchte ich aber gerne in ein eigenes Thema verlagern.
:arrow: viewtopic.php?f=3&t=10259

Viele Grüße
Wer Rechtschreibfehler findet, darf sie behalten oder an den Meistbietenden versteigern. / Everybody finding a misspelling is allowed to keep or sell it.
aker


WSUS Offline Update „Community Edition“ 12.5.1 / 11.9.9esr
https://gitlab.com/wsusoffline
aker
 
Posts: 3880
Joined: 02.03.2011, 15:32


Return to Linux

Who is online

Users browsing this forum: No registered users and 2 guests