crl/crt download don't work due to blanks in URL

crl/crt download don't work due to blanks in URL

Postby mani » 07.01.2022, 11:13

Happy new year
After updating from a beta version to the release version in December I thought first that I made a mistake and therefore I started with a virgin version again, but have the same error:
    2022-01-07 00:23:13 - Info: Determining static update links for win x86 glb ...
    2022-01-07 00:23:13 - Info: Created file ValidStaticLinks-win-x86-glb.txt
    2022-01-07 00:23:13 - Info: Downloading/validating 16 link(s) from input file ValidStaticLinks-win-x86-glb.txt ...
    --2022-01-07 00:23:13-- http://crl.microsoft.com/pki/crl/produc ... -06-23.crl
    Resolving crl.microsoft.com (crl.microsoft.com)... 104.83.4.248, 104.83.4.218, 2a02:26f0:dc::6853:508, ...
    Connecting to crl.microsoft.com (crl.microsoft.com)|104.83.4.248|:80... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 824 [application/octet-stream]
    Saving to: '../client/win/glb/MicRooCerAut_2010-06-23.crl'

    0K 100% 58.2M=0s

    2022-01-07 00:23:14 (58.2 MB/s) - '../client/win/glb/MicRooCerAut_2010-06-23.crl' saved [824/824]

    --2022-01-07 00:23:14-- https://www.microsoft.com/pki/certs/Mic ... _03_22.crt
    Resolving http://www.microsoft.com (http://www.microsoft.com)... 184.51.10.83, 2a02:26f0:10e:3b1::356e, 2a02:26f0:10e:398::356e
    Connecting to http://www.microsoft.com (http://www.microsoft.com)|184.51.10.83|:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 1521 (1.5K) [application/octet-stream]
    Saving to: '../client/win/glb/MicRooCerAut2011_2011_03_22.crt'

    0K 100% 37.9M=0s

    2022-01-07 00:23:15 (37.9 MB/s) - '../client/win/glb/MicRooCerAut2011_2011_03_22.crt' saved [1521/1521]

    --2022-01-07 00:23:15-- https://www.microsoft.com/pkiops/certs/Microsoft
    Resolving http://www.microsoft.com (http://www.microsoft.com)... 184.51.10.83, 2a02:26f0:10e:398::356e, 2a02:26f0:10e:3b1::356e
    Connecting to http://www.microsoft.com (http://www.microsoft.com)|184.51.10.83|:443... connected.
    HTTP request sent, awaiting response... 403 Forbidden
    2022-01-07 00:23:15 ERROR 403: Forbidden.

    2022-01-07 00:23:15 - Error: Download/validation of Microsoft failed

    .
    .
    2022-01-07 00:23:18 - Warning: There were 14 runtime errors while downloading/validating links from input file ValidStaticLinks-win-x86-glb.txt. See the download log for details

with adding at line 429 to 40-configure-downloaders.bash
Code: Select all
    if [[ ${input_file##*/} == 'ValidStaticLinks-win-x86-glb.txt' && -n $local_filename ]]
    then
        download_link="$download_link $local_filename $skip_rest"
        local_filename=''
    fi

i get all the certificates, but they get deleted immediately afterwards:
    2022-01-07 11:03:47 - Info: Downloaded/validated 16 link(s)
    2022-01-07 11:03:47 - Info: Cleaning up download directory ../client/win/glb ...
    2022-01-07 11:03:47 - Info: Trashing/deleting obsolete file Microsoft ECC Product Root Certificate Authority 2018.crl ...
    2022-01-07 11:03:47 - Info: The file Microsoft ECC Product Root Certificate Authority 2018.crl was deleted directly.
    2022-01-07 11:03:47 - Info: Trashing/deleting obsolete file Microsoft ECC Product Root Certificate Authority 2018.crt ...

Is this a mis-configuration on my side or a bug?
    2022-01-06 17:31:04 - Info: Starting download-updates.bash 2.6 (2021-10-18)
    2022-01-06 17:31:04 - Info: Command line: ./download-updates.bash w100,w100-x64,o2k13-x64 deu,enu -includecpp -includedotnet
    2022-01-06 17:31:04 - Info: Running on WSUS Offline Update, Community Edition 12.6.1
    Local time: Thu, 06 Jan 2022 17:31:04 +0100
    OS type: linux-gnu
    Kernel name: Linux
    Kernel details: Linux server1 5.10.0-9-amd64 #1 SMP Debian 5.10.70-1 (2021-09-30) x86_64 GNU/Linux
    Hardware: x86_64
    Bash version: 5.1.4(1)-release
    Terminal type: dumb

    Linux distribution
    Distributor ID: Debian
    Description: Debian GNU/Linux 11 (bullseye)
    Release: 11
    Codename: bullseye

    Environment
    LC_ALL=C
    LC_COLLATE=
    LC_CTYPE=
    LC_MESSAGES=
    LANG=

    Resolution of the installation directory
    Canonical name: /files/wsusCE/sh/download-updates.bash
    Script name: download-updates.bash
    Home directory: /files/wsusCE/sh
    Working directory: /files/wsusCE/sh
    WSUS Offline Upd.: /files/wsusCE
    Temp directory: /tmp/download-updates.J8dm3l

    Configuration variables from the preferences file
    prefer_seconly: disabled

Mani
mani
 
Posts: 21
Joined: 04.04.2017, 15:50

Re: crl/crt download don't work due to blanks in URL

Postby hbuhrmester » 07.01.2022, 21:55

I guess, it's a bug:

Long time ago, the Windows script AddCustomLanguageSupport.cmd did insert trailing spaces into the custom static download files. Adding a space to the Internal Field Separator (IFS) was a simple workaround to delete trailing spaces. But this breaks now, that we have URLs with spaces.

Necessary fixes:

Code: Select all
Index: common-tasks/40-configure-downloaders.bash
===================================================================
--- common-tasks/40-configure-downloaders.bash   (Revision 185)
+++ common-tasks/40-configure-downloaders.bash   (Arbeitskopie)
@@ -417,14 +417,7 @@
 
     log_info_message "Downloading/validating ${number_of_links} link(s) from input file ${input_file##*/} ..."
 
-    # Setting IFS to a comma and space removes spurious spaces at the
-    # end of the line. These may be found in some files created by the
-    # Windows script AddCustomLanguageSupport.cmd:
-    #
-    # ../static/custom/StaticDownloadLinks-dotnet-x64-glb.txt
-    # ../static/custom/StaticDownloadLinks-dotnet-x86-glb.txt
-
-    while IFS=', ' read -r download_link local_filename skip_rest
+    while IFS=',' read -r download_link local_filename skip_rest
     do
         remote_filename="${download_link##*/}"
 
Index: libraries/cleanup-client-directories.bash
===================================================================
--- libraries/cleanup-client-directories.bash   (Revision 185)
+++ libraries/cleanup-client-directories.bash   (Arbeitskopie)
@@ -179,15 +179,9 @@
             esac
 
             # Keep files, which are in the current download set
-            #
-            # Use percent-encoding for filenames with spaces, to compare
-            # them with the input files in the static directory. This
-            # is necessary for *.crt and *.crl files, because the input
-            # file StaticDownloadLinks-win-glb.txt uses percent encoding
-            # in the first place.
             if [[ -s "${valid_links}" ]]
             then
-                if grep -F -i -q "${filename// /%20}" "${valid_links}"
+                if grep -F -i -q "${filename}" "${valid_links}"
                 then
                     log_debug_message "Found file \"${filename}\""
                     continue
@@ -200,7 +194,7 @@
             # "valid static files".
             if [[ "${keep_valid_static_files}" == "enabled" ]]
             then
-                if grep -F -i -q -r "${filename// /%20}" "../static"
+                if grep -F -i -q -r "${filename}" "../static"
                 then
                     log_info_message "Kept valid static file \"${filename}\""
                     continue
hbuhrmester
 
Posts: 517
Joined: 11.10.2013, 20:59

Re: crl/crt download don't work due to blanks in URL

Postby aker » 08.01.2022, 11:38

I made that change due to some misbehaviour in the Windows scripts when using the "%20" syntax (I don't remember the exact reason for it anymore as I was really busy at that time) and fixed the Windows scripts to handle the change correctly.

EDIT: Changes have been integrated:
https://gitlab.com/wsusoffline/wsusoffline/-/commit/385d4d381479929c0f245ab3486452b801bead34
https://gitlab.com/wsusoffline/wsusoffline/-/commit/f96ffd4703ad6c9c2b4a59daff5315e3f46b761b

@hbuhrmester
As I'm not good with Linux scripts:
Should I merge this change into wsusou and republish it?
EDIT2: I just uploaded a "hf1".
Wer Rechtschreibfehler findet, darf sie behalten oder an den Meistbietenden versteigern. / Everybody finding a misspelling is allowed to keep or sell it.
aker

WSUS Offline Update „Community Edition“
https://gitlab.com/wsusoffline/wsusoffline/-/releases
aker
 
Posts: 3979
Joined: 02.03.2011, 15:32

Re: crl/crt download don't work due to blanks in URL

Postby mani » 08.01.2022, 11:45

Thanks, I can confirm that 16 Certificates are now in ../client/win/glb again.

Mani
mani
 
Posts: 21
Joined: 04.04.2017, 15:50


Return to Linux

Who is online

Users browsing this forum: No registered users and 3 guests