Error: File integrity verification failure

[Killerbob]

Trying to get updates for windows 8.1 and this is the log

16/04/2015 23:11:36.40 - Info: Starting WSUS Offline Update download (v. 9.5.4) for w63-x64 glb
16/04/2015 23:11:36.40 - Info: Option /includedotnet detected
16/04/2015 23:11:36.40 - Info: Option /verify detected
16/04/2015 23:11:36.41 - Info: Option /exitonerror detected
16/04/2015 23:11:36.55 - Info: Set time zone to LOC-1:00
16/04/2015 23:11:36.80 - Info: Updated static download definitions
16/04/2015 23:11:36.97 - Info: Downloaded/validated mkisofs tool
16/04/2015 23:11:37.04 - Info: Found sigcheck.exe version 2.20.0.0
16/04/2015 23:11:37.05 - Info: Verified integrity of Windows Update Agent installation and catalog files
16/04/2015 23:11:38.50 - Info: Downloaded/validated most recent Windows Update Agent installation and catalog files
16/04/2015 23:11:38.50 - Info: Verified digital file signatures of Windows Update Agent installation and catalog files
16/04/2015 23:11:38.50 - Info: Created integrity database for Windows Update Agent installation and catalog files
16/04/2015 23:11:39.96 - Info: Verified integrity of .NET Frameworks' installation files
16/04/2015 23:11:39.96 - Info: Skipped download/validation of .NET Frameworks' files (x64) due to 'same day' rule
16/04/2015 23:11:41.94 - Info: Verified integrity of C++ Runtime Libraries' installation files
16/04/2015 23:11:41.94 - Info: Skipped download/validation of C++ Runtime Libraries' installation files due to 'same day' rule
16/04/2015 23:11:42.31 - Error: File integrity verification failure

[aker]

Please delete all files inside .\client\md.

[hbuhrmester]

hashdeep produces a lot of useful output, but it is easily lost, because the window of cmd.exe is rather small, the text scrolls too fast, and the window is immediately closed, as soon as the script DownloadUpdates.cmd finishes.

cmd.exe could be started with the option /K to change this. For example, try this simple command in the "Run" window:

Code: Select all

cmd.exe /k echo hello world

It would also help, if the output of hashdeep could be saved. To do so, change the line:

Code: Select all

..\bin\%HASHDEEP_EXE% -a -l -vv -k hashes-wsus.txt -r ..\wsus

to:

Code: Select all

..\bin\%HASHDEEP_EXE% -a -l -vv -k hashes-wsus.txt -r ..\wsus >>..\%DOWNLOAD_LOGFILE%

Then the output will be neatly inserted into the logfile:

Code: Select all

17.04.2015 20:22:14,28 - Info: Starting WSUS Offline Update download (v. 9.6) for w61 glb
17.04.2015 20:22:14,28 - Info: Option /includedotnet detected
17.04.2015 20:22:14,31 - Info: Option /includewle detected
17.04.2015 20:22:14,33 - Info: Option /includemsse detected
17.04.2015 20:22:14,35 - Info: Option /includewddefs detected
17.04.2015 20:22:14,72 - Info: Option /verify detected
17.04.2015 20:22:14,74 - Info: Option /exitonerror detected
17.04.2015 20:22:14,76 - Info: Option /proxy detected
17.04.2015 20:22:15,17 - Info: Set time zone to LOC-2:00
17.04.2015 20:22:15,88 - Info: Updated static download definitions
17.04.2015 20:22:15,92 - Info: Downloaded/validated mkisofs tool
17.04.2015 20:22:18,22 - Info: Found sigcheck.exe version 2.10.0.0
hashdeep.exe: Audit passed
   Input files examined: 0
  Known files expecting: 0
          Files matched: 3
Files partially matched: 0
            Files moved: 0
        New files found: 0
  Known files not found: 0
17.04.2015 20:22:18,22 - Info: Verified integrity of Windows Update Agent installation and catalog files

If anything is wrong, hashdeep will explain the reasons:

Code: Select all

18.04.2015  0:21:54,67 - Info: Verified integrity of C++ Runtime Libraries' installation files
18.04.2015  0:21:54,67 - Info: Skipped download/validation of C++ Runtime Libraries' installation files due to 'same day' rule
..\wddefs\x86-glb\mpas-fe.exe: No match
..\wddefs\x86-glb\mpas-fe.exe: Known file not used
hashdeep.exe: Audit failed
   Input files examined: 0
  Known files expecting: 0
          Files matched: 1
Files partially matched: 0
            Files moved: 0
        New files found: 1
  Known files not found: 1
18.04.2015  0:22:07,02 - Error: File integrity verification failure

hashdeep is designed to compare file sizes and file checksums: If an existing file does not match any of the stored values for file size and checksums, it will be "no match". If a stored record for file size and checksums does not match an existing file, it appears to be "not used".

The "audit" mode is very strict. It requires, that

• there is one file for every known hash in the checksum file
• there is one line of checksums for every file, which is specified on the command line

Any wrong number of files or stored checksums will cause the audit to fail.

There are some other comparison modes. The "matching" mode -m simply lists matching files, but it won't give a simple answer like "passed" or "failed". The man page of hashdeep explains the differences like this:

Code: Select all

-a    Audit  mode.  Each  input  file  is  compared against the set of
      knowns.  An audit is said to pass if each input file is  matched
      against  exactly  one file in set of knowns. Any collisions, new
      files, or missing files will make the  audit  fail.  Using  this
      flag  alone  produces a message, either "Audit passed" or "Audit
      Failed". Use the verbose modes, -v, for more details.  Using  -v
      prints  the  number of files in each category. Using -v a second
      time prints any discrepancies. Using -v a third time prints  the
      results for every file examined and every known file.
      Due  to  limitations  in the program, any filenames with Unicode
      characters will appear to have moved during an  audit.  See  the
      section "UNICODE SUPPORT" below.

-m    Positive  matching,  requires  at  least one use of the -k flag.
      The input files are examined one at a time, and only those files
      that match the list of known hashes are output. The only accept‐
      able format for known hashes is the output of previous  hashdeep
      runs.
       If standard input is used with the -m flag, displays "stdin" if
      the input matches one of the hashes in the list of known hashes.
      If the hash does not match, the program displays no output.
       This flag may not be used in conjunction with the -x, -X, or -a
      flags.  See the section "UNICODE SUPPORT" below.

When hashdeep is creating the checksum files, it may print some short error messages to standard error. These can also be redirected to the download log:

Code: Select all

..\bin\%HASHDEEP_EXE% -c md5,sha1,sha256 -W hashes-wsus.txt -l -r ..\wsus 2>>..\%DOWNLOAD_LOGFILE%

Greetings,
Hartmut

[hbuhrmester]

The four virus definition files mpas-feX64.exe, mpas-fe.exe, mpam-fex64.exe, and mpam-fe.exe seem to be a major problem. I have a quite stable connection, and I can see the outdoor DSLAM of the Deutsche Telekom right from my window. This is a grey box, as shown in https://de.wikipedia.org/wiki/DSLAM , and it is only 20 meters away.

Still, I already got four damaged files, all of them virus definition files. The other downloads are never damaged.

The problem is, that these files may be changing on the server, while wget is retrieving them. This will interrupt the download, but wget will just try again and continue the download. wget keeps the already downloaded parts and writes to the same file, until the download is complete. But if the file contents change between these tries, the saved file will be damaged.

I am now testing a new approach to download these files:

1. wget is used with the option --tries=1. It only gets one try for each download.
2. Then the script must do the error handling itself: It must distinguish between transient errors and major errors like "connection refused" or "file not found".
3. After transient errors, the download is started again up to ten times, with a delay ranging from 10 seconds to 90 seconds.

This is similar to what wget can do itself, but the difference is, that wget can not just continue the download, but has to start from nothing on each try. As a result, wget should finally download one complete, undamaged file.

As a simple Linux script, this approach looks like:

Code: Select all

#!/bin/bash

# Filename: get-virus-definitions.sh
#
# A simple script to download the four virus definition files
# mpas-feX64.exe, mpas-fe.exe, mpam-fex64.exe, and mpam-fe.exe, using
# a new, robust download method.
#
# Run the script from wsusoffline/sh or another directory at the same
# level, because the script expects to find the static directory at
# "../static".

global_download_dir=~/tmp/wsusoffline
download_logfile=~/tmp/wsusoffline/download.log
debug=1


function download_with_wget {
    if [ $# -ne 2 ]; then
        echo "function download_with_wget: Wrong number of arguments."
    fi
    local download_dir="$1"
    local download_link="$2"
    local max_tries=10
    local default_wait_time=10
    if [ $debug -eq 0 ]; then
        local wget_verbosity="--no-verbose"
        local wget_throttling=""
    else
        local wget_verbosity="--verbose --progress=dot:mega"
        local wget_throttling="--limit-rate=20k"
    fi

    mkdir -p "${download_dir}"

    for ((i=1 ; i<=${max_tries} ; i++)); do

        wget --tries=1 --timeout=300 \
        --timestamping --unlink \
        ${wget_verbosity} ${wget_throttling} \
        --append-output="${download_logfile}" \
        --directory-prefix="${download_dir}" \
        "${download_link}"

        last_return_code=$?
        case ${last_return_code} in
            0)
                break
                ;;
            1)
                echo "Transient error: Generic error while retrieving ${download_link}." >> "${download_logfile}"
                ;;
            2)
                echo "Unrecoverable error: Parse error while retrieving ${download_link}." >> "${download_logfile}"
                break
                ;;
            3)
                echo "Transient error: File I/O error while retrieving ${download_link}." >> "${download_logfile}"
                ;;
            4)
                echo "Transient error: Network failure while retrieving ${download_link}." >> "${download_logfile}"
                ;;
            5)
                echo "Unrecoverable error: SSL verification failure while retrieving ${download_link}." >> "${download_logfile}"
                break
                ;;
            6)
                echo "Unrecoverable error: Username/password authentication failure while retrieving ${download_link}." >> "${download_logfile}"
                break
                ;;
            7)
                echo "Unrecoverable error: Protocol error while retrieving ${download_link}." >> "${download_logfile}"
                break
                ;;
            8)
                echo "Unrecoverable error: Server error while retrieving ${download_link}." >> "${download_logfile}"
                break
                ;;
            *)
                echo "Unknown condition." >> "${download_logfile}"
                break
                ;;
        esac

        # Sleep from 10 seconds to 90 seconds between tries
        if [ ${i} -lt ${max_tries} ]; then
            let wait_time=${i}*${default_wait_time}
            echo "Retrying in ${wait_time} seconds..." >> "${download_logfile}"
            sleep ${wait_time}
            wget_throttling=""
        fi

    done
}

grep --no-filename "mpa[ms]-fe" ../static/StaticDownloadLink-wddefs-*-glb.txt ../static/StaticDownloadLinks-msse-*-glb.txt | while read global_download_link; do
    download_with_wget ${global_download_dir} ${global_download_link}
    sleep 10s
done

The download log after running this script looks like:

Code: Select all

--2015-04-18 17:56:52--  http://download.microsoft.com/download/DefinitionUpdates/mpas-fe.exe
Auflösen des Hostnamen »download.microsoft.com (download.microsoft.com)«... 80.157.151.10, 80.157.151.9, 80.157.151.17, ...
Verbindungsaufbau zu download.microsoft.com (download.microsoft.com)|80.157.151.10|:80... verbunden.
HTTP-Anforderung gesendet, warte auf Antwort... 200 OK
Länge: 32642840 (31M) [application/octet-stream]
Größen stimmen nicht überein (lokal 16777216) -- erneuter Download.

--2015-04-18 17:56:52--  http://download.microsoft.com/download/DefinitionUpdates/mpas-fe.exe
Wiederverwendung der bestehenden Verbindung zu download.microsoft.com:80.
HTTP-Anforderung gesendet, warte auf Antwort... 200 OK
Länge: 32642840 (31M) [application/octet-stream]
In »»/home/anwender/tmp/wsusoffline/mpas-fe.exe«« speichern.

     0K ........ ........ ........ ........ ........ ........  9% 20,0K 24m3s
  3072K ........ ........ ........ ........ ........ ........ 19% 20,0K 21m28s
  6144K ........ ........ ........ ........ ........ ........ 28% 20,0K 18m54s
  9216K ........ ........ ........ ........ ........ ........ 38% 20,0K 16m20s
 12288K ........ ........ ........ ........ ........ ........ 48% 20,0K 13m46s
 15360K ........ ........ ........ ........ ........ ........ 57% 20,0K 11m12s
 18432K ........ ........ ........ ........ ........ ........ 67% 20,0K 8m39s
 21504K ........ ........ ........ ........ ........ ........ 77% 20,0K 6m5s
 24576K ........ ........ ........ ........ ........ ........ 86% 20,0K 3m32s
 27648K ..                                                    87% 20,8K=23m9s

2015-04-18 18:20:01 (20,0 KB/s) - Verbindung bei Byte 28445501 geschlossen. Aufgegeben.

Transient error: Network failure while retrieving http://download.microsoft.com/download/DefinitionUpdates/mpas-fe.exe.
Retrying in 10 seconds...
--2015-04-18 18:20:12--  http://download.microsoft.com/download/DefinitionUpdates/mpas-fe.exe
Auflösen des Hostnamen »download.microsoft.com (download.microsoft.com)«... 80.157.151.10, 80.157.151.17, 2003:8:0:27::509d:970a, ...
Verbindungsaufbau zu download.microsoft.com (download.microsoft.com)|80.157.151.10|:80... verbunden.
HTTP-Anforderung gesendet, warte auf Antwort... 200 OK
Länge: 32642840 (31M) [application/octet-stream]
Größen stimmen nicht überein (lokal 28445501) -- erneuter Download.

--2015-04-18 18:20:12--  http://download.microsoft.com/download/DefinitionUpdates/mpas-fe.exe
Wiederverwendung der bestehenden Verbindung zu download.microsoft.com:80.
HTTP-Anforderung gesendet, warte auf Antwort... 200 OK
Länge: 32644888 (31M) [application/octet-stream]
In »»/home/anwender/tmp/wsusoffline/mpas-fe.exe«« speichern.

     0K ........ ........ ........ ........ ........ ........  9%  694K 42s
  3072K ........ ........ ........ ........ ........ ........ 19%  688K 37s
  6144K ........ ........ ........ ........ ........ ........ 28%  688K 33s
  9216K ........ ........ ........ ........ ........ ........ 38%  686K 28s
 12288K ........ ........ ........ ........ ........ ........ 48%  688K 24s
 15360K ........ ........ ........ ........ ........ ........ 57%  687K 20s
 18432K ........ ........ ........ ........ ........ ........ 67%  689K 15s
 21504K ........ ........ ........ ........ ........ ........ 77%  689K 11s
 24576K ........ ........ ........ ........ ........ ........ 86%  689K 6s
 27648K ........ ........ ........ ........ ........ ........ 96%  689K 2s
 30720K ........ ........ ..                                 100%  689K=46s

2015-04-18 18:20:58 (689 KB/s) - »»/home/anwender/tmp/wsusoffline/mpas-fe.exe«« gespeichert [32644888/32644888]

Note, that the download rate is deliberately limited on the first try, to actually provoke a download error.

The saved file after restarting the download is okay:

Code: Select all

$ wine ../bin/sigcheck.exe ~/tmp/wsusoffline/mpas-fe.exe

Sigcheck v2.1 - File version and signature viewer
Copyright (C) 2004-2014 Mark Russinovich
Sysinternals - www.sysinternals.com

Z:\home\anwender\tmp\wsusoffline\mpas-fe.exe:
        Verified:       Signed
        Signing date:   14:48 18.04.2015
        Publisher:      Microsoft Corporation
        Description:    AntiMalware Definition Update
        Product:        Microsoft Malware Protection
        Prod version:   1.195.3711.0
        File version:   1.195.3711.0
        MachineType:    32-bit

Greetings,
Hartmut

[boco]

So true. Not only can the file change while it is downloaded, there are more possible causes: A/V programs or firewalls, or the simple fact that MS has many download mirrors that are not in sync!
Please note that the corrupted file is actually deleted later as the sigcheck fails.

Apart from your proposed changes I'd even go further: MPAM/MPAS should become a run of its own due to their tendency to fail. Currently a failed MPAM/MPAS leads to a busted OS run because the signature is then missing on the resulting medium. And it would have the additional benefit of having to be run one time instead of a part of each and every run.

That's currently the one and only reason I have to break and restart download runs often.

[hbuhrmester]

Apart from your proposed changes I'd even go further: MPAM/MPAS should become a run of its own due to their tendency to fail. Currently a failed MPAM/MPAS leads to a busted OS run because the signature is then missing on the resulting medium. And it would have the additional benefit of having to be run one time instead of a part of each and every run.

Hello,

I translated the Linux shell script of my previous post to a Windows batch script. It is meant to download the four virus definition files, using its own error handling to prevent damaged files. The implementation is still somewhat basic, but it works.

This Windows script uses the usual download directories for the definition updates. It can be used alongside the main update script, to get the virus definition files more often.

This is useful, because these files change often, typically several times a day.

Updates for Windows or Office are usually only released once a month, on the official "patch day", which is the second Tuesday each month. Of course, urgent patches will be released in between.

The script should be placed into wsusoffline/cmd or another directory at the same level, so that it can find needed applications and directories.

Note: The file signature check requires Sysinternals Sigcheck version 2.0 or higher. Previous versions of this tools used different command line options.

Code: Select all

@echo off
setlocal enableextensions enabledelayedexpansion

rem file GetVirusDefinitions.cmd

rem A short example script for a more reliable download of the virus
rem definition updates. This script uses wget with the option --tries=1
rem and implements its own error handling, rather than letting wget have
rem its way.

rem TODO: There should be a small delay between tries. This could be
rem implemented with the sleep command from the GNUWin32 project. There
rem is also a sleep command in the Windows Resource Kit Tools and a
rem "timeout" command introduced in Windows Vista or Windows 7. A common
rem workaround, which works with all Windows versions, is the use of
rem "ping" with the number of repeats in seconds.

rem http://stackoverflow.com/questions/166044/sleeping-in-a-batch-file
rem http://stackoverflow.com/questions/5419976/is-it-possible-to-issue-a-wait-command-in-a-batch-file

rem Start of main program

    echo Getting Microsoft Security Essentials definiton updates (x64)...
    echo.
    set DownloadFolder=..\client\msse\x64-glb
    set DownloadLink=http://download.microsoft.com/download/DefinitionUpdates/mpam-fex64.exe
    set DownloadFile=mpam-fex64.exe
    set HashedFolder=..\msse
    set HashesFile=hashes-msse.txt

    call :DownloadWithWget %DownloadFolder% %DownloadLink%
    call :VerifyFileSignature %DownloadFolder% %DownloadFile%
    call :UpdateIntegrityDatabase %HashedFolder% %HashesFile%
    echo.

    echo Getting Microsoft Security Essentials definiton updates (x86)...
    echo.
    set DownloadFolder=..\client\msse\x86-glb
    set DownloadLink=http://download.microsoft.com/download/DefinitionUpdates/mpam-fe.exe
    set DownloadFile=mpam-fe.exe
    set HashedFolder=..\msse
    set HashesFile=hashes-msse.txt

    call :DownloadWithWget %DownloadFolder% %DownloadLink%
    call :VerifyFileSignature %DownloadFolder% %DownloadFile%
    call :UpdateIntegrityDatabase %HashedFolder% %HashesFile%
    echo.

    echo Getting Windows Defender definiton updates (x64)...
    echo.
    set DownloadFolder=..\client\wddefs\x64-glb
    set DownloadLink=http://download.microsoft.com/download/DefinitionUpdates/mpas-feX64.exe
    set DownloadFile=mpas-feX64.exe
    set HashedFolder=..\wddefs
    set HashesFile=hashes-wddefs.txt

    call :DownloadWithWget %DownloadFolder% %DownloadLink%
    call :VerifyFileSignature %DownloadFolder% %DownloadFile%
    call :UpdateIntegrityDatabase %HashedFolder% %HashesFile%
    echo.

    echo Getting Windows Defender definiton updates (x86)...
    echo.
    set DownloadFolder=..\client\wddefs\x86-glb
    set DownloadLink=http://download.microsoft.com/download/DefinitionUpdates/mpas-fe.exe
    set DownloadFile=mpas-fe.exe
    set HashedFolder=..\wddefs
    set HashesFile=hashes-wddefs.txt

    call :DownloadWithWget %DownloadFolder% %DownloadLink%
    call :VerifyFileSignature %DownloadFolder% %DownloadFile%
    call :UpdateIntegrityDatabase %HashedFolder% %HashesFile%

rem End of main program
goto :eof


rem Labels can work like functions, if they are called with "call". In
rem Microsoft speak, this creates a new "batch context". This batch
rem context can take positional parameters like a batch file. It is left
rem with "goto :eof". The script resumes execution after the command,
rem which created the batch context.

rem Start of function "DownloadWithWget"
:DownloadWithWget

    echo Function: DownloadWithWget
    echo DownloadFolder: %1
    echo DownloadLink: %2
    echo.

    rem Start or restart download
    rem See "help set" for option /A
    set /A NumberOfTries=0
    :Try
    set /A NumberOfTries="NumberOfTries+1"
    echo Number of tries: %NumberOfTries%
    if %NumberOfTries% GTR 10 (
        echo Giving up
        goto :Finally
    )

    echo wget --tries=1 --timeout=60 --timestamping --directory-prefix=%1 %2
    ..\bin\wget --tries=1 --timeout=60 --timestamping --directory-prefix=%1 %2

    rem See "help if" for the usage of %errorlevel%
    goto :CatchError%errorlevel%

    rem The Windows port of wget doesn't seem to use all return codes
    rem of the Linux version. If the connection is closed, it returns
    rem the generic error code 1, while this would be a network failure
    rem (error code 4) on Linux.
    :CatchError0
    echo No error: Download/validation successfull
    goto :Finally
    :CatchError1
    rem Connection closed
    echo Transient error: Generic error
    rem sleep for 10 seconds
    ping 127.0.0.1 -n 11 -w 1000 >nul 2>nul
    goto :Try
    :CatchError2
    echo Unrecoverable error: Parse error
    goto :Finally
    :CatchError3
    echo Transient error: File I/O error
    rem sleep for 10 seconds
    ping 127.0.0.1 -n 11 -w 1000 >nul 2>nul
    goto :Try
    :CatchError4
    echo Transient error: Network failure
    rem sleep for 10 seconds
    ping 127.0.0.1 -n 11 -w 1000 >nul 2>nul
    goto :Try
    :CatchError5
    echo Unrecoverable error: SSL verification failure
    goto :Finally
    :CatchError6
    echo Unrecoverable error: Username/password authentication failure
    goto :Finally
    :CatchError7
    echo Unrecoverable error: Protocol error
    goto :Finally
    :CatchError8
    rem Error 404, file not found
    echo Unrecoverable error: Server error
    goto :Finally
    :Finally
    echo.
    echo.

rem End of function "DownloadWithWget"
goto :eof


rem Start of function "VerifyFileSignature"
:VerifyFileSignature

    echo Function: VerifyFileSignature
    echo DownloadFolder: %1
    echo DownloadFile: %2
    echo.

    rem All tokens in the tabular output of sigcheck are already quoted.
    rem They must not be double-quoted, or the comparison will fail.
    for /F "tokens=1,2 delims=," %%i in ('..\bin\sigcheck.exe -q -c %1\%2') do (
        echo i = %%i
        echo j = %%j
        echo.
        if %%j=="Unsigned" (
            echo Warning: deleting unsigned file %%i
            del %%i
            echo.
        )
    )
    echo.

rem End of function "VerifyFileSignature"
goto :eof


rem Start of function "UpdateIntegrityDatabase"
:UpdateIntegrityDatabase

    echo Function: UpdateIntegrityDatabase
    echo HashedFolder: %1 (referenced from ..\client\md)
    echo HashesFile: %2
    echo.

    pushd ..\client\md
    ..\bin\hashdeep.exe -c md5,sha1,sha256 -W %2 -l -r %1
    popd
    echo.

rem End of function "UpdateIntegrityDatabase"
goto :eof


[boco]

Security updates are released on second Tuesday of the moth. Feature updates are, if not released with the security patches, released on the other, rarely known patch day (we just had one at the 21st). Since WSUSOU uses mainly security patches only the first patch day is of interest for us.

With that script we could remove the downloading of the signature files from the main program. I could imagine the following order of action in the future:
1. Get all the main patches and statics that rarely change/update.
2. Call the script (if desired) to update the signatures at the end.
3. Make the media.