DownloadUpdates.sh - Timestamping and -O

[nebyah]

Code: Select all

wget -nv -N --timeout=120 http://mse.dlservice.microsoft.com/download/0/2/C/02C8AB73-0774-4975-826F-9E8A0FD7F65D/dede/x86/mseinstall.exe -O ../client/msse/x86-glb/mseinstall-x86-deu.exe
WARNING: timestamping does nothing in combination with -O. See the manual

When -O is used, wget ignores -N (timestamping). This causes some files (vcredist*, mseinstall*, possibly others) to be downloaded several times, each time wsusoffline is run.

Might I suggest something like:

Code: Select all

mkdir -p ../cache/mseinstall-x68-deu.exe \
&& wget -nv -N --timeout=120 http://mse.dlservice.microsoft.com/download/0/2/C/02C8AB73-0774-4975-826F-9E8A0FD7F65D/dede/x86/mseinstall.exe -P ../cache/mseinstall-x86-deu.exe/ \
&& cp -a ../cache/mseinstall-x86-deu.exe/mseinstall.exe  ../client/msse/x86-glb/mseinstall-x86-deu.exe

[WSUSUpdateAdmin]

Hi!

Yes, thanks, you're right, but the modification you suggest can't be done in that simple way, because all downloads are handled by the subroutine doWget(), so one has to modify this sub depending on required renaming.

Regards
Torsten Wittrock

[nebyah]

I can see how that might be a problem.
(untested code)

Code: Select all

foo()
{
 oldname=$1
 newname=$2
 cachedir=`echo $oldname | tr -d "/:"`
 cachefile=`basename $oldname`
 mkdir -p ../cache/$cachedir \
 && doWget $oldname -P ../cache/$cachedir/ \
 && cp -a ../cache/$cachedir/$cachefile  $newname
}


Then anywhere where you're currently using doWget $oldname -O /path/to/$newname

Code: Select all

foo $oldname ../path/to/$newname


[WSUSUpdateAdmin]

Hi nebyah,

I'ld like to have it the way the Windows part (DownloadUpdates.cmd) does it:

1. If exist <path>/<target name> rename <path>/<target name> to <path>/<URL file name>
2. doWget <URL> -P <path>
3. Rename <path>/<URL file name> to <path>/<target name>

This avoids both
- the warnings about wget -N -O parameter combination and
- the re-downloading of already existing files.

Do you have a Linux solution for that?

Thanks & regards
Torsten Wittrock

[harry]

How about something like this:

Code: Select all

...
if [ "$msse" == "1" ]; then
  echo "Downloading MSSE files..."
  if echo $sys | grep x64 > /dev/null 2>&1; then
    mssestring=`cat ../temp/StaticUrls-msse-x64-glb.txt | grep ,`
    arr=$(echo $mssestring | tr " " "\n")
    for x in $arr
      do
        oldname=`echo $x | awk -F"," '{print $1}'`
        newname=`echo $x | awk -F"," '{print $2}'`
        tmpname=`basename $oldname`
        mkdir -p ../client/msse/x64-glb
        if [ -f "../client/msse/x64-glb/$newname" ]; then
          mv -f "../client/msse/x64-glb/$newname" "../client/msse/x64-glb/$tmpname"
        fi
        doWget $oldname -P ../client/msse/x64-glb
        if [ -n $newname ]; then
          mv -f "../client/msse/x64-glb/$tmpname" "../client/msse/x64-glb/$newname"
        fi
      done
  else
    mssestring=`cat ../temp/StaticUrls-msse-x86-glb.txt | grep ,`
    arr=$(echo $mssestring | tr " " "\n")
    for x in $arr
      do
        oldname=`echo $x | awk -F"," '{print $1}'`
        newname=`echo $x | awk -F"," '{print $2}'`
        tmpname=`basename $oldname`
        mkdir -p ../client/msse/x86-glb
        if [ -f "../client/msse/x86-glb/$newname" ]; then
          mv -f "../client/msse/x86-glb/$newname" "../client/msse/x86-glb/$tmpname"
        fi
        doWget $oldname -P ../client/msse/x86-glb
        if [ -n $newname ]; then
          mv -f "../client/msse/x86-glb/$tmpname" "../client/msse/x86-glb/$newname"
        fi
      done
  fi
fi
...

or as a fucntion:

Code: Select all

...
doWget_with_rename()
{
  local old_new=$1
  local loadtopath=$2
  local oldname=`echo $old_new | awk -F"," '{print $1}'`
  local newname=`echo $old_new | awk -F"," '{print $2}'`
  local tmpname=`basename $oldname`
  mkdir -p $loadtopath
  if [ -f "$loadtopath/$newname" ]; then
    mv -f "$loadtopath/$newname" "$loadtopath/$tmpname"
  fi
  doWget $oldname -P $loadtopath
  local errWget=$?
  if [ -n $newname ]; then
    mv -f "$loadtopath/$tmpname" "$loadtopath/$newname"
  fi
  return $errWget
}
...
if [ "$msse" == "1" ]; then
  echo "Downloading MSSE files..."
  if echo $sys | grep x64 > /dev/null 2>&1; then
    mssestring=`cat ../temp/StaticUrls-msse-x64-glb.txt | grep ,`
    arr=$(echo $mssestring | tr " " "\n")
    for x in $arr
      do
        doWget_with_rename $x ../client/msse/x64-glb
      done
  else
    mssestring=`cat ../temp/StaticUrls-msse-x86-glb.txt | grep ,`
    arr=$(echo $mssestring | tr " " "\n")
    for x in $arr
      do
        doWget_with_rename $x ../client/msse/x86-glb
      done
  fi
fi
...

[nebyah]

I ought to be less lazy, and test this stuff, but...

Code: Select all

foo()
{
oldname=$1  #This would be the full download URL.
newname=$2 #This would be the full path we are saving to.
cachefile=`basename $oldname` #Filename part of URL.

#delete temp file if it exists. 
#We had better do this, in case $newname does not exist and ../temp/$cachefile does exist, but represents a different $oldname.
#This situation should not occur, but could cause problems if it does occur. Therefore, to be safe, we delete first.
rm -f  "../temp/$cachefile"

 # copy existing.  mv would be quicker, but this is safer - if dWget fails, current version will still exist.
#  -a copies attributes such as modified time.
cp -a "$newname" "../temp/$cachefile"

doWget $oldname -P ../temp/ && mv ../temp/$cachefile  $newname
rm -f  "../temp/$cachefile"   #Cleanup if doWget fails.
}

# later...
foo $oldname ../client/cpp/$newname


[WSUSUpdateAdmin]

Hi and thank you all!
Something like this ( http://trac.wsusoffline.net/browser/trunk (r326))?

RTW

[Spider-Mann]

In der "DownloadUpdates.cmd" wird das Problem folgendermaßen gelöst:

Code: Select all

565    echo Downloading/validating installation files for C++ Runtime Libraries...
566    for %%i in (x64 x86) do (
567      for /F "tokens=1,2 delims=," %%j in (..\static\StaticDownloadLinks-cpp-%%i-glb.txt) do (
568        if "%%k" NEQ "" (
569          if exist ..\client\cpp\%%k (
570            echo Renaming file ..\client\cpp\%%k to %%~nxj...
571            ren ..\client\cpp\%%k %%~nxj
572            echo %DATE% %TIME% - Info: Renamed file ..\client\cpp\%%k to %%~nxj >>%DOWNLOAD_LOGFILE%
573          )
574        )
575        %WGET_PATH% -N -P ..\client\cpp %%j
576        if errorlevel 1 (
577          if exist ..\client\cpp\%%~nxj del ..\client\cpp\%%~nxj
578          echo Warning: Download of %%j failed.
579          echo %DATE% %TIME% - Warning: Download of %%j failed >>%DOWNLOAD_LOGFILE%
580        )
581        if "%%k" NEQ "" (
582          if exist ..\client\cpp\%%~nxj (
583            echo Renaming file ..\client\cpp\%%~nxj to %%k...
584            ren ..\client\cpp\%%~nxj %%k
585            echo %DATE% %TIME% - Info: Renamed file ..\client\cpp\%%~nxj to %%k >>%DOWNLOAD_LOGFILE%
586          )
587        )
588      )
589    )

und

Code: Select all

669    echo Downloading/validating Microsoft Security Essentials installation files...
670    copy /Y ..\static\StaticDownloadLinks-msse-%TARGET_ARCH%-glb.txt "%TEMP%\StaticDownloadLinks-msse-%TARGET_ARCH%-glb.txt" >nul
671    if exist ..\static\custom\StaticDownloadLinks-msse-%TARGET_ARCH%-glb.txt (
672      for /F %%i in (..\static\custom\StaticDownloadLinks-msse-%TARGET_ARCH%-glb.txt) do echo %%i>>"%TEMP%\StaticDownloadLinks-msse-%TARGET_ARCH%-glb.txt"
673    )
674    for /F "usebackq tokens=1,2 delims=," %%i in ("%TEMP%\StaticDownloadLinks-msse-%TARGET_ARCH%-glb.txt") do (
675      if "%%j" NEQ "" (
676        if exist ..\client\msse\%TARGET_ARCH%-glb\%%j (
677          echo Renaming file ..\client\msse\%TARGET_ARCH%-glb\%%j to %%~nxi...
678          ren ..\client\msse\%TARGET_ARCH%-glb\%%j %%~nxi
679          echo %DATE% %TIME% - Info: Renamed file ..\client\msse\%TARGET_ARCH%-glb\%%j to %%~nxi >>%DOWNLOAD_LOGFILE%
680        )
681      )
682      %WGET_PATH% -N -P ..\client\msse\%TARGET_ARCH%-glb %%i
683      if errorlevel 1 (
684        if exist ..\client\msse\%TARGET_ARCH%-glb\%%~nxi del ..\client\msse\%TARGET_ARCH%-glb\%%~nxi
685        echo Warning: Download of %%i failed.
686        echo %DATE% %TIME% - Warning: Download of %%i failed >>%DOWNLOAD_LOGFILE%
687      )
688      if "%%j" NEQ "" (
689        if exist ..\client\msse\%TARGET_ARCH%-glb\%%~nxi (
690          echo Renaming file ..\client\msse\%TARGET_ARCH%-glb\%%~nxi to %%j...
691          ren ..\client\msse\%TARGET_ARCH%-glb\%%~nxi %%j
692          echo %DATE% %TIME% - Info: Renamed file ..\client\msse\%TARGET_ARCH%-glb\%%~nxi to %%j >>%DOWNLOAD_LOGFILE%
693        )
694      )
695    )


[nebyah]

Hi and thank you all!
Something like this ( http://trac.wsusoffline.net/trac.fcgi/browser/trunk (r326))?

RTW

You could skip the tests and send errors (eg file not found) to /dev/null:

Code: Select all

        mv -f "../client/cpp/$newname" "../client/cpp/$tmpname" 2>/dev/null
       doWget $oldname -P ../client/cpp
        mv -f "../client/cpp/$tmpname" "../client/cpp/$newname" 2>/dev/null


But is this code safe?
If wget fails half way through the download, will you be replacing an older but functional version of $newname with a corrupted version? Or does wget only replace the file after download is completed successfully?

[Spider-Mann]

In DownloadUpdates.sh Revision 326 steht noch folgendes:

Code: Select all

Line 1234 -> doWget $oldname -O ../client/cpp/$newname
Line 1243 -> doWget $oldname -O ../client/cpp/$newname
Line 1263 -> doWget $oldname -O ../client/msse/x64-glb/$newname
Line 1272 -> doWget $oldname -O ../client/msse/x86-glb/$newname


und in den folgenden Zeilen stimmt auch etwas nicht:

Code: Select all

Line 1167 -> doWget -i ../temp/StaticUrls-wddefs-x64-glb.txt -P ../client/wddefs/x64-glb
Line 1169 -> doWget -i ../temp/StaticUrls-wddefs-x86-glb.txt -P ../client/wddefs/x86-glb

Line 1286 -> doWget -i ../temp/StaticUrls-wddefs-x64-glb.txt -P ../client/wddefs
Line 1288 -> doWget -i ../temp/StaticUrls-wddefs-x86-glb.txt -P ../client/wddefs


entweder die ersten oder die zweiten beiden Zeilen sind fehlerhaft.