DownloadUpdates.sh - Timestamping and -O

DownloadUpdates.sh - Timestamping and -O

Postby nebyah » 07.12.2011, 04:42

Code: Select all
wget -nv -N --timeout=120 http://mse.dlservice.microsoft.com/download/0/2/C/02C8AB73-0774-4975-826F-9E8A0FD7F65D/dede/x86/mseinstall.exe -O ../client/msse/x86-glb/mseinstall-x86-deu.exe
WARNING: timestamping does nothing in combination with -O. See the manual

When -O is used, wget ignores -N (timestamping). This causes some files (vcredist*, mseinstall*, possibly others) to be downloaded several times, each time wsusoffline is run.

Might I suggest something like:
Code: Select all
mkdir -p ../cache/mseinstall-x68-deu.exe \
&& wget -nv -N --timeout=120 http://mse.dlservice.microsoft.com/download/0/2/C/02C8AB73-0774-4975-826F-9E8A0FD7F65D/dede/x86/mseinstall.exe -P ../cache/mseinstall-x86-deu.exe/ \
&& cp -a ../cache/mseinstall-x86-deu.exe/mseinstall.exe  ../client/msse/x86-glb/mseinstall-x86-deu.exe
nebyah
 
Posts: 10
Joined: 18.02.2010, 22:29

Re: DownloadUpdates.sh - Timestamping and -O

Postby WSUSUpdateAdmin » 12.12.2011, 12:26

Hi!

Yes, thanks, you're right, but the modification you suggest can't be done in that simple way, because all downloads are handled by the subroutine doWget(), so one has to modify this sub depending on required renaming.

Regards
Torsten Wittrock
WSUSUpdateAdmin
Administrator
 
Posts: 2217
Joined: 07.07.2009, 14:38

Re: DownloadUpdates.sh - Timestamping and -O

Postby nebyah » 13.12.2011, 02:53

I can see how that might be a problem.
(untested code)
Code: Select all
foo()
{
 oldname=$1
 newname=$2
 cachedir=`echo $oldname | tr -d "/:"`
 cachefile=`basename $oldname`
 mkdir -p ../cache/$cachedir \
 && doWget $oldname -P ../cache/$cachedir/ \
 && cp -a ../cache/$cachedir/$cachefile  $newname
}

Then anywhere where you're currently using doWget $oldname -O /path/to/$newname
Code: Select all
foo $oldname ../path/to/$newname
nebyah
 
Posts: 10
Joined: 18.02.2010, 22:29

Re: DownloadUpdates.sh - Timestamping and -O

Postby WSUSUpdateAdmin » 16.12.2011, 13:31

Hi nebyah,

I'ld like to have it the way the Windows part (DownloadUpdates.cmd) does it:

1. If exist <path>/<target name> rename <path>/<target name> to <path>/<URL file name>
2. doWget <URL> -P <path>
3. Rename <path>/<URL file name> to <path>/<target name>

This avoids both
- the warnings about wget -N -O parameter combination and
- the re-downloading of already existing files.

Do you have a Linux solution for that?

Thanks & regards
Torsten Wittrock
WSUSUpdateAdmin
Administrator
 
Posts: 2217
Joined: 07.07.2009, 14:38

Re: DownloadUpdates.sh - Timestamping and -O

Postby harry » 16.12.2011, 14:53

How about something like this:
Code: Select all
...
if [ "$msse" == "1" ]; then
  echo "Downloading MSSE files..."
  if echo $sys | grep x64 > /dev/null 2>&1; then
    mssestring=`cat ../temp/StaticUrls-msse-x64-glb.txt | grep ,`
    arr=$(echo $mssestring | tr " " "\n")
    for x in $arr
      do
        oldname=`echo $x | awk -F"," '{print $1}'`
        newname=`echo $x | awk -F"," '{print $2}'`
        tmpname=`basename $oldname`
        mkdir -p ../client/msse/x64-glb
        if [ -f "../client/msse/x64-glb/$newname" ]; then
          mv -f "../client/msse/x64-glb/$newname" "../client/msse/x64-glb/$tmpname"
        fi
        doWget $oldname -P ../client/msse/x64-glb
        if [ -n $newname ]; then
          mv -f "../client/msse/x64-glb/$tmpname" "../client/msse/x64-glb/$newname"
        fi
      done
  else
    mssestring=`cat ../temp/StaticUrls-msse-x86-glb.txt | grep ,`
    arr=$(echo $mssestring | tr " " "\n")
    for x in $arr
      do
        oldname=`echo $x | awk -F"," '{print $1}'`
        newname=`echo $x | awk -F"," '{print $2}'`
        tmpname=`basename $oldname`
        mkdir -p ../client/msse/x86-glb
        if [ -f "../client/msse/x86-glb/$newname" ]; then
          mv -f "../client/msse/x86-glb/$newname" "../client/msse/x86-glb/$tmpname"
        fi
        doWget $oldname -P ../client/msse/x86-glb
        if [ -n $newname ]; then
          mv -f "../client/msse/x86-glb/$tmpname" "../client/msse/x86-glb/$newname"
        fi
      done
  fi
fi
...

or as a fucntion:
Code: Select all
...
doWget_with_rename()
{
  local old_new=$1
  local loadtopath=$2
  local oldname=`echo $old_new | awk -F"," '{print $1}'`
  local newname=`echo $old_new | awk -F"," '{print $2}'`
  local tmpname=`basename $oldname`
  mkdir -p $loadtopath
  if [ -f "$loadtopath/$newname" ]; then
    mv -f "$loadtopath/$newname" "$loadtopath/$tmpname"
  fi
  doWget $oldname -P $loadtopath
  local errWget=$?
  if [ -n $newname ]; then
    mv -f "$loadtopath/$tmpname" "$loadtopath/$newname"
  fi
  return $errWget
}
...
if [ "$msse" == "1" ]; then
  echo "Downloading MSSE files..."
  if echo $sys | grep x64 > /dev/null 2>&1; then
    mssestring=`cat ../temp/StaticUrls-msse-x64-glb.txt | grep ,`
    arr=$(echo $mssestring | tr " " "\n")
    for x in $arr
      do
        doWget_with_rename $x ../client/msse/x64-glb
      done
  else
    mssestring=`cat ../temp/StaticUrls-msse-x86-glb.txt | grep ,`
    arr=$(echo $mssestring | tr " " "\n")
    for x in $arr
      do
        doWget_with_rename $x ../client/msse/x86-glb
      done
  fi
fi
...
harry
 
Posts: 707
Joined: 29.10.2009, 17:02

Re: DownloadUpdates.sh - Timestamping and -O

Postby nebyah » 19.12.2011, 07:02

I ought to be less lazy, and test this stuff, but...
Code: Select all
foo()
{
oldname=$1  #This would be the full download URL.
newname=$2 #This would be the full path we are saving to.
cachefile=`basename $oldname` #Filename part of URL.

#delete temp file if it exists. 
#We had better do this, in case $newname does not exist and ../temp/$cachefile does exist, but represents a different $oldname.
#This situation should not occur, but could cause problems if it does occur. Therefore, to be safe, we delete first.
rm -f  "../temp/$cachefile"

 # copy existing.  mv would be quicker, but this is safer - if dWget fails, current version will still exist.
#  -a copies attributes such as modified time.
cp -a "$newname" "../temp/$cachefile"

doWget $oldname -P ../temp/ && mv ../temp/$cachefile  $newname
rm -f  "../temp/$cachefile"   #Cleanup if doWget fails.
}

# later...
foo $oldname ../client/cpp/$newname
nebyah
 
Posts: 10
Joined: 18.02.2010, 22:29

Re: DownloadUpdates.sh - Timestamping and -O

Postby WSUSUpdateAdmin » 19.12.2011, 12:41

Hi and thank you all! :)
Something like this ( :arrow: http://trac.wsusoffline.net/browser/trunk (r326))?

RTW
Last edited by aker on 28.07.2013, 18:42, edited 1 time in total.
Reason: Trac Link aktualisiert
WSUSUpdateAdmin
Administrator
 
Posts: 2217
Joined: 07.07.2009, 14:38

Re: DownloadUpdates.sh - Timestamping and -O

Postby Spider-Mann » 19.12.2011, 15:48

In der "DownloadUpdates.cmd" wird das Problem folgendermaßen gelöst:

Code: Select all
565    echo Downloading/validating installation files for C++ Runtime Libraries...
566    for %%i in (x64 x86) do (
567      for /F "tokens=1,2 delims=," %%j in (..\static\StaticDownloadLinks-cpp-%%i-glb.txt) do (
568        if "%%k" NEQ "" (
569          if exist ..\client\cpp\%%k (
570            echo Renaming file ..\client\cpp\%%k to %%~nxj...
571            ren ..\client\cpp\%%k %%~nxj
572            echo %DATE% %TIME% - Info: Renamed file ..\client\cpp\%%k to %%~nxj >>%DOWNLOAD_LOGFILE%
573          )
574        )
575        %WGET_PATH% -N -P ..\client\cpp %%j
576        if errorlevel 1 (
577          if exist ..\client\cpp\%%~nxj del ..\client\cpp\%%~nxj
578          echo Warning: Download of %%j failed.
579          echo %DATE% %TIME% - Warning: Download of %%j failed >>%DOWNLOAD_LOGFILE%
580        )
581        if "%%k" NEQ "" (
582          if exist ..\client\cpp\%%~nxj (
583            echo Renaming file ..\client\cpp\%%~nxj to %%k...
584            ren ..\client\cpp\%%~nxj %%k
585            echo %DATE% %TIME% - Info: Renamed file ..\client\cpp\%%~nxj to %%k >>%DOWNLOAD_LOGFILE%
586          )
587        )
588      )
589    )

und
Code: Select all
669    echo Downloading/validating Microsoft Security Essentials installation files...
670    copy /Y ..\static\StaticDownloadLinks-msse-%TARGET_ARCH%-glb.txt "%TEMP%\StaticDownloadLinks-msse-%TARGET_ARCH%-glb.txt" >nul
671    if exist ..\static\custom\StaticDownloadLinks-msse-%TARGET_ARCH%-glb.txt (
672      for /F %%i in (..\static\custom\StaticDownloadLinks-msse-%TARGET_ARCH%-glb.txt) do echo %%i>>"%TEMP%\StaticDownloadLinks-msse-%TARGET_ARCH%-glb.txt"
673    )
674    for /F "usebackq tokens=1,2 delims=," %%i in ("%TEMP%\StaticDownloadLinks-msse-%TARGET_ARCH%-glb.txt") do (
675      if "%%j" NEQ "" (
676        if exist ..\client\msse\%TARGET_ARCH%-glb\%%j (
677          echo Renaming file ..\client\msse\%TARGET_ARCH%-glb\%%j to %%~nxi...
678          ren ..\client\msse\%TARGET_ARCH%-glb\%%j %%~nxi
679          echo %DATE% %TIME% - Info: Renamed file ..\client\msse\%TARGET_ARCH%-glb\%%j to %%~nxi >>%DOWNLOAD_LOGFILE%
680        )
681      )
682      %WGET_PATH% -N -P ..\client\msse\%TARGET_ARCH%-glb %%i
683      if errorlevel 1 (
684        if exist ..\client\msse\%TARGET_ARCH%-glb\%%~nxi del ..\client\msse\%TARGET_ARCH%-glb\%%~nxi
685        echo Warning: Download of %%i failed.
686        echo %DATE% %TIME% - Warning: Download of %%i failed >>%DOWNLOAD_LOGFILE%
687      )
688      if "%%j" NEQ "" (
689        if exist ..\client\msse\%TARGET_ARCH%-glb\%%~nxi (
690          echo Renaming file ..\client\msse\%TARGET_ARCH%-glb\%%~nxi to %%j...
691          ren ..\client\msse\%TARGET_ARCH%-glb\%%~nxi %%j
692          echo %DATE% %TIME% - Info: Renamed file ..\client\msse\%TARGET_ARCH%-glb\%%~nxi to %%j >>%DOWNLOAD_LOGFILE%
693        )
694      )
695    )
Spider-Mann
 
Posts: 19
Joined: 31.10.2011, 21:10

Re: DownloadUpdates.sh - Timestamping and -O

Postby nebyah » 19.12.2011, 21:56

WSUSUpdateAdmin wrote:Hi and thank you all! :)
Something like this ( :arrow: http://trac.wsusoffline.net/trac.fcgi/browser/trunk (r326))?

RTW


You could skip the tests and send errors (eg file not found) to /dev/null:
Code: Select all
        mv -f "../client/cpp/$newname" "../client/cpp/$tmpname" 2>/dev/null
       doWget $oldname -P ../client/cpp
        mv -f "../client/cpp/$tmpname" "../client/cpp/$newname" 2>/dev/null


But is this code safe?
If wget fails half way through the download, will you be replacing an older but functional version of $newname with a corrupted version? Or does wget only replace the file after download is completed successfully?
nebyah
 
Posts: 10
Joined: 18.02.2010, 22:29

Re: DownloadUpdates.sh - Timestamping and -O

Postby Spider-Mann » 20.12.2011, 07:34

In DownloadUpdates.sh Revision 326 steht noch folgendes:
Code: Select all
Line 1234 -> doWget $oldname -O ../client/cpp/$newname
Line 1243 -> doWget $oldname -O ../client/cpp/$newname
Line 1263 -> doWget $oldname -O ../client/msse/x64-glb/$newname
Line 1272 -> doWget $oldname -O ../client/msse/x86-glb/$newname


und in den folgenden Zeilen stimmt auch etwas nicht:
Code: Select all
Line 1167 -> doWget -i ../temp/StaticUrls-wddefs-x64-glb.txt -P ../client/wddefs/x64-glb
Line 1169 -> doWget -i ../temp/StaticUrls-wddefs-x86-glb.txt -P ../client/wddefs/x86-glb

Line 1286 -> doWget -i ../temp/StaticUrls-wddefs-x64-glb.txt -P ../client/wddefs
Line 1288 -> doWget -i ../temp/StaticUrls-wddefs-x86-glb.txt -P ../client/wddefs
entweder die ersten oder die zweiten beiden Zeilen sind fehlerhaft.
Spider-Mann
 
Posts: 19
Joined: 31.10.2011, 21:10

Next

Return to Linux

Who is online

Users browsing this forum: No registered users and 2 guests