Suggested fix (workaround) for DownloadUpdates.sh

Suggested fix (workaround) for DownloadUpdates.sh

Postby binhex » 29.08.2017, 20:29

Hi!

I've had some problems to get things work on Ubuntu 16.04, so I decided to post here the fix (workaround) that worked for me.

I've tried to download updates for Windows XP using command:
Code: Select all
bash DownloadUpdates.sh wxp rus /makeiso /dotnet /excludesp  /msse /wddefs


The command itself didn't fail but some parts of download script did:
Code: Select all
grep: memory exhausted


The fix (workaround) was to split pattern files for grep into smaller chunks with the script named "grep-F-i-v-f":
Code: Select all
#!/bin/bash

# replacement for grep -F -i -v -f $1 $2

echo >&2 DEBUG: $0 $*

split $1 $1-
cp $2 $2.tmp

for chunk in $1-??
do
    grep -F -i -v -f $chunk $2.tmp > $2.tmp-new
    mv $2.tmp-new $2.tmp
done

cat $2.tmp

rm $1-??
rm $2.tmp


Download script was modified accordingly with command:
Code: Select all
sed 's/grep -F -i -v -f/.\/grep-F-i-v-f/' -i DownloadUpdates.sh


Also, download script was trying to use "md5deep" which is seem obsolete, so I had to make a link with such name:
Code: Select all
ln -s /usr/bin/hashdeep ./md5deep
export PATH=$PATH:.


Hope it helps someone else too.
binhex
 

Re: Suggested fix (workaround) for DownloadUpdates.sh

Postby Dalai » 29.08.2017, 22:26

How much RAM does the system have you tested on, and which architecture?

Regards
Dalai
Dalai
 
Posts: 1041
Joined: 12.07.2016, 21:00

Re: Suggested fix (workaround) for DownloadUpdates.sh

Postby binhex » 30.08.2017, 18:10

The system has 4GB RAM and 2GB swap. Almost all memory was used by single grep process just before it terminated.
binhex
 

Re: Suggested fix (workaround) for DownloadUpdates.sh

Postby binhex » 30.08.2017, 20:00

Also, it seems that fix for version 9.2.2
Code: Select all
 - Fix: Download part erroneously excluded Security Updates kb958644, kb2378111, kb2478971 and kb2712808

was applied only to DownloadUpdates.cmd.

Looks like DownloadUpdates.sh downloads "ExcludeList-superseded-exclude.txt" but never actually uses it.

Again, following workaround resolved the issue (MBSA audit finally passed on target machine):
Code: Select all
@@ -634,15 +634,17 @@
 test -f ../temp/Urls-${OS_sys}-${lang}.txt && \
   cp ../temp/Urls-${OS_sys}-${lang}.txt ../temp/tmpUrls-${OS_sys}-${lang}.txt
 
-> ../temp/tmpExcludeList-${sys}.txt
+> ../temp/tmpExcludeList-${sys}-all.txt
 
 for Pfad in exclude exclude/custom; do
 test -f ../$Pfad/ExcludeList-${OS_sys}.txt && \
-    cat ../$Pfad/ExcludeList-${OS_sys}.txt >> ../temp/tmpExcludeList-${sys}.txt
+    cat ../$Pfad/ExcludeList-${OS_sys}.txt >> ../temp/tmpExcludeList-${sys}-all.txt
   done
 
 test -f ../exclude/ExcludeList-superseded.txt && \
-    cat ../exclude/ExcludeList-superseded.txt >> ../temp/tmpExcludeList-${sys}.txt
+    cat ../exclude/ExcludeList-superseded.txt >> ../temp/tmpExcludeList-${sys}-all.txt
+test -f ../exclude/ExcludeList-superseded-exclude.txt && \
+    ./grep-F-i-v-f ../exclude/ExcludeList-superseded-exclude.txt ../temp/tmpExcludeList-${sys}-all.txt > ../temp/tmpExcludeList-${sys}.txt
 test -f ../temp/tmpUrls-${OS_sys}-${lang}.txt && \
     ./grep-F-i-v-f ../temp/tmpExcludeList-${sys}.txt ../temp/tmpUrls-${OS_sys}-${lang}.txt > ../temp/ValidUrls-${sys}-${lang}.txt
 


Hope it's helpful.
binhex
 

Re: Suggested fix (workaround) for DownloadUpdates.sh

Postby hbuhrmester » 31.08.2017, 10:00

The calculation of superseded and dynamic updates will be much faster and use less RAM, if you set the locale to "C". Then strings are sorted and compared by their byte order.

The "natural" sorting by the user's locale is much slower: it will break long strings into smaller chunks, filter out punctuation marks and compare the remaining tokens.

So you could run the command:

Code: Select all
export LC_ALL=C


in the terminal, before calling the download script, or insert it at the top of the script DownloadUpdates.sh.

PS:

This environment variable was inserted into later versions of the script DownloadUpdates.sh, since WSUS Offline Update, version 9.8: viewtopic.php?f=9&t=4908
hbuhrmester
 
Posts: 525
Joined: 11.10.2013, 20:59

Re: Suggested fix (workaround) for DownloadUpdates.sh

Postby binhex » 31.08.2017, 18:19

Thanks for the great tip about changing locale! It did the trick without aforementioned split workaround. I guess it should also be applied to 9.2.x branch as it is still used for updating XP to latest possible patch level.
binhex
 


Return to Linux

Who is online

Users browsing this forum: No registered users and 24 guests