Page 1 of 1

Suggested fix (workaround) for DownloadUpdates.sh

PostPosted: 29.08.2017, 20:29
by binhex
Hi!

I've had some problems to get things work on Ubuntu 16.04, so I decided to post here the fix (workaround) that worked for me.

I've tried to download updates for Windows XP using command:
Code: Select all
bash DownloadUpdates.sh wxp rus /makeiso /dotnet /excludesp  /msse /wddefs


The command itself didn't fail but some parts of download script did:
Code: Select all
grep: memory exhausted


The fix (workaround) was to split pattern files for grep into smaller chunks with the script named "grep-F-i-v-f":
Code: Select all
#!/bin/bash

# replacement for grep -F -i -v -f $1 $2

echo >&2 DEBUG: $0 $*

split $1 $1-
cp $2 $2.tmp

for chunk in $1-??
do
    grep -F -i -v -f $chunk $2.tmp > $2.tmp-new
    mv $2.tmp-new $2.tmp
done

cat $2.tmp

rm $1-??
rm $2.tmp


Download script was modified accordingly with command:
Code: Select all
sed 's/grep -F -i -v -f/.\/grep-F-i-v-f/' -i DownloadUpdates.sh


Also, download script was trying to use "md5deep" which is seem obsolete, so I had to make a link with such name:
Code: Select all
ln -s /usr/bin/hashdeep ./md5deep
export PATH=$PATH:.


Hope it helps someone else too.

Re: Suggested fix (workaround) for DownloadUpdates.sh

PostPosted: 29.08.2017, 22:26
by Dalai
How much RAM does the system have you tested on, and which architecture?

Regards
Dalai

Re: Suggested fix (workaround) for DownloadUpdates.sh

PostPosted: 30.08.2017, 18:10
by binhex
The system has 4GB RAM and 2GB swap. Almost all memory was used by single grep process just before it terminated.

Re: Suggested fix (workaround) for DownloadUpdates.sh

PostPosted: 30.08.2017, 20:00
by binhex
Also, it seems that fix for version 9.2.2
Code: Select all
 - Fix: Download part erroneously excluded Security Updates kb958644, kb2378111, kb2478971 and kb2712808

was applied only to DownloadUpdates.cmd.

Looks like DownloadUpdates.sh downloads "ExcludeList-superseded-exclude.txt" but never actually uses it.

Again, following workaround resolved the issue (MBSA audit finally passed on target machine):
Code: Select all
@@ -634,15 +634,17 @@
 test -f ../temp/Urls-${OS_sys}-${lang}.txt && \
   cp ../temp/Urls-${OS_sys}-${lang}.txt ../temp/tmpUrls-${OS_sys}-${lang}.txt
 
-> ../temp/tmpExcludeList-${sys}.txt
+> ../temp/tmpExcludeList-${sys}-all.txt
 
 for Pfad in exclude exclude/custom; do
 test -f ../$Pfad/ExcludeList-${OS_sys}.txt && \
-    cat ../$Pfad/ExcludeList-${OS_sys}.txt >> ../temp/tmpExcludeList-${sys}.txt
+    cat ../$Pfad/ExcludeList-${OS_sys}.txt >> ../temp/tmpExcludeList-${sys}-all.txt
   done
 
 test -f ../exclude/ExcludeList-superseded.txt && \
-    cat ../exclude/ExcludeList-superseded.txt >> ../temp/tmpExcludeList-${sys}.txt
+    cat ../exclude/ExcludeList-superseded.txt >> ../temp/tmpExcludeList-${sys}-all.txt
+test -f ../exclude/ExcludeList-superseded-exclude.txt && \
+    ./grep-F-i-v-f ../exclude/ExcludeList-superseded-exclude.txt ../temp/tmpExcludeList-${sys}-all.txt > ../temp/tmpExcludeList-${sys}.txt
 test -f ../temp/tmpUrls-${OS_sys}-${lang}.txt && \
     ./grep-F-i-v-f ../temp/tmpExcludeList-${sys}.txt ../temp/tmpUrls-${OS_sys}-${lang}.txt > ../temp/ValidUrls-${sys}-${lang}.txt
 


Hope it's helpful.

Re: Suggested fix (workaround) for DownloadUpdates.sh

PostPosted: 31.08.2017, 10:00
by hbuhrmester
The calculation of superseded and dynamic updates will be much faster and use less RAM, if you set the locale to "C". Then strings are sorted and compared by their byte order.

The "natural" sorting by the user's locale is much slower: it will break long strings into smaller chunks, filter out punctuation marks and compare the remaining tokens.

So you could run the command:

Code: Select all
export LC_ALL=C


in the terminal, before calling the download script, or insert it at the top of the script DownloadUpdates.sh.

PS:

This environment variable was inserted into later versions of the script DownloadUpdates.sh, since WSUS Offline Update, version 9.8: viewtopic.php?f=9&t=4908

Re: Suggested fix (workaround) for DownloadUpdates.sh

PostPosted: 31.08.2017, 18:19
by binhex
Thanks for the great tip about changing locale! It did the trick without aforementioned split workaround. I guess it should also be applied to 9.2.x branch as it is still used for updating XP to latest possible patch level.