Page 1 of 1

Suggested fix (workaround) for

PostPosted: 29.08.2017, 20:29
by binhex

I've had some problems to get things work on Ubuntu 16.04, so I decided to post here the fix (workaround) that worked for me.

I've tried to download updates for Windows XP using command:
Code: Select all
bash wxp rus /makeiso /dotnet /excludesp  /msse /wddefs

The command itself didn't fail but some parts of download script did:
Code: Select all
grep: memory exhausted

The fix (workaround) was to split pattern files for grep into smaller chunks with the script named "grep-F-i-v-f":
Code: Select all

# replacement for grep -F -i -v -f $1 $2

echo >&2 DEBUG: $0 $*

split $1 $1-
cp $2 $2.tmp

for chunk in $1-??
    grep -F -i -v -f $chunk $2.tmp > $2.tmp-new
    mv $2.tmp-new $2.tmp

cat $2.tmp

rm $1-??
rm $2.tmp

Download script was modified accordingly with command:
Code: Select all
sed 's/grep -F -i -v -f/.\/grep-F-i-v-f/' -i

Also, download script was trying to use "md5deep" which is seem obsolete, so I had to make a link with such name:
Code: Select all
ln -s /usr/bin/hashdeep ./md5deep
export PATH=$PATH:.

Hope it helps someone else too.

Re: Suggested fix (workaround) for

PostPosted: 29.08.2017, 22:26
by Dalai
How much RAM does the system have you tested on, and which architecture?


Re: Suggested fix (workaround) for

PostPosted: 30.08.2017, 18:10
by binhex
The system has 4GB RAM and 2GB swap. Almost all memory was used by single grep process just before it terminated.

Re: Suggested fix (workaround) for

PostPosted: 30.08.2017, 20:00
by binhex
Also, it seems that fix for version 9.2.2
Code: Select all
 - Fix: Download part erroneously excluded Security Updates kb958644, kb2378111, kb2478971 and kb2712808

was applied only to DownloadUpdates.cmd.

Looks like downloads "ExcludeList-superseded-exclude.txt" but never actually uses it.

Again, following workaround resolved the issue (MBSA audit finally passed on target machine):
Code: Select all
@@ -634,15 +634,17 @@
 test -f ../temp/Urls-${OS_sys}-${lang}.txt && \
   cp ../temp/Urls-${OS_sys}-${lang}.txt ../temp/tmpUrls-${OS_sys}-${lang}.txt
-> ../temp/tmpExcludeList-${sys}.txt
+> ../temp/tmpExcludeList-${sys}-all.txt
 for Pfad in exclude exclude/custom; do
 test -f ../$Pfad/ExcludeList-${OS_sys}.txt && \
-    cat ../$Pfad/ExcludeList-${OS_sys}.txt >> ../temp/tmpExcludeList-${sys}.txt
+    cat ../$Pfad/ExcludeList-${OS_sys}.txt >> ../temp/tmpExcludeList-${sys}-all.txt
 test -f ../exclude/ExcludeList-superseded.txt && \
-    cat ../exclude/ExcludeList-superseded.txt >> ../temp/tmpExcludeList-${sys}.txt
+    cat ../exclude/ExcludeList-superseded.txt >> ../temp/tmpExcludeList-${sys}-all.txt
+test -f ../exclude/ExcludeList-superseded-exclude.txt && \
+    ./grep-F-i-v-f ../exclude/ExcludeList-superseded-exclude.txt ../temp/tmpExcludeList-${sys}-all.txt > ../temp/tmpExcludeList-${sys}.txt
 test -f ../temp/tmpUrls-${OS_sys}-${lang}.txt && \
     ./grep-F-i-v-f ../temp/tmpExcludeList-${sys}.txt ../temp/tmpUrls-${OS_sys}-${lang}.txt > ../temp/ValidUrls-${sys}-${lang}.txt

Hope it's helpful.

Re: Suggested fix (workaround) for

PostPosted: 31.08.2017, 10:00
by hbuhrmester
The calculation of superseded and dynamic updates will be much faster and use less RAM, if you set the locale to "C". Then strings are sorted and compared by their byte order.

The "natural" sorting by the user's locale is much slower: it will break long strings into smaller chunks, filter out punctuation marks and compare the remaining tokens.

So you could run the command:

Code: Select all
export LC_ALL=C

in the terminal, before calling the download script, or insert it at the top of the script


This environment variable was inserted into later versions of the script, since WSUS Offline Update, version 9.8: viewtopic.php?f=9&t=4908

Re: Suggested fix (workaround) for

PostPosted: 31.08.2017, 18:19
by binhex
Thanks for the great tip about changing locale! It did the trick without aforementioned split workaround. I guess it should also be applied to 9.2.x branch as it is still used for updating XP to latest possible patch level.