CopyToTarget linux script - where to submit

CopyToTarget linux script - where to submit

Postby Valicek1 » 28.08.2017, 11:11

Hello,

as far as I have missed CopyToTarget CMD in linux version, I have rewritten batch script into shell variant - I have to make some nasty hacks to paths to make rsync copy as it was probbably intended. Now, after testing created packages on lots of configurations, I decided my code as mature to share with community.

In other projects, it's normal to submit git pull request, but I don't see anything similar to git, maybe some SVN stuff in Trac - but I'm not familiar with SVN.

Where I can submit (and how) my code?

Thanks for your answer and thanks for this amazing linux variant of wsusoffline.

Changelog:
Last edited by Valicek1 on 02.09.2017, 13:23, edited 1 time in total.
Valicek1
 

Re: CopyToTarget linux script - where to submit

Postby Dalai » 28.08.2017, 14:14

Valicek1 wrote:Where I can submit (and how) my code?

I suggest to either post it here (if it's not too long), or upload it somewhere (e.g. pastebin.com) and link to it here.

Regards
Dalai
Dalai
 
Posts: 1041
Joined: 12.07.2016, 21:00

Re: CopyToTarget linux script - where to submit

Postby Valicek1 » 28.08.2017, 14:40

Probbably, pastebin would be best place. https://pastebin.com/cdfi4YTR - it's here. Tried to make it compatible as close as possible.

Only different things from windows are:
- exit on error is explicit - when some of commands called by script fails, whole script fails too
- hardlinks - optional support for hardlinks to save some space.
Valicek1
 

Re: CopyToTarget linux script - where to submit

Postby hbuhrmester » 29.08.2017, 15:56

Actually, I already have a copy-to-target script. It is the script 70-synchronize-with-target.bash in the directory available-tasks. It can be enabled by moving it to the directory update-generator-tasks or by creating a symbolic link in that directory. It could also be sourced from the script get-all-updates.bash, after running the download jobs.

The script is very simple, though: It only defines a source and a target directory, and then uses rsync to synchronize the target with the source. By default, the source directory is the client directory, which will be copied as a whole to a large enough external drive.

A more elaborate script with more options, logging and error checking is certainly welcome. But, after browsing through the script, I have some comments and suggestions:


shellcheck complains

First of all, the utility shellcheck has numerous complaints:

Code: Select all
In copy-to-target.bash line 60:
readonly script_dir=$(dirname $(realpath $0))
                              ^-- SC2046: Quote this to prevent word splitting.
                                         ^-- SC2086: Double quote to prevent globbing and word splitting.


In copy-to-target.bash line 83:
    toPut="$(date "+%x %X") $@"
          ^-- SC2124: Assigning an array to a string! Assign as array, or use * instead of @ to concatenate.


In copy-to-target.bash line 84:
    echo $toPut
         ^-- SC2086: Double quote to prevent globbing and word splitting.


In copy-to-target.bash line 85:
    echo "$toPut" >> $script_dir/$logfile
                     ^-- SC2086: Double quote to prevent globbing and word splitting.


In copy-to-target.bash line 95:
    exit $rc
         ^-- SC2086: Double quote to prevent globbing and word splitting.


In copy-to-target.bash line 130:
log "CMD: $0 $@"
             ^-- SC2145: Argument mixes string and array. Use * or separate argument.


In copy-to-target.bash line 141:
if contains ${WinTargets[@]} $update
            ^-- SC2068: Double quote array expansions to avoid re-splitting elements.
                             ^-- SC2086: Double quote to prevent globbing and word splitting.


In copy-to-target.bash line 144:
    contains ${WinLangs[@]} $language || die 1 "Unknown Windows Language -> exiting"
             ^-- SC2068: Double quote array expansions to avoid re-splitting elements.
                            ^-- SC2086: Double quote to prevent globbing and word splitting.


In copy-to-target.bash line 147:
    if contains ${OfcTargets[@]} $update
                ^-- SC2068: Double quote array expansions to avoid re-splitting elements.
                                 ^-- SC2086: Double quote to prevent globbing and word splitting.


In copy-to-target.bash line 150:
        contains ${OfcLangs[@]} $language || die 1 "Unknown Office Language -> exiting"
                 ^-- SC2068: Double quote array expansions to avoid re-splitting elements.
                                ^-- SC2086: Double quote to prevent globbing and word splitting.


In copy-to-target.bash line 183:
            if echo $update | grep -E $WddefsTargets > /dev/null
                    ^-- SC2086: Double quote to prevent globbing and word splitting.


In copy-to-target.bash line 212:
cd $script_dir
   ^-- SC2086: Double quote to prevent globbing and word splitting.


In copy-to-target.bash line 229:
[ -f $excl_prefix-$update.txt ] && cp $excl_prefix-$update.txt $filter || cp  $excl_prefix-$update-x86.txt $filter
                  ^-- SC2086: Double quote to prevent globbing and word splitting.
                                ^-- SC2015: Note that A && B || C is not if-then-else. C may run when A is true.
                                                   ^-- SC2086: Double quote to prevent globbing and word splitting.
                                                               ^-- SC2086: Double quote to prevent globbing and word splitting.
                                                                                           ^-- SC2086: Double quote to prevent globbing and word splitting.
                                                                                                           ^-- SC2086: Double quote to prevent globbing and word splitting.


In copy-to-target.bash line 232:
[ -f $filter_dir/custom/ExcludeListUSB-$update.txt ] && cat $filter_dir/custom/ExcludeListUSB-$update.txt >> $filter
                                       ^-- SC2086: Double quote to prevent globbing and word splitting.
                                                                                              ^-- SC2086: Double quote to prevent globbing and word splitting.
                                                                                                             ^-- SC2086: Double quote to prevent globbing and word splitting.


In copy-to-target.bash line 234:
[ -f $filter_dir/custom/ExcludeListUSB-$update-x86.txt ] && cat $filter_dir/custom/ExcludeListUSB-$update-x86.txt >> $filter
                                       ^-- SC2086: Double quote to prevent globbing and word splitting.
                                                                                                  ^-- SC2086: Double quote to prevent globbing and word splitting.
                                                                                                                     ^-- SC2086: Double quote to prevent globbing and word splitting.


In copy-to-target.bash line 238:
for loc in ${locales[@]}
           ^-- SC2068: Double quote array expansions to avoid re-splitting elements.


In copy-to-target.bash line 241:
    [ ! "$language" == "$loc" ] && echo "$loc/" >> $filter
                                                   ^-- SC2086: Double quote to prevent globbing and word splitting.


In copy-to-target.bash line 245:
[ $opt_exclude_sp -eq 1 ] && cat $filter_dir/ExcludeList-SPs.txt >> $filter
                                                                    ^-- SC2086: Double quote to prevent globbing and word splitting.


In copy-to-target.bash line 246:
[ $opt_exclude_sw -eq 1 ] && cat $filter_dir/ExcludeList-software.txt >> $filter
                                                                         ^-- SC2086: Double quote to prevent globbing and word splitting.


In copy-to-target.bash line 247:
[ $opt_include_dotnet -eq 0 ] && cat $filter_dir/ExcludeListISO-dotnet.txt >> $filter
                                                                              ^-- SC2086: Double quote to prevent globbing and word splitting.


In copy-to-target.bash line 248:
[ $opt_include_msse -eq 0 ] && cat $filter_dir/ExcludeList-msse.txt >> $filter
                                                                       ^-- SC2086: Double quote to prevent globbing and word splitting.


In copy-to-target.bash line 249:
[ $opt_include_wddefs -eq 0 ] && cat $filter_dir/ExcludeList-wddefs.txt >> $filter
                                                                           ^-- SC2086: Double quote to prevent globbing and word splitting.


In copy-to-target.bash line 252:
cat $filter | sed 's/\\/\//g' | sed 's/\r$//' > $filter.tmp
    ^-- SC2086: Double quote to prevent globbing and word splitting.
    ^-- SC2002: Useless cat. Consider 'cmd < file | ..' or 'cmd file | ..' instead.
                                                ^-- SC2086: Double quote to prevent globbing and word splitting.


In copy-to-target.bash line 254:
cat /dev/null > $filter
                ^-- SC2086: Double quote to prevent globbing and word splitting.


In copy-to-target.bash line 256:
for line in $(cat $filter.tmp)
            ^-- SC2013: To read lines rather than words, pipe/redirect to a 'while read' loop.
                  ^-- SC2086: Double quote to prevent globbing and word splitting.


In copy-to-target.bash line 259:
    if echo $line | grep -E '.*/$'  > /dev/null
            ^-- SC2086: Double quote to prevent globbing and word splitting.


In copy-to-target.bash line 261:
        echo $line >> $filter
             ^-- SC2086: Double quote to prevent globbing and word splitting.
                      ^-- SC2086: Double quote to prevent globbing and word splitting.


In copy-to-target.bash line 263:
        printf '*%s*\n' $line >> $filter
                        ^-- SC2086: Double quote to prevent globbing and word splitting.
                                 ^-- SC2086: Double quote to prevent globbing and word splitting.


In copy-to-target.bash line 267:
[ -f $target_dir ] && rm -f $target_dir
     ^-- SC2086: Double quote to prevent globbing and word splitting.
                            ^-- SC2086: Double quote to prevent globbing and word splitting.


In copy-to-target.bash line 272:
    rm -Rf $target_dir
           ^-- SC2086: Double quote to prevent globbing and word splitting.


In copy-to-target.bash line 276:
mkdir -p $target_dir
         ^-- SC2086: Double quote to prevent globbing and word splitting.


In copy-to-target.bash line 286:
    tar=$(realpath $target_dir)
                   ^-- SC2086: Double quote to prevent globbing and word splitting.


In copy-to-target.bash line 287:
    rsync -a  --log-file=$log_dir/rsync.txt --exclude-from=$filter --link-dest=$src $src/ $tar
                                                           ^-- SC2086: Double quote to prevent globbing and word splitting.
                                                                               ^-- SC2086: Double quote to prevent globbing and word splitting.
                                                                                    ^-- SC2086: Double quote to prevent globbing and word splitting.
                                                                                          ^-- SC2086: Double quote to prevent globbing and word splitting.


In copy-to-target.bash line 289:
    rsync -a  --log-file=$log_dir/rsync.txt --exclude-from=$filter  . $target_dir
                                                           ^-- SC2086: Double quote to prevent globbing and word splitting.
                                                                      ^-- SC2086: Double quote to prevent globbing and word splitting.


In copy-to-target.bash line 298:
for f in $(find $target_dir -printf "%P\n")
         ^-- SC2044: For loops over find output are fragile. Use find -exec or a while read loop.
                ^-- SC2086: Double quote to prevent globbing and word splitting.


In copy-to-target.bash line 304:
    if [ -f $target_dir/$f ]
            ^-- SC2086: Double quote to prevent globbing and word splitting.
                        ^-- SC2086: Double quote to prevent globbing and word splitting.


In copy-to-target.bash line 307:
        if [ ! -f $f ]
                  ^-- SC2086: Double quote to prevent globbing and word splitting.


In copy-to-target.bash line 310:
            rm -f $target_dir/$f
                  ^-- SC2086: Double quote to prevent globbing and word splitting.
                              ^-- SC2086: Double quote to prevent globbing and word splitting.


In copy-to-target.bash line 314:
        if [ ! -d $f ]
                  ^-- SC2086: Double quote to prevent globbing and word splitting.


In copy-to-target.bash line 317:
            rm -rf $target_dir/$f
                   ^-- SC2115: Use "${var:?}" to ensure this never expands to / .
                   ^-- SC2086: Double quote to prevent globbing and word splitting.
                               ^-- SC2086: Double quote to prevent globbing and word splitting.


I try to solve most of them, with the exception, that I sometimes explicitly want word splitting of some variables. This could still be changed by implementing these variables as indexed arrays (which I may do someday).

For some reason, shellcheck doesn't like UTF-8 characters, if it is run within a terminal emulator. But shellcheck is also nicely integrated in Geany, and then it can be called as "Lint" from the Build menu.


Declaration of variables with "local" and a command substitution should be in separate lines.

I learned, that this is a common pitfall: http://mywiki.wooledge.org/BashPitfalls ... command.29

This example only mentions "local", but I assume, that the rule refers to "declare" and "readonly" as well.


The description of the function containsElement seems to be wrong.

Instead of:
Code: Select all
# Does Array contain element?
# Usage: containsElement "requested string" "${array[@]}"


it should be:
Code: Select all
# Does Array contain element?
# Usage: containsElement "${array[@]}" "requested string"


The function is actually used this way.


The script should check, if rsync is installed.

In Debian 9 Stretch (stable), rsync is an optional package, which is not installed by default. Then the script should check, if rsync is actually available.

This is also missing in my script, but it was supposed to be tested in the script 30-check-needed-applications.bash instead.


The directories wddefs and msse seem to be used wrong.

The directory wddefs contains virus definition files for the original Windows Defender. This was the built-in anti-virus software of Windows Vista and Windows 7. It was an optional installation for Windows XP. These virus definitions are not suitable for Windows 8 and later.

The built-in Defender of Window 8, 8.1 and 10 uses the same virus definitions as Microsoft Security Essentials. These Windows versions don't need the installation programs, but they do need the virus definitions from the directory msse.

Note, that the script DownloadUpdates.cmd uses the option /includewddefs in two different ways:
  • For Windows Vista and 7 (w60, w61) it includes the directory wddefs.
  • For Windows 8, 8.1 and 10 (w62, w63, w100) it includes the directory msse.

The meaning of /includewddefs is changed internally:

DownloadUpdates.cmd, lines 152 - 162
Code: Select all
if /i "%3"=="/includemsse" set INC_MSSE=1
if /i "%3"=="/includewddefs" (
  echo %1 | %SystemRoot%\System32\find.exe /I "w62" >nul 2>&1
  if errorlevel 1 (
    echo %1 | %SystemRoot%\System32\find.exe /I "w63" >nul 2>&1
    if errorlevel 1 (
      echo %1 | %SystemRoot%\System32\find.exe /I "w100" >nul 2>&1
      if errorlevel 1 (set INC_WDDEFS=1) else (set INC_MSSE=1)
    ) else (set INC_MSSE=1)
  )
)


But this is confusing for users, if they try to understand, what exactly will be downloaded:

In my download scripts, I added a third option -includewddefs8, which is msse without the installers. This looks like a good solution to me, because it removes the ambiguities of /includewddefs.


There should be an option to synchronize the client directory as a whole.

I don't really like the incremental copying of subdirectories with lots of include and exclude lists. This may be necessary for the creation of ISO images, because they should not get too large, if they are to be burned to optical disks.

But for copying to an external disk, this is not needed. External hard disk drives are always large enough, and even USB flash disks are as large as 128 GB today. Then the simplest approach would be to copy the client directory as a whole to the external drive. Actually, this is why my own script is so short.

A copy-to-target script should have such an option. The Windows script CopyToTarget.cmd supports this approach with the targets "all all-x86 all-x64", in addition to copying Windows or Office downloads one by one.


Use the rsync option --delete instead of implementing the cleanup in the script.

rsync does have an option to delete obsolete files, which are not in the source directory anymore. This may be easier and safer than implementing the cleanup code yourself. But I don't know, how this works with the exclude lists, which are used by the script.


Development platform?

I don't know, if someday another development platform will be used. The current server is outdated, and it doesn't support https ( viewtopic.php?f=5&t=5564 ). This is not very convincing.

GitHub and "pull requests" may be overrated, though. Even Linus Torvalds, who created git, doesn't use them. He just wants to get email with patches, and then he tries them out himself.
hbuhrmester
 
Posts: 525
Joined: 11.10.2013, 20:59

Re: CopyToTarget linux script - where to submit

Postby Valicek1 » 29.08.2017, 22:43

Thanks for exhausting answer, I'll try to reply yesterday. Some things are my fault, some are done as it is by good means, but it's too late for me to write adequate answer now.
Valicek1
 

Re: CopyToTarget linux script - where to submit

Postby Valicek1 » 02.09.2017, 13:53

hbuhrmester wrote:Actually, I already have a copy-to-target script. It is the script 70-synchronize-with-target.bash in the directory available-tasks. It can be enabled by moving it to the directory update-generator-tasks or by creating a symbolic link in that directory. It could also be sourced from the script get-all-updates.bash, after running the download jobs.

The script is very simple, though: It only defines a source and a target directory, and then uses rsync to synchronize the target with the source. By default, the source directory is the client directory, which will be copied as a whole to a large enough external drive.

A more elaborate script with more options, logging and error checking is certainly welcome. But, after browsing through the script, I have some comments and suggestions:


I know about it, but I was missing opinion to create "Small" image with files for some target/sort of targets - example, I have complete WSUS mirror on my server on network mount, but from time to time, I need to create just windows 7/8/10 image. This is what was the script meant for.

hbuhrmester wrote:shellcheck complains

First of all, the utility shellcheck has numerous complaints:

Code: Select all
....


I try to solve most of them, with the exception, that I sometimes explicitly want word splitting of some variables. This could still be changed by implementing these variables as indexed arrays (which I may do someday).

For some reason, shellcheck doesn't like UTF-8 characters, if it is run within a terminal emulator. But shellcheck is also nicely integrated in Geany, and then it can be called as "Lint" from the Build menu.


Shellcheck was great idea... The most of complains was unnecessary from the nature how the varriables were assigned, "strict" mode used and so on, but I have corrected them just to clean output of shellcheck. For better orientation, to see if there isnt anything "special" - yes, bashs behaviour is sometimes really obscure.

hbuhrmester wrote:Declaration of variables with "local" and a command substitution should be in separate lines.

I learned, that this is a common pitfall: http://mywiki.wooledge.org/BashPitfalls ... command.29


Yes, sad true, same as errorcodes in pipe, [ sometest ] && command || othercommand and so on. This is one of reasons why i love using "unofficial bash strict mode"

hbuhrmester wrote:This example only mentions "local", but I assume, that the rule refers to "declare" and "readonly" as well.

Yes, it would be same, but from the nature of declare and readonly, it is impossible to define variable in one line and assign it in another one.

hbuhrmester wrote:The description of the function containsElement seems to be wrong.

Instead of:
...
it should be:
...

The function is actually used this way.

Yes, my bad. Corrected.

hbuhrmester wrote:The script should check, if rsync is installed.

Corrected, didn't know about this change from Debian 9.

hbuhrmester wrote:The directories wddefs and msse seem to be used wrong.

The directory wddefs contains virus definition files for the original Windows Defender. This was the built-in anti-virus software of Windows Vista and Windows 7. It was an optional installation for Windows XP. These virus definitions are not suitable for Windows 8 and later.

The built-in Defender of Window 8, 8.1 and 10 uses the same virus definitions as Microsoft Security Essentials. These Windows versions don't need the installation programs, but they do need the virus definitions from the directory msse.

Note, that the script DownloadUpdates.cmd uses the option /includewddefs in two different ways:
  • For Windows Vista and 7 (w60, w61) it includes the directory wddefs.
  • For Windows 8, 8.1 and 10 (w62, w63, w100) it includes the directory msse.

The meaning of /includewddefs is changed internally:

Yes, my script was doing it in the opposite way - msse for windows 7, wddefs for others. Now, it should be okay.

hbuhrmester wrote:DownloadUpdates.cmd, lines 152 - 162
Code: Select all
if /i "%3"=="/includemsse" set INC_MSSE=1
if /i "%3"=="/includewddefs" (
  echo %1 | %SystemRoot%\System32\find.exe /I "w62" >nul 2>&1
  if errorlevel 1 (
    echo %1 | %SystemRoot%\System32\find.exe /I "w63" >nul 2>&1
    if errorlevel 1 (
      echo %1 | %SystemRoot%\System32\find.exe /I "w100" >nul 2>&1
      if errorlevel 1 (set INC_WDDEFS=1) else (set INC_MSSE=1)
    ) else (set INC_MSSE=1)
  )
)



Tried to rewrite thins into bash.

hbuhrmester wrote:In my download scripts, I added a third option -includewddefs8, which is msse without the installers. This looks like a good solution to me, because it removes the ambiguities of /includewddefs.


Was thinking about it, but not sure how it works -> I didn't tried to implement this.

hbuhrmester wrote:There should be an option to synchronize the client directory as a whole.

I don't really like the incremental copying of subdirectories with lots of include and exclude lists. This may be necessary for the creation of ISO images, because they should not get too large, if they are to be burned to optical disks.


I don't like concept of exclude lists too, but this IMO comes from windows cmd possiblities. If it was primairly written by linux guys, it would look better, I believe.

hbuhrmester wrote:But for copying to an external disk, this is not needed. External hard disk drives are always large enough, and even USB flash disks are as large as 128 GB today. Then the simplest approach would be to copy the client directory as a whole to the external drive. Actually, this is why my own script is so short.

A copy-to-target script should have such an option. The Windows script CopyToTarget.cmd supports this approach with the targets "all all-x86 all-x64", in addition to copying Windows or Office downloads one by one.



Copy "all" - thinking how to implement it, I know, it should be there, but not sure how it works under windows, so I didn't tried to make it.

hbuhrmester wrote:Use the rsync option --delete instead of implementing the cleanup in the script.

rsync does have an option to delete obsolete files, which are not in the source directory anymore. This may be easier and safer than implementing the cleanup code yourself. But I don't know, how this works with the exclude lists, which are used by the script.


I didn't realize there are two --delete options in rsync - so I implemented cleanup, in persuation of myself that --delete will delete excluded files like --delete-excluded. Will fix soon.

hbuhrmester wrote:Development platform?

I don't know, if someday another development platform will be used. The current server is outdated, and it doesn't support https ( viewtopic.php?f=5&t=5564 ). This is not very convincing.

GitHub and "pull requests" may be overrated, though. Even Linus Torvalds, who created git, doesn't use them. He just wants to get email with patches, and then he tries them out himself.


I was just aking :) HTTPS supprort is great argument, as outdated, yes, really really outdated PHP 5.2 isn't secure much more.. Githubs PR are nice, but it's decision of every software project to use it or not - for me, it's much simpler to create fork, commit and push than format patch. In the other hand, github performs very slow for linux sized git repos.

Thanks for your time, i hope I answered everything appropriately.
Valicek1
 

Re: CopyToTarget linux script - where to submit

Postby hbuhrmester » 02.09.2017, 18:21

Hello,
thanks for your contributions!

Regarding the function "contains" again: Some people think, that "eval is evil" and that indirection should be avoided, if possible. It makes the code slightly more obscure, though maybe not in a short function like this.

Still, it isn't really necessary. The function could also be written like:

Code: Select all
# Does Array contain element?
# Usage: contains "requested string" "${array[@]}"
# ========================================================================
function contains() {
    if (( $# < 2 )); then
        die 1 "At least two parameters are expected."
    fi
    local value="$1"
    shift 1

    local i=""
    for i in "$@"; do
        if [[ "$i" == "$value" ]]; then
            return 0
        fi
    done
    return 1
}


(exchanging the order of the parameters again)

There are probably more complex use cases, where indirection could be used:
http://mywiki.wooledge.org/BashFAQ/006



(about a possible option -includewddefs8)

Was thinking about it, but not sure how it works -> I didn't tried to implement this.


For Windows 8 and higher, you only need the files:

Code: Select all
mpam-fe.exe
mpam-fex64.exe
nis_full_x86.exe
nis_full_x64.exe


and you could exclude:

Code: Select all
MSEInstall-*.exe


It doesn't make a big difference, but somebody once complained, that the installers were included in Windows 8 downloads.



Copy "all" - thinking how to implement it, I know, it should be there, but not sure how it works under windows, so I didn't tried to make it.


"Copying all" should be the easiest option - just don't use any exclude list at all, and rsync will copy everything.

But you could also use the exclude lists:

Code: Select all
wsusoffline/exclude/ExcludeListUSB-all.txt
wsusoffline/exclude/ExcludeListUSB-all-x64.txt
wsusoffline/exclude/ExcludeListUSB-all-x86.txt


which would be applied just like the exclude lists for the other targets.

The ExcludeListUSB-all.txt only removes the file UpdateInstaller.au3, which is the AutoIt source code for the application UpdateInstaller.exe. The source code is not really needed for users; it is only included, because the GPL demands, that it should be made available somewhere.

https://www.autoitscript.com/
hbuhrmester
 
Posts: 525
Joined: 11.10.2013, 20:59


Return to Linux

Who is online

Users browsing this forum: No registered users and 40 guests