Actually, I already have a copy-to-target script. It is the script 70-synchronize-with-target.bash in the directory available-tasks. It can be enabled by moving it to the directory update-generator-tasks or by creating a symbolic link in that directory. It could also be sourced from the script get-all-updates.bash, after running the download jobs.
The script is very simple, though: It only defines a source and a target directory, and then uses rsync to synchronize the target with the source. By default, the source directory is the client directory, which will be copied as a whole to a large enough external drive.
A more elaborate script with more options, logging and error checking is certainly welcome. But, after browsing through the script, I have some comments and suggestions:
shellcheck complainsFirst of all, the utility shellcheck has numerous complaints:
- Code: Select all
In copy-to-target.bash line 60:
readonly script_dir=$(dirname $(realpath $0))
^-- SC2046: Quote this to prevent word splitting.
^-- SC2086: Double quote to prevent globbing and word splitting.
In copy-to-target.bash line 83:
toPut="$(date "+%x %X") $@"
^-- SC2124: Assigning an array to a string! Assign as array, or use * instead of @ to concatenate.
In copy-to-target.bash line 84:
echo $toPut
^-- SC2086: Double quote to prevent globbing and word splitting.
In copy-to-target.bash line 85:
echo "$toPut" >> $script_dir/$logfile
^-- SC2086: Double quote to prevent globbing and word splitting.
In copy-to-target.bash line 95:
exit $rc
^-- SC2086: Double quote to prevent globbing and word splitting.
In copy-to-target.bash line 130:
log "CMD: $0 $@"
^-- SC2145: Argument mixes string and array. Use * or separate argument.
In copy-to-target.bash line 141:
if contains ${WinTargets[@]} $update
^-- SC2068: Double quote array expansions to avoid re-splitting elements.
^-- SC2086: Double quote to prevent globbing and word splitting.
In copy-to-target.bash line 144:
contains ${WinLangs[@]} $language || die 1 "Unknown Windows Language -> exiting"
^-- SC2068: Double quote array expansions to avoid re-splitting elements.
^-- SC2086: Double quote to prevent globbing and word splitting.
In copy-to-target.bash line 147:
if contains ${OfcTargets[@]} $update
^-- SC2068: Double quote array expansions to avoid re-splitting elements.
^-- SC2086: Double quote to prevent globbing and word splitting.
In copy-to-target.bash line 150:
contains ${OfcLangs[@]} $language || die 1 "Unknown Office Language -> exiting"
^-- SC2068: Double quote array expansions to avoid re-splitting elements.
^-- SC2086: Double quote to prevent globbing and word splitting.
In copy-to-target.bash line 183:
if echo $update | grep -E $WddefsTargets > /dev/null
^-- SC2086: Double quote to prevent globbing and word splitting.
In copy-to-target.bash line 212:
cd $script_dir
^-- SC2086: Double quote to prevent globbing and word splitting.
In copy-to-target.bash line 229:
[ -f $excl_prefix-$update.txt ] && cp $excl_prefix-$update.txt $filter || cp $excl_prefix-$update-x86.txt $filter
^-- SC2086: Double quote to prevent globbing and word splitting.
^-- SC2015: Note that A && B || C is not if-then-else. C may run when A is true.
^-- SC2086: Double quote to prevent globbing and word splitting.
^-- SC2086: Double quote to prevent globbing and word splitting.
^-- SC2086: Double quote to prevent globbing and word splitting.
^-- SC2086: Double quote to prevent globbing and word splitting.
In copy-to-target.bash line 232:
[ -f $filter_dir/custom/ExcludeListUSB-$update.txt ] && cat $filter_dir/custom/ExcludeListUSB-$update.txt >> $filter
^-- SC2086: Double quote to prevent globbing and word splitting.
^-- SC2086: Double quote to prevent globbing and word splitting.
^-- SC2086: Double quote to prevent globbing and word splitting.
In copy-to-target.bash line 234:
[ -f $filter_dir/custom/ExcludeListUSB-$update-x86.txt ] && cat $filter_dir/custom/ExcludeListUSB-$update-x86.txt >> $filter
^-- SC2086: Double quote to prevent globbing and word splitting.
^-- SC2086: Double quote to prevent globbing and word splitting.
^-- SC2086: Double quote to prevent globbing and word splitting.
In copy-to-target.bash line 238:
for loc in ${locales[@]}
^-- SC2068: Double quote array expansions to avoid re-splitting elements.
In copy-to-target.bash line 241:
[ ! "$language" == "$loc" ] && echo "$loc/" >> $filter
^-- SC2086: Double quote to prevent globbing and word splitting.
In copy-to-target.bash line 245:
[ $opt_exclude_sp -eq 1 ] && cat $filter_dir/ExcludeList-SPs.txt >> $filter
^-- SC2086: Double quote to prevent globbing and word splitting.
In copy-to-target.bash line 246:
[ $opt_exclude_sw -eq 1 ] && cat $filter_dir/ExcludeList-software.txt >> $filter
^-- SC2086: Double quote to prevent globbing and word splitting.
In copy-to-target.bash line 247:
[ $opt_include_dotnet -eq 0 ] && cat $filter_dir/ExcludeListISO-dotnet.txt >> $filter
^-- SC2086: Double quote to prevent globbing and word splitting.
In copy-to-target.bash line 248:
[ $opt_include_msse -eq 0 ] && cat $filter_dir/ExcludeList-msse.txt >> $filter
^-- SC2086: Double quote to prevent globbing and word splitting.
In copy-to-target.bash line 249:
[ $opt_include_wddefs -eq 0 ] && cat $filter_dir/ExcludeList-wddefs.txt >> $filter
^-- SC2086: Double quote to prevent globbing and word splitting.
In copy-to-target.bash line 252:
cat $filter | sed 's/\\/\//g' | sed 's/\r$//' > $filter.tmp
^-- SC2086: Double quote to prevent globbing and word splitting.
^-- SC2002: Useless cat. Consider 'cmd < file | ..' or 'cmd file | ..' instead.
^-- SC2086: Double quote to prevent globbing and word splitting.
In copy-to-target.bash line 254:
cat /dev/null > $filter
^-- SC2086: Double quote to prevent globbing and word splitting.
In copy-to-target.bash line 256:
for line in $(cat $filter.tmp)
^-- SC2013: To read lines rather than words, pipe/redirect to a 'while read' loop.
^-- SC2086: Double quote to prevent globbing and word splitting.
In copy-to-target.bash line 259:
if echo $line | grep -E '.*/$' > /dev/null
^-- SC2086: Double quote to prevent globbing and word splitting.
In copy-to-target.bash line 261:
echo $line >> $filter
^-- SC2086: Double quote to prevent globbing and word splitting.
^-- SC2086: Double quote to prevent globbing and word splitting.
In copy-to-target.bash line 263:
printf '*%s*\n' $line >> $filter
^-- SC2086: Double quote to prevent globbing and word splitting.
^-- SC2086: Double quote to prevent globbing and word splitting.
In copy-to-target.bash line 267:
[ -f $target_dir ] && rm -f $target_dir
^-- SC2086: Double quote to prevent globbing and word splitting.
^-- SC2086: Double quote to prevent globbing and word splitting.
In copy-to-target.bash line 272:
rm -Rf $target_dir
^-- SC2086: Double quote to prevent globbing and word splitting.
In copy-to-target.bash line 276:
mkdir -p $target_dir
^-- SC2086: Double quote to prevent globbing and word splitting.
In copy-to-target.bash line 286:
tar=$(realpath $target_dir)
^-- SC2086: Double quote to prevent globbing and word splitting.
In copy-to-target.bash line 287:
rsync -a --log-file=$log_dir/rsync.txt --exclude-from=$filter --link-dest=$src $src/ $tar
^-- SC2086: Double quote to prevent globbing and word splitting.
^-- SC2086: Double quote to prevent globbing and word splitting.
^-- SC2086: Double quote to prevent globbing and word splitting.
^-- SC2086: Double quote to prevent globbing and word splitting.
In copy-to-target.bash line 289:
rsync -a --log-file=$log_dir/rsync.txt --exclude-from=$filter . $target_dir
^-- SC2086: Double quote to prevent globbing and word splitting.
^-- SC2086: Double quote to prevent globbing and word splitting.
In copy-to-target.bash line 298:
for f in $(find $target_dir -printf "%P\n")
^-- SC2044: For loops over find output are fragile. Use find -exec or a while read loop.
^-- SC2086: Double quote to prevent globbing and word splitting.
In copy-to-target.bash line 304:
if [ -f $target_dir/$f ]
^-- SC2086: Double quote to prevent globbing and word splitting.
^-- SC2086: Double quote to prevent globbing and word splitting.
In copy-to-target.bash line 307:
if [ ! -f $f ]
^-- SC2086: Double quote to prevent globbing and word splitting.
In copy-to-target.bash line 310:
rm -f $target_dir/$f
^-- SC2086: Double quote to prevent globbing and word splitting.
^-- SC2086: Double quote to prevent globbing and word splitting.
In copy-to-target.bash line 314:
if [ ! -d $f ]
^-- SC2086: Double quote to prevent globbing and word splitting.
In copy-to-target.bash line 317:
rm -rf $target_dir/$f
^-- SC2115: Use "${var:?}" to ensure this never expands to / .
^-- SC2086: Double quote to prevent globbing and word splitting.
^-- SC2086: Double quote to prevent globbing and word splitting.
I try to solve most of them, with the exception, that I sometimes explicitly want word splitting of some variables. This could still be changed by implementing these variables as indexed arrays (which I may do someday).
For some reason, shellcheck doesn't like UTF-8 characters, if it is run within a terminal emulator. But shellcheck is also nicely integrated in Geany, and then it can be called as "Lint" from the Build menu.
Declaration of variables with "local" and a command substitution should be in separate lines.I learned, that this is a common pitfall:
http://mywiki.wooledge.org/BashPitfalls ... command.29This example only mentions "local", but I assume, that the rule refers to "declare" and "readonly" as well.
The description of the function containsElement seems to be wrong.Instead of:
- Code: Select all
# Does Array contain element?
# Usage: containsElement "requested string" "${array[@]}"
it should be:
- Code: Select all
# Does Array contain element?
# Usage: containsElement "${array[@]}" "requested string"
The function is actually used this way.
The script should check, if rsync is installed.In Debian 9 Stretch (stable), rsync is an optional package, which is not installed by default. Then the script should check, if rsync is actually available.
This is also missing in my script, but it was supposed to be tested in the script 30-check-needed-applications.bash instead.
The directories wddefs and msse seem to be used wrong.The directory wddefs contains virus definition files for the original Windows Defender. This was the built-in anti-virus software of Windows Vista and Windows 7. It was an optional installation for Windows XP. These virus definitions are
not suitable for Windows 8 and later.
The built-in Defender of Window 8, 8.1 and 10 uses the same virus definitions as Microsoft Security Essentials. These Windows versions don't need the installation programs, but they do need the virus definitions from the directory msse.
Note, that the script DownloadUpdates.cmd uses the option /includewddefs in two different ways:
- For Windows Vista and 7 (w60, w61) it includes the directory wddefs.
- For Windows 8, 8.1 and 10 (w62, w63, w100) it includes the directory msse.
The meaning of /includewddefs is changed internally:
DownloadUpdates.cmd, lines 152 - 162- Code: Select all
if /i "%3"=="/includemsse" set INC_MSSE=1
if /i "%3"=="/includewddefs" (
echo %1 | %SystemRoot%\System32\find.exe /I "w62" >nul 2>&1
if errorlevel 1 (
echo %1 | %SystemRoot%\System32\find.exe /I "w63" >nul 2>&1
if errorlevel 1 (
echo %1 | %SystemRoot%\System32\find.exe /I "w100" >nul 2>&1
if errorlevel 1 (set INC_WDDEFS=1) else (set INC_MSSE=1)
) else (set INC_MSSE=1)
)
)
But this is confusing for users, if they try to understand, what exactly will be downloaded:
In my download scripts, I added a third option -includewddefs8, which is msse without the installers. This looks like a good solution to me, because it removes the ambiguities of /includewddefs.
There should be an option to synchronize the client directory as a whole.I don't really like the incremental copying of subdirectories with lots of include and exclude lists. This may be necessary for the creation of ISO images, because they should not get too large, if they are to be burned to optical disks.
But for copying to an external disk, this is not needed. External hard disk drives are always large enough, and even USB flash disks are as large as 128 GB today. Then the simplest approach would be to copy the client directory as a whole to the external drive. Actually, this is why my own script is so short.
A copy-to-target script should have such an option. The Windows script CopyToTarget.cmd supports this approach with the targets "all all-x86 all-x64", in addition to copying Windows or Office downloads one by one.
Use the rsync option --delete instead of implementing the cleanup in the script.rsync does have an option to delete obsolete files, which are not in the source directory anymore. This may be easier and safer than implementing the cleanup code yourself. But I don't know, how this works with the exclude lists, which are used by the script.
Development platform?I don't know, if someday another development platform will be used. The current server is outdated, and it doesn't support https (
viewtopic.php?f=5&t=5564 ). This is not very convincing.
GitHub and "pull requests" may be overrated, though. Even Linus Torvalds, who created git, doesn't use them. He just wants to get email with patches, and then he tries them out himself.