A complete rewrite of the Linux scripts

Re: A complete rewrite of the Linux scripts

Postby mani » 21.01.2018, 15:34

The newest wget version 1.19 has an option "--compression=none" to avoid the new feature of automatic decompression........

after editing 40-configure-downloaders.bash (wget_common_options) it worked for this run, but the option is naturally gone after the update ;-)

BTW: can you add a delete of the update files before the download, because it doesn't get downloaded from wget again, due to existence of this files in the tmp directory

BTW: i would be great to have an extra add-on option in the preferences to avoid the adding after every update

Thanks
Mani
mani
 
Posts: 21
Joined: 04.04.2017, 15:50

Re: A complete rewrite of the Linux scripts

Postby hbuhrmester » 21.01.2018, 15:43

I have uploaded the file again as "sh-new-1.0.tgz".

The server seems to handle this file slightly differently: The content-type is now "application/x-gtar", and there is no content-encoding set. Then wget should not be lead to decompress the content on the fly.

Code: Select all
hb1@debian:~/tmp$ wget --debug http://downloads.hartmut-buhrmester.de/sh-new-1.0.tgz
DEBUG output created by Wget 1.18 on linux-gnu.

Reading HSTS entries from /home/hb1/.wget-hsts
URI encoding = »UTF-8«
Converted file name 'sh-new-1.0.tgz' (UTF-8) -> 'sh-new-1.0.tgz' (UTF-8)
--2018-01-21 15:24:44--  http://downloads.hartmut-buhrmester.de/sh-new-1.0.tgz
Auflösen des Hostnamens »downloads.hartmut-buhrmester.de (downloads.hartmut-buhrmester.de)« … 80.237.132.245, 2a01:488:42:1000:50ed:84f5:3f:9682
Caching downloads.hartmut-buhrmester.de => 80.237.132.245 2a01:488:42:1000:50ed:84f5:3f:9682
Verbindungsaufbau zu downloads.hartmut-buhrmester.de (downloads.hartmut-buhrmester.de)|80.237.132.245|:80 … verbunden.
Created socket 3.
Releasing 0x00920240 (new refcount 1).

---request begin---
GET /sh-new-1.0.tgz HTTP/1.1
User-Agent: Wget/1.18 (linux-gnu)
Accept: */*
Accept-Encoding: identity
Host: downloads.hartmut-buhrmester.de
Connection: Keep-Alive

---request end---
HTTP-Anforderung gesendet, auf Antwort wird gewartet …
---response begin---
HTTP/1.1 200 OK
Date: Sun, 21 Jan 2018 14:24:45 GMT
Content-Type: application/x-gtar
Content-Length: 242429
Connection: keep-alive
Server: Apache
Last-Modified: Sun, 21 Jan 2018 14:24:21 GMT
ETag: "3b2fd-5634a11d31684"
Accept-Ranges: bytes

---response end---
200 OK
Registered socket 3 for persistent reuse.
Länge: 242429 (237K) [application/x-gtar]
Wird in »»sh-new-1.0.tgz«« gespeichert.

sh-new-1.0.tgz            100%[=====================================>] 236,75K   631KB/s    in 0,4s   

2018-01-21 15:24:45 (631 KB/s) - »»sh-new-1.0.tgz«« gespeichert [242429/242429]


I've changed the file available-version.txt as well.

The file type, as identified by xdg-open, is still the same:

Code: Select all
hb1@debian:~/tmp$ xdg-mime query filetype sh-new-1.0.tar.gz
application/x-compressed-tar
hb1@debian:~/tmp$ xdg-mime query filetype sh-new-1.0.tgz
application/x-compressed-tar


Regards,
hbuhrmester
hbuhrmester
 
Posts: 525
Joined: 11.10.2013, 20:59

Re: A complete rewrite of the Linux scripts

Postby mani » 21.01.2018, 16:11

there is definitely a behavior change in wget

Differences:
wget -S http://downloads.hartmut-buhrmester.de/ ... 1.0.tar.gz
1.18:
Content-Encoding: x-gzip
2018-01-21 14:58:22 (789 KB/s) - ‘sh-new-1.0.tar.gz’ saved [242429/242429]
1.19:
Content-Encoding: gzip
2018-01-21 16:06:36 (777 KB/s) - »sh-new-1.0.tar.gz« gespeichert [583680]

Note: the name is not changed!!!!!
on 1.19 with option compression=none is the behavior similar as 1.18
wget --compression=none -S http://downloads.hartmut-buhrmester.de/ ... 1.0.tar.gz
Content-Encoding: x-gzip
2018-01-21 16:13:35 (780 KB/s) - »sh-new-1.0.tar.gz.2« gespeichert [242429/242429]
mani
 
Posts: 21
Joined: 04.04.2017, 15:50

Re: A complete rewrite of the Linux scripts

Postby Dalai » 21.01.2018, 16:16

mani wrote:It looks like the newest debian version has some strange changes.....

Looks like it. The NEWS file of wget-1.19.4.tar.gz (at https://ftp.gnu.org/gnu/wget/) contains this:
Code: Select all
* Changes in Wget 1.19.4
* A major bug that caused GZip'ed pages to never be decompressed has been fixed
* Support for Content-Encoding and Transfer-Encoding have been marked as
  experimental and disabled by default

* Changes in Wget 1.19.3
* Prevent erroneous decompression of .gz and .tgz files with broken servers

[...]

* Changes in Wget 1.19.2
[...]
* New option --compression for gzip Content-Encoding


Regards
Dalai
Dalai
 
Posts: 1041
Joined: 12.07.2016, 21:00

Re: A complete rewrite of the Linux scripts

Postby hbuhrmester » 21.01.2018, 16:29

Note: the name is not changed!!!!!


This is not a redirect; it is just another upload with a different name...

You surely need to try the new URL http://downloads.hartmut-buhrmester.de/sh-new-1.0.tgz

Regards,
hbuhrmester
hbuhrmester
 
Posts: 525
Joined: 11.10.2013, 20:59

Re: A complete rewrite of the Linux scripts

Postby mani » 21.01.2018, 16:57

Dear Hartmut
I meant that the decompressed file has still the compressed file name

the tgz file works perfect with wget version 1.19.2, because wget (option -S) doesn't find a Content-Encoding
2018-01-21 16:56:09 (783 KB/s) - »sh-new-1.0.tgz« gespeichert [242429/242429]

Mani

P.S.: lets wait for the version 1.19.4 on Debian testing and than I will check again
mani
 
Posts: 21
Joined: 04.04.2017, 15:50

Re: A complete rewrite of the Linux scripts

Postby mani » 27.01.2018, 11:33

good morning
Today I installed wget version 1.19.3-2 and it doesn't deflate the tar.gz files automatic anymore and the update runs flawless.

best regards
Mani
mani
 
Posts: 21
Joined: 04.04.2017, 15:50

Re: A complete rewrite of the Linux scripts

Postby hbuhrmester » 27.01.2018, 14:09

Very well,

the self update of the Linux scripts should also work with wget 1.19.2 now, because I uploaded the file as "sh-new-1.0.tgz", and I also changed the files:

http://downloads.hartmut-buhrmester.de/ ... ersion.txt
http://downloads.hartmut-buhrmester.de/ ... ew-1.0.txt

to point to the new tgz archive, rather than the tar.gz file.

The self-update script for the Linux scripts downloads the file "available-version.txt" first. This file has four pieces of information, separated by spaces:

  1. the version number
  2. the URL of the archive
  3. the URL of the hashes file
  4. the name of the expanded directory

This information can be read into four variables with the bash built-in command "read", and these are then used for the self-update of the Linux scripts.

So these URLs are not hard-coded in the script, and they could be adjusted to work with wget 1.19.2 as well.

This is similar to the self-update of WSUS Offline Update: The file "StaticDownloadLink-recent.txt" has the URL for the most recent version of WSUS Offline Update.

Regards,
hbuhrmester
hbuhrmester
 
Posts: 525
Joined: 11.10.2013, 20:59

Re: A complete rewrite of the Linux scripts

Postby WSUSUpdateAdmin » 16.03.2018, 18:35

Moin!

Ich habe gestern eine relevante Änderung/Erweiterung vorgenommen und die Datei ExcludeList-superseded-exclude-seconly.txt eingeführt, die bei der Berechnung von ExcludeList-superseded-seconly.txt zusätzlich zur Datei ExcludeList-superseded-exclude.txt wirkt (siehe http://trac.wsusoffline.net/changeset/9 ... pdates.cmd).
Sie soll KB-Nummern aufnehmen, die Microsoft im Katalog fälschlicherweise als "superseded" ausweist, die aber nur für /seconly-Builds relevant sind.
Eine "custom"-Version der Datei ist daher nicht geplant.

Gruß & ein schönes Wochenende,
Torsten
WSUSUpdateAdmin
Administrator
 
Posts: 2245
Joined: 07.07.2009, 14:38

Re: A complete rewrite of the Linux scripts

Postby username » 30.03.2018, 05:31

Thanks for these scripts!

- Fedora 27 (and probably earlier) dependencies for wsusoffline/sh/documentation/Installation_Guide.txt
- use_file_signature_verification?

Code: Select all
#!/usr/bin/env bash -x
# NOTE: this runs sudo in the middle of the script.

# web 'http://download.wsusoffline.net/'
LATEST="wsusoffline1121.zip"
wget "http://download.wsusoffline.net/$LATEST"
unzip "$LATEST"
cd wsusoffline/sh
bash ./fix-file-permissions.bash

# Fedora 27
sudo dnf install -y xmlstarlet cabextract md5deep wget trash-cli

# Will this fail? Is there a way to get the necessary public keys?
# There should probably be a warning about this in the docs and the
# script.
sed -i 's/^use_file_signature_verification="disabled"/use_file_signature_verification="enabled"/g' ./sh/download-updates.bash

# Windows 10 x64 & *
bash ./download-updates.bash w100-x64 enu -includesp -includecpp -includedotnet -includewddefs8 -includemsse
username
 
Posts: 1
Joined: 30.03.2018, 05:05

PreviousNext

Return to Linux

Who is online

Users browsing this forum: No registered users and 6 guests