Consider using aria2 as a replacement to wget

[qoole]

Hiya,

I've often noted that the bottleneck for WOU is wget. I have a 120mbit/s connection and most of the time wget will only download @ ~1-2MiB/s (8-16mbit/s).
I did a bit of tinkering with DownloadUpdates.cmd and replaced all references to wget with aria2 and I now download at 15MiB/s on almost all files!

This is due to aria2 splitting downloads into chunks and downloading them in parallel. It will also download multiple files when presented with a file containing multiple urls (as happens quite often in DownloadUpdates.cmd).

Seeing as I can't seem to attach .txt or .cmd files I have uploaded my version of DownloadUpdates.cmd to: http://pastebin.com/NZAw2M95
aria2 can be downloaded from http://aria2.sourceforge.net/

I would love to hear your feedback on this mod!

Thanks,

Alex

[boco]

In the long run, replacing wget will be inevitable anyway. The current version we use doesn't even support IPv6, for example.

Will definitely take a look at that tool.

[WSUSUpdateAdmin]

Obviously, there are many alternatives for wget (see http://www.mynitor.com/2010/09/30/comma ... ch-better/, http://superuser.com/questions/25538/wh ... nt-of-wget and others), so maybe we should evaluate and discuss a bit more.
Right now, there's no need for hurry in my eyes.
RTW

[brb8two]

Hi,

To cut down on the tools to test, I started looking into the links you provided.

Curl has a nice comparison table the helps trim down the options. (Plus Mulk was listed else ware but looks to be using curl library)
http://curl.haxx.se/docs/comparison-table.html
curl snarf wget pavuk fget fetch lftp aria2 HTTrack

i.e. at minimum there should be a native Windows Program (/port). Which leave us with the following;
curl wget aria2 HTTrack Mulk

You mentioned IPv6 Support and the above selection all support it (I have not found a Windows compiled wget with IPv6 tho.)
(not sure about Mulk, can't find it Identified as IPv6 but it uses cURL library and it supports IPv6)

The point of changing would be for "Parallel Downloads" support to improve large file download speeds. Which leave us with the following;
aria2 & HTTrack & Mulk

Next would be to see which supports similar inputs to wget (i.e. static link files as we don't want to re-write them all)
I have not gone that far but the work has been done for Aria2 in this sense.


On a separate topic, I'm going to switch from wget to Aria2 for my project to give it a go. (got this one file download of ~900MB)

[brb8two]

my 2c about aria2 (now that I've given it a go.)

For files under ~50MB there is not much performance gain over wget. (I tested with Adobe Reader)

For a 900Mb download I wget would take ~90mins, aria2 took only 10mins. (I tested with NI-DAQmx config drivers)

Considering WSUS (most recent check of all versions) has ~ 23 files that are above 50MB (e.g. largest being windows6.1-KB976932-X64 = 940MB) aria2 could help speed up the download.

while trying to use for CDBurner XP Pro however wget changes the filename to the correct one and aria2 fails at this.
Also both fail at resolving the correct file name for the latest Java download.

[brb8two]

If you think Aria2 is too big to include in WSUS, this script will download and extract it

Call it "DownloadAria.cmd"

Code: Select all

REM Download Aria2 Compiled for Win32
IF EXIST ..\bin\aria2c.exe GOTO :EOF
pushd ..\bin
wget -N http://downloads.sourceforge.net/project/aria2/stable/aria2-1.17.1/aria2-1.17.1-win-32bit-build1.zip?r=http%3A%2F2Fsourceforge.net%2Fprojects%2Faria2%2Ffiles%2Fstable%2Faria2-1.17.1%2F"&"ts=1377074326"&"use_mirror=jaist
for /F %%A IN ('DIR /OD /b  *aria*.zip') DO SET FILE=%%A
unzip -p %FILE% */aria2c.exe >aria2c.exe
popd


I've got one for wget using a javaScript to download it (From the "windows Equivalent of wget" page you linked to ealier).

[WSUSUpdateAdmin]

Hi!

Thanks a lot for investigation and evaluation!

If the input files for wget and aria2 are compatible, I think I'll integrate aria2, but this will take a while.

Thanks again & kind regards
Torsten Wittrock

[brb8two]

Hi,

As "Alex" aka qoole (first post) indicated, he has made the changes to his copy of (DownloadUpdates.cmd) and posted it http://pastebin.com/NZAw2M95.

So all you'd need to do is compare with WSUS version and if your happy with the mods, perhaps Alex would like a "rem *** Contrib/mod: Alex (goole) ****" line added or something similar.

[WSUSUpdateAdmin]

Hi.

http://trac.wsusoffline.net/browser/trunk (r494):

• NOTE: If you like to use aria2 instead of wget for downloading, please call ActivateAria2Downloads.cmd once
• Optional use of aria2 instead of wget to speed up downloading (Thanks to "qoole" and "brb8two")

...so everyone can play with it and decide on his/her own.

Thanks & regards
Torsten Wittrock

[Gerby]

Hi!

The aria2 alternative doesn't really convince me, yet. It may have advantages if you have a large bandwidth on your download stream. But this advantage only comes into play on large files, so usually when building the repository for the first time.

On the other side, aria2 is slower when the dynamic update lists are processed. This is, because for each file an instance of the downloading program (wget/aria2) is started. This works more fluently with wget. And this is the scenario that occurs much more often than an actual download of a (large) file.

For the future: Maybe there could be done some tweaking on processing multiple files from the dynamic download list with aria2. Would it be an option to create a temporary URL list first and then pass the whole list as parameter to aria2 (just a thought, not investigated about that)?

Of course there are other advantages of aria2 (e.g. no need for timezone tweak, IPv6 support, 64-bit binary). However, for now, it was a good idea from Torsten still letting the user decide which download tool to use.

Just my two cents.
Greetz
Gerby