[ipxe-devel] undionly.kpxe http download slow

[Levente LEVAI]

On 2013-10-28 18:56, Michael Brown wrote:
> On 28/10/13 17:05, Daniel Pittman wrote:
>> It didn't include a "Content-Length" header in the HTTP response, just
>> sent an HTTP chunked file down to iPXE.  This led to insanely slow
>> downloads; once we patched Sinatra to include that header downloads
>> are comparable to TFTP performance.
>>
>> Our theory, which we never investigated enough to verify, was that
>> this is some sort of memory management issue in the downloader, which
>> spends increasing amounts of time reallocating and moving memory
>> around as the download buffer grows.  (eg: over an hour for 170MB,
>> compared to less than a minute once Content-Length is included.)
>
> Yes; that will be the behaviour if no Content-Length is provided. 
> Downloaded images are placed at the top of free (32-bit) memory, and 
> if the size is unknown then it will effectively perform a realloc() 
> for each 4kB chunk.  Once the image gets sufficiently large, the time 
> taken for a realloc() starts to cause dropped packets, which further 
> impacts performance.
>
> With a Content-Length present, HTTP should be substantially faster 
> than TFTP, since the window size for TCP transfers is 256kB compared 
> to 1.5kB (one packet) for TFTP.  For reference, HTTP transfers should 
> achieve full line speed on a link of up to 1Gbps, which gives ~120MB/s 
> of downloaded data.
>
> Michael
I looked into what is being sent (tcpdump), and yes, you are absolutely 
right: publicfile httpd
sends no Content-Length header. I tried with nginx (which does send this 
header), it was
difficult to measure the speed, certainly below 2secs for 20MB (=link 
speed as you wrote,
I have a 100Mb line).

Maybe all this should be mentioned somewhere in the docs.

Thanks for the insight!

BR
Levente