[ipxe-devel] undionly.kpxe http download slow

[Michael Brown]

On 28/10/13 17:05, Daniel Pittman wrote:
> It didn't include a "Content-Length" header in the HTTP response, just
> sent an HTTP chunked file down to iPXE.  This led to insanely slow
> downloads; once we patched Sinatra to include that header downloads
> are comparable to TFTP performance.
>
> Our theory, which we never investigated enough to verify, was that
> this is some sort of memory management issue in the downloader, which
> spends increasing amounts of time reallocating and moving memory
> around as the download buffer grows.  (eg: over an hour for 170MB,
> compared to less than a minute once Content-Length is included.)

Yes; that will be the behaviour if no Content-Length is provided. 
Downloaded images are placed at the top of free (32-bit) memory, and if 
the size is unknown then it will effectively perform a realloc() for 
each 4kB chunk.  Once the image gets sufficiently large, the time taken 
for a realloc() starts to cause dropped packets, which further impacts 
performance.

With a Content-Length present, HTTP should be substantially faster than 
TFTP, since the window size for TCP transfers is 256kB compared to 1.5kB 
(one packet) for TFTP.  For reference, HTTP transfers should achieve 
full line speed on a link of up to 1Gbps, which gives ~120MB/s of 
downloaded data.

Michael