[ipxe-devel] vmxnet3 port

[Joshua Oreman]

On Thu, Jan 5, 2012 at 12:22 AM, Pete Holland <pholland27 at gmail.com> wrote:
> So I'm working on a port of the vmxnet3 driver to ipxe (mainly because
> the default pxe client in ESX is not suitable for my needs).
> One problem I've run into is that the vmxnet3 driver seems to require
> far more memory (by default somewhere between 1-2 MB) from
> the heap than ipxe normally has (128K) for setting up all of its DMA
> structures.  I am currently hacking malloc.c to up the heap size
> but I'm wondering if there is a 'correct' way of doing this in terms
> of ipxe development (since, if I'm successful, I'd like to contribute
> the
> port to the project).

[Disclaimer: it's been a while since I've worked significantly on
iPXE, although I used to be a pretty active developer.]

This is definitely an issue that's come up before, e.g. with
Infiniband drivers that require several megabytes of memory to store
their firmware. There are technical reasons why iPXE's heap is as
small as it is, so increasing its size dramatically is, unfortunately,
likely to make your patch unsuitable for general inclusion.

I'm not familiar with vmxnet specifically, but oftentimes the size of
the DMA ring can be dramatically reduced from what would be typical in
an OS network driver. iPXE won't need more than about 16 TX or RX
descriptors at once, whereas I've seen Linux drivers allocate 512 or
more of each. This might allow you to make the DMA allocations within
the confines of iPXE's heap. That said, there are definitely some
drivers that have a rather large minimum ring size, and for which this
method won't work.

If you do need lots of memory for your driver, you can use the
umalloc()/ufree() allocator instead of malloc()/free(). This allocates
memory external to iPXE's heap; the same facility is used to store
images that have been fetched but not yet executed. You get a "user
pointer" (typedefed integer type) instead of a real pointer, to allow
iPXE to run on architectures that don't allow this external memory to
be directly addressed (e.g. segmented 16-bit x86). There are functions
such as copy_from_user() and user_to_phys() that allow you to work
with these user pointers in a way that doesn't sacrifice portability.

> The second problem is that I am at the point where I have set up the
> BARs correctly (as I can read/write the MAC address, link state is
> readable,
> and resetting the device seems to jigger some logs into vmware.log
> indicating that it is successfully doing the reset), however
> 'activating' the device fails (with no indication in vmware.log or
> hostd.log as to why).
> At this point, I'm about to go hat-in-hand to the vmware dev
> responsible for the Linux vmxnet3 driver for some insight, but I
> figured I'd ask if
> anyone else here has tried this before, heard of anyone trying this,
> or otherwise has a thought on the matter.

Unfortunately, much hardware (real and virtual) is quite finicky about
the exact manner in which it's initialized. Being unfamiliar with the
specifics of vmxnet, I can only suggest triple-checking you're doing
everything necessary in exactly the right way, and wish you the best
of luck.

Hope this helps,
Josh

> Cheers,
> Pete
> _______________________________________________
> ipxe-devel mailing list
> ipxe-devel at lists.ipxe.org
> https://lists.ipxe.org/mailman/listinfo/ipxe-devel