[ipxe-devel] virtnet_open_legacy() fails to find virtqueues on Google Compute Engine

Wed Dec 14 09:17:50 UTC 2016

Hi Akihiro,

On Wed, Dec 14, 2016 at 8:51 AM, Akihiro Suda <suda.kyoto at gmail.com> wrote:
> Hi Christian,
>
> Thank you for the suggestion, I'll try to report this to google.
>
> I found that the cause of this issue is because GCE's VIRTIO_PCI_QUEUE_NUM
> is 4096, which is larger than iPXE's MAX_QUEUE_NUM (256).
>
> https://git.ipxe.org/ipxe.git/blob/26050fd4c87c50503d5bd573b2ec91703676e211:/src/drivers/bus/virtio-pci.c#l43
> https://git.ipxe.org/ipxe.git/blob/26050fd4c87c50503d5bd573b2ec91703676e211:/src/include/ipxe/virtio-ring.h#l24
>
>
> IIUC this issue will be resolved if GCE supports virtio 1.0, which would
> allow iPXE to use custom queue num smaller than the virtio device's max
> value.

iPXE virtio 1.0 code does not cap the queue size to MAX_QUEUE_NUM. In
fact, it does not check it against any maximum and still uses
fixed-sized virtqueue data structures, just like legacy. This is a
bug.

> However, it would be great if iPXE can support setting MAX_QUEUE_NUM to 4096
> (via `config/general.h`, probably).

Even better, iPXE should try to use the host provided size in legacy
mode, i.e. no MAX_QUEUE_NUM needed. The few extra dynamic allocations
shouldn't hurt anything.

> Actually I tried to set MAX_QUEUE_NUM to 4096 and got virtnet_open_legacy()
> succeeded, but it still doesn't work:
>
>     iPXE> dhcp
>     DHCP 0x120e44 entering discovery state
>     Configuring (net0 42:01:0a:92:00:0a)...DHCP 0x120e44 DHCPDISCOVER
>     DHCP 0x120e44 DHCPOFFER from 169.254.169.254:67 for 10.146.0.10
>     .DHCP 0x120e44 DHCPDISCOVER
>     ..DHCP 0x120e44 entering request state
>     DHCP 0x120e44 DHCPREQUEST to 169.254.169.254:67 for 10.146.0.10
>     DHCP 0x120e44 DHCPREQUEST to 169.254.169.254:67 for 10.146.0.10
>     .DHCP 0x120e44 DHCPREQUEST to 169.254.169.254:67 for 10.146.0.10
>     .DHCP 0x120e44 DHCPREQUEST to 169.254.169.254:67 for 10.146.0.10
>     ...DHCP 0x120e44 DHCPREQUEST to 169.254.169.254:67 for 10.146.0.10
>     ....... No configuration methods succeeded (http://ipxe.org/040ee119)
>     iPXE>
>
> I haven't captured DHCP packets (because I'm unsure how to do so in GCE's
> fully managed DHCP server), but I guess DHCPACK is actually sent from the
> DHCP server but somehow iPXE missed it.

Do you see any interesting log output with DEBUG=virtio-net,virtio-pci ?

> I'd appreciate if anyone can support me to set MAX_QUEUE_NUM to 4096.

I'll look into it. My proposal is to keep using 128 as the cap in
virtio 1.0 and support any queue size in legacy, only subject to
available memory.

> 2016-12-14 16:19 GMT+09:00 Christian Nilsson <nikize at gmail.com>:
>>
>> You might want to report this to google as well if you haven't already.
>>
>> On Wed, Dec 14, 2016 at 7:16 AM, Akihiro Suda <suda.kyoto at gmail.com>
>> wrote:
>> > Hello,
>> >
>> > I tried to boot iPXE on a Google Compute Engine instance, but the virtio
>> > driver doesn't work because virtnet_open_legacy() fails to find
>> > virtqueues.
>> >
>> > Please let me know what I can do for providing further information to
>> > debug
>> > the issue.
>> >
>> > The serial console output with "DEBUG=virtio-net" is as follows:
>> >
>> >     iPXE initialising devices...Common virtio capability not found!
>> >     VIRTIO-NET 0xdf584 busaddr=0000:00:04.0 ioaddr=0xc040 irq=11
>> >     VIRTIO-NET 0xdf584 mac=42:01:0a:92:00:0a
>> >     ok
>> >
>> >     iPXE 1.0.0+ (26050) -- Open Source Network Boot Firmware --
>> > http://ipxe.org
>> >     Features: DNS HTTP iSCSI TFTP SRP AoE ELF MBOOT PXE bzImage Menu
>> > PXEXT
>> >     iPXE> ifstat
>> >     net0: 42:01:0a:92:00:0a using virtio-net on 0000:00:04.0 (closed)
>> >       [Link:up, TX:0 TXE:0 RX:0 RXE:0]
>> >     iPXE> dhcp
>> >     VIRTIO-NET 0xdf584 cannot register queue 0
>> >     Could not open net0: No such file or directory
>> > (http://ipxe.org/2d5c403b)
>> >     iPXE>
>> >
>> >
>> > This ENOENT is raised from virtnet_open_legacy():
>> >
>> > https://git.ipxe.org/ipxe.git/blob/26050fd4c87c50503d5bd573b2ec91703676e211:/src/drivers/net/virtio-net.c#l205
>> >
>> >
>> > I also tested some previous commits, but all of them failed, so the
>> > issue is
>> > not a regression in some recent commit:
>> >
>> >  * 26050fd (Dec  8, 2016): the last commit when I tested
>> >  * 7b499f8 (Apr 11, 2016): the last commit without support for virtio
>> > 1.0
>> >  * e4419ff (Jul  2, 2010):  the first commit with the "native iPXE
>> > driver"
>> > (The commit date is older than 232c208, but actually newer than 232c208)
>> >  * 232c208 (Jul 11, 2010): the last commit without the "native iPXE
>> > driver".
>> > The error message is "ERROR: queue size 4096 > 512", "Cannot register
>> > queue
>> > #0"
>> >
>> >
>> > Regards,
>> >
>> > Akihiro Suda
>> >
>> >
>> > _______________________________________________
>> > ipxe-devel mailing list
>> > ipxe-devel at lists.ipxe.org
>> > https://lists.ipxe.org/mailman/listinfo.cgi/ipxe-devel
>> >
>
>
>
> _______________________________________________
> ipxe-devel mailing list
> ipxe-devel at lists.ipxe.org
> https://lists.ipxe.org/mailman/listinfo.cgi/ipxe-devel
>