<div dir="ltr">Hello, Ladi,<div><br></div><div>Thank you a lot for the analysis.</div><div><br></div><div>> <span style="color:rgb(0,0,0);font-size:13.3333px">At this point we should engage somebody from Google to help us understand why the host behaves this way, i.e. why it doesn't let us use the device in "simple" mode with nothing but VIRTIO_NET_F_MAC.</span></div><div><span style="color:rgb(0,0,0);font-size:13.3333px"><br></span></div><div>This site seems the right place for that: <a href="https://googlecloudplatform.uservoice.com/forums/302595-compute-engine/filters/top">https://googlecloudplatform.uservoice.com/forums/302595-compute-engine/filters/top</a> <br></div><div><br></div><div>Since you have a better understanding of the issue than me, can you please consider making a post to that place?</div><div><br></div><div>Thank you again</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">2016-12-21 1:28 GMT+09:00 Ladi Prosek <span dir="ltr"><<a href="mailto:lprosek@redhat.com" target="_blank">lprosek@redhat.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:monospace,monospace"><span style="font-family:arial,sans-serif">Hi Akihiro,</span></div><span class=""><div class="gmail_default" style="font-family:monospace,monospace"><span style="font-family:arial,sans-serif"><br></span></div><div class="gmail_default" style="font-family:monospace,monospace"><span style="font-family:arial,sans-serif">On Tue, Dec 20, 2016 at 2:58 AM, Akihiro Suda </span><span dir="ltr" style="font-family:arial,sans-serif"><<a href="mailto:suda.kyoto@gmail.com" target="_blank">suda.kyoto@gmail.com</a>></span><span style="font-family:arial,sans-serif"> wrote:</span><br></div></span><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hello, Ladi,<span class="m_-3982545736899824438gmail-m_1089263003359519948gmail-m_443485469150077857gmail-m_6255584689611316807m_-1148929363715887492gmail-m_-8888738478967729338gmail-"><div><br></div><span class=""><div>> <span style="font-family:monospace,monospace;font-size:21px">Could you please help me set up a GCE instance with iPXE? I haven't been able to make it produce any output. Do you build iPXE with CONSOLE_SERIAL? How do you create the .tar.gz for GCE to import the image from? Thanks!</span></div><div><span style="font-family:monospace,monospace;font-size:21px"><br></span></div></span></span><span class=""><div><span style="font-family:monospace,monospace;font-size:21px">Yes, I built the image with CONSOLE_SERIAL.</span></div><div><span style="font-family:monospace,monospace;font-size:21px">For creating .tar.gz, I did:</span></div><div><span style="font-family:monospace,monospace;font-size:21px"><br></span></div><div><div><font face="monospace, monospace"><span style="font-size:21px"> cp bin/ipxe.usb /tmp/disk.raw</span></font></div><div><font face="monospace, monospace"><span style="font-size:21px"> ( cd /tmp; tar Sczvf ipxe.tar.gz disk.raw )</span></font></div><div><font face="monospace, monospace"><span style="font-size:21px"> gsutil cp /tmp/ipxe.tar.gz gs://blahblahblah</span></font></div></div><div><span style="font-family:monospace,monospace;font-size:21px"><br></span></div><div><span style="font-family:monospace,monospace;font-size:21px">Also, for clarity, I attached a complete script I'm using.</span></div><div><br></div><div><span style="font-family:monospace,monospace;font-size:21px">Thank you for looking into this!</span></div><div><br></div></span></div></blockquote><div><br></div><div><div class="gmail_default" style="font-family:monospace,monospace">Many thanks for the script, it made testing so much easier!</div><div class="gmail_default" style="font-family:monospace,monospace"><br></div><div class="gmail_default"><font face="monospace, monospace">The problem is that GCE hosts use advanced virtio-net features like host-side checksum offload (</font><span style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0);font-size:13.3333px">VIRTIO_NET_F_CSUM) even if they are not accepted by the iPXE driver. The driver uses only one static instance of the virtio-net header in </span><font color="#000000" face="arial, helvetica, sans-serif"><span style="font-size:13.3333px">virtnet_nic.</span></font><font color="#000000" face="monospace"><span style="font-size:13.3333px">empty_header for both tx and rx buffers. The rationale is that without advanced features, the header will never be modified and will stay all zeros so it can be shared. Here's the comment in virtio-net.c:</span></font></div><div class="gmail_default"><font color="#000000" face="monospace"><span style="font-size:13.3333px"><br></span></font></div><div class="gmail_default"><font color="#000000"><div class="gmail_default" style="font-size:13.3333px;font-family:monospace"><span class="m_-3982545736899824438gmail-m_1089263003359519948gmail-m_443485469150077857gmail-m_6255584689611316807m_-1148929363715887492gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>/* Share a single zeroed virtio net header between all</div><div class="gmail_default" style="font-size:13.3333px;font-family:monospace"><span class="m_-3982545736899824438gmail-m_1089263003359519948gmail-m_443485469150077857gmail-m_6255584689611316807m_-1148929363715887492gmail-Apple-tab-span" style="white-space:pre-wrap"> </span> * rx and tx packets. This works because this driver</div><div class="gmail_default" style="font-size:13.3333px;font-family:monospace"><span class="m_-3982545736899824438gmail-m_1089263003359519948gmail-m_443485469150077857gmail-m_6255584689611316807m_-1148929363715887492gmail-Apple-tab-span" style="white-space:pre-wrap"> </span> * does not use any advanced features so none of the</div><div class="gmail_default" style="font-size:13.3333px;font-family:monospace"><span class="m_-3982545736899824438gmail-m_1089263003359519948gmail-m_443485469150077857gmail-m_6255584689611316807m_-1148929363715887492gmail-Apple-tab-span" style="white-space:pre-wrap"> </span> * header fields get used.</div><div class="gmail_default" style="font-size:13.3333px;font-family:monospace"><span class="m_-3982545736899824438gmail-m_1089263003359519948gmail-m_443485469150077857gmail-m_6255584689611316807m_-1148929363715887492gmail-Apple-tab-span" style="white-space:pre-wrap"> </span> */</div><div style="font-size:13.3333px;font-family:monospace"><br></div><div style="font-size:13.3333px;font-family:monospace">But I see this buffer written to by the host. After receiving a DHCP packet, the flags field is updated to 0x2 (VIRTIO_NET_HDR_F_DATA_VALID), and receiving non-UDP non-TCP traffic resets it back to 0x0. This explains why pinging the instance helps. Without pinging, the first packet is sent ok, the first received packed sets <span style="font-size:13.3333px">VIRTIO_NET_HDR_F_DATA_VAL<wbr>ID, and subsequent sends fail because the flag stays set which is illegal (virtio spec </span><span class="m_-3982545736899824438gmail-m_1089263003359519948gmail-m_443485469150077857gmail-m_6255584689611316807m_-1148929363715887492gmail-titlemark" style="color:rgb(85,38,129);font-family:arial,helvetica,sans-serif;font-size:14.6667px">5.1.6.2.1</span><span style="color:rgb(85,38,129);font-family:arial,helvetica,sans-serif;font-size:14.6667px"> </span><a id="m_-3982545736899824438gmail-m_1089263003359519948gmail-m_443485469150077857gmail-m_6255584689611316807m_-1148929363715887492gmail-x1-1710001" style="font-family:arial,helvetica,sans-serif;font-size:14.6667px"></a><span style="color:rgb(85,38,129);font-family:arial,helvetica,sans-serif;font-size:14.6667px">Driver Requirements: Packet Transmission "</span><span style="font-family:arial,helvetica,sans-serif;font-size:13.3333px">The driver MUST NOT set the VIRTIO_NET_HDR_F_DATA_VALID bit in </span><span class="m_-3982545736899824438gmail-m_1089263003359519948gmail-m_443485469150077857gmail-m_6255584689611316807m_-1148929363715887492gmail-aeti-10" style="font-style:italic;font-family:arial,helvetica,sans-serif;font-size:13.3333px">flags</span><span style="font-family:arial,helvetica,sans-serif;font-size:13.3333px">."). </span><span style="font-family:arial,helvetica,sans-serif;font-size:13.3333px">With pinging, the incoming ICMP packets keep resetting the flags field back to 0x0, making successful transmission possible.</span></div><div style="font-size:13.3333px;font-family:monospace"><span style="font-family:arial,helvetica,sans-serif;font-size:13.3333px"><br></span></div><div style="font-size:13.3333px"><font face="arial, helvetica, sans-serif">Even if we allocate separate headers for each packet to try to work around this, we'd be running into checksumming issues with UDP and TCP, because the host expects us to use its precomputed partial checksums.</font></div><div style="font-size:13.3333px"><font face="arial, helvetica, sans-serif"><br></font></div><div><div><span style="font-size:13.3333px">virtio_net_hdr</span></div><div><span style="font-size:13.3333px">{</span></div><div><span style="font-size:13.3333px"> flags = </span><span style="font-size:13.3333px">VIRTIO_NET_HDR_F_NEEDS_CSUM</span><br></div><div><span style="font-size:13.3333px"> gso_type = </span><span style="font-size:13.3333px">VIRTIO_NET_HDR_GSO_TCPV4</span><br></div><div><span style="font-size:13.3333px"> hdr_len = 0x42</span></div><div><span style="font-size:13.3333px">...</span></div></div><div><span style="font-size:13.3333px">}</span></div><div><span style="font-size:13.3333px"><br></span></div><div><div><span style="font-size:13.3333px">IPv4 RX 10.132.0.2<-90.130.70.73 len 1500 (1500) proto 6 id 8f25 csum 00a6</span></div><div><span style="font-size:13.3333px">TCP checksum incorrect (is 3268 including checksum field, should be 0000)</span></div><div style="font-size:13.3333px"><br></div></div><div style="font-size:13.3333px">I have also tried reordering the virtio initialization steps in virtnet_open_legacy hoping that it would help, but no joy there either.</div><div style="font-size:13.3333px"><br></div><div style="font-size:13.3333px">At this point we should engage somebody from Google to help us understand why the host behaves this way, i.e. why it doesn't let us use the device in "simple" mode with nothing but VIRTIO_NET_F_MAC.</div><div style="font-size:13.3333px"><br></div></font></div></div><div><div class="gmail_default" style="font-family:monospace,monospace">Thanks,</div><div class="gmail_default" style="font-family:monospace,monospace">Ladi</div><br></div><div><div class="h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div></div></div><div class="m_-3982545736899824438gmail-m_1089263003359519948gmail-m_443485469150077857gmail-m_6255584689611316807m_-1148929363715887492gmail-m_-8888738478967729338gmail-HOEnZb"><div class="m_-3982545736899824438gmail-m_1089263003359519948gmail-m_443485469150077857gmail-m_6255584689611316807m_-1148929363715887492gmail-m_-8888738478967729338gmail-h5"><div class="gmail_extra"><div class="gmail_quote">2016-12-20 1:21 GMT+09:00 Ladi Prosek <span dir="ltr"><<a href="mailto:lprosek@redhat.com" target="_blank">lprosek@redhat.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><span><div style="font-family:monospace,monospace"><span style="font-family:arial,sans-serif">On Mon, Dec 19, 2016 at 10:40 AM, Akihiro Suda </span><span dir="ltr" style="font-family:arial,sans-serif"><<a href="mailto:suda.kyoto@gmail.com" target="_blank">suda.kyoto@gmail.com</a>></span><span style="font-family:arial,sans-serif"> wrote:</span><br></div></span><div class="gmail_extra"><div class="gmail_quote"><span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hello, Ladi,</div><div><br></div><div>Thank you a lot for providing the patch.</div><div>However, unfortunately, it does not seem to have any effect.</div><div><br></div><div>BTW, when I was tackling with the issue without your patch, I found some interesting behavior:</div><div><br></div><div>- Pinging from another instance (Linux) to an iPXE instance seems always working as expected. (i.e. iPXE can successfully "pong" by transmitting Ethernet packets)</div><div>- When an iPXE instance is being pinged, it can successfully send non-"pong" packets as well. So `dhcp` and `imgfetch <a href="http://10.146.0.2/a.ipxe" target="_blank">http://10.146.0.2/a.ipxe`</a> work!</div><div><br></div><div>So, I guess the issue is somehow related to interruption. </div><div>When iPXE is frequently interrupted due to RX, the interruptions seems triggering TX as well.</div><div>However, TX seems failing when it is not triggered by frequent RX interruptions.</div><div>(Please correct me if my guess is wrong 😅)</div><div><br></div></div></blockquote><div><br></div></span><div><div style="font-family:monospace,monospace">iPXE doesn't rely on interrupts, virtqueues are polled in a loop. The log you posted suggests that the host is really transmitting, or at least accepting outbound packets for transmission. I'm not sure what guest driver issue would explain this behavior :(</div><div style="font-family:monospace,monospace"><br></div><div style="font-family:monospace,monospace">Could you please help me set up a GCE instance with iPXE? I haven't been able to make it produce any output. Do you build iPXE with CONSOLE_SERIAL? How do you create the .tar.gz for GCE to import the image from? Thanks!</div><br></div><div><div class="m_-3982545736899824438gmail-m_1089263003359519948gmail-m_443485469150077857gmail-m_6255584689611316807m_-1148929363715887492gmail-m_-8888738478967729338gmail-m_-6110462976158404824h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div></div></div><div class="m_-3982545736899824438gmail-m_1089263003359519948gmail-m_443485469150077857gmail-m_6255584689611316807m_-1148929363715887492gmail-m_-8888738478967729338gmail-m_-6110462976158404824m_-8764367700832313336gmail-HOEnZb"><div class="m_-3982545736899824438gmail-m_1089263003359519948gmail-m_443485469150077857gmail-m_6255584689611316807m_-1148929363715887492gmail-m_-8888738478967729338gmail-m_-6110462976158404824m_-8764367700832313336gmail-h5"><div class="gmail_extra"><br><div class="gmail_quote">2016-12-19 18:00 GMT+09:00 Ladi Prosek <span dir="ltr"><<a href="mailto:lprosek@redhat.com" target="_blank">lprosek@redhat.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Akihiro,<br>
<div><div class="m_-3982545736899824438gmail-m_1089263003359519948gmail-m_443485469150077857gmail-m_6255584689611316807m_-1148929363715887492gmail-m_-8888738478967729338gmail-m_-6110462976158404824m_-8764367700832313336gmail-m_600783110557361012h5"><br>
On Mon, Dec 19, 2016 at 7:17 AM, Akihiro Suda <<a href="mailto:suda.kyoto@gmail.com" target="_blank">suda.kyoto@gmail.com</a>> wrote:<br>
> Hello Ladi,<br>
><br>
> Thank you a lot for looking into this issue and writing patches about the<br>
> queue size. The patch set looks good to me.<br>
><br>
>>> I haven't captured DHCP packets (because I'm unsure how to do so in GCE's<br>
>>> fully managed DHCP server), but I guess DHCPACK is actually sent from the<br>
>>> DHCP server but somehow iPXE missed it.<br>
>><br>
>> Do you see any interesting log output with DEBUG=virtio-net,virtio-pci ?<br>
><br>
> Did not see any interesting log. No error about TX is printed, and it just<br>
> fails to receive DHCPACK.<br>
><br>
> So, for ease of debugging with capturing packets, I tried to do `imgfetch<br>
> http://...` with a static IP address (`set net0/ip`) instead.<br>
><br>
> The result implies that "iPXE believes it has sent the second Ethernet<br>
> packet without an error, but it is not sent actually".<br>
><br>
> This assumption is consistent with my report about DHCP.<br>
> iPXE can successfully send the 1st packet (DHCPDISCOVER), but seems failing<br>
> to send the 2nd packet (DHCPREQUEST). It should be the reason why it cannot<br>
> receive DHCPACK.<br>
><br>
><br>
> Here is the log. In the client log, we can see "TX ... PSH ACK", but it is<br>
> not received on the server side.<br>
><br>
> === iPXE Client 10.146.0.10 (DEBUG=virtio-net:7,virtio-pci<wbr>:7,tcp:7,ipv4:7<br>
> CONFIG=cloud, 26050fd4 with Ladi's 4 great patches) ===<br>
><br>
> iPXE> imgfetch <a href="http://10.146.0.2/a.ipxe" rel="noreferrer" target="_blank">http://10.146.0.2/a.ipxe</a><br>
> <a href="http://10.146.0.2/a.ipxe...TCP" rel="noreferrer" target="_blank">http://10.146.0.2/a.ipxe...TC<wbr>P</a> 0x12eda4 allocated<br>
> TCP 0x12eda4 transitioned from CLOSED to SYN_SENT<br>
> TCP 0x12eda4 bound to port 46673<br>
> TCP 0x12eda4 timer fired in SYN_SENT for 609cb55e..609cb55e 00000000<br>
> TCP 0x12eda4 TX 46673->80 609cb55e..609cb55f 00000000 0 SYN<br>
> IPv4 TX 10.146.0.10->10.146.0.2 len 64 proto 6 id 0100 csum 6489<br>
> VIRTIO-NET 0xee704 enqueuing iobuf 0x12ef90 on vq 1<br>
> VIRTIO-NET 0xee704 tx complete iobuf 0x12ef90<br>
> VIRTIO-NET 0xee704 rx complete iobuf 0x12f5f4 len 74<br>
> VIRTIO-NET 0xee704 enqueuing iobuf 0x1335f4 on vq 0<br>
> IPv4 RX 10.146.0.10<-10.146.0.2 len 60 proto 6 id 0000 csum 258d<br>
> TCP 0x12eda4 RX 46673<-80 609cb55f 5ae4dacb..5ae4dacc 0 SYN<br>
> ACK<br>
> TCP 0x12eda4 using timestamps, SACK, TX window x128, RX window x512<br>
> TCP 0x12eda4 transitioned from SYN_SENT to ESTABLISHED<br>
> TCP 0x12eda4 TX 46673->80 609cb55f..609cb5c1 5ae4dacc 98 PSH<br>
> ACK<br>
> IPv4 TX 10.146.0.10->10.146.0.2 len 150 proto 6 id 0201 csum 6332<br>
> VIRTIO-NET 0xee704 enqueuing iobuf 0x12ec94 on vq 1<br>
> VIRTIO-NET 0xee704 tx complete iobuf 0x12ec94<br>
> VIRTIO-NET 0xee704 rx complete iobuf 0x12fdf4 len 74<br>
> VIRTIO-NET 0xee704 enqueuing iobuf 0x12f5f4 on vq 0<br>
> IPv4 RX 10.146.0.10<-10.146.0.2 len 60 proto 6 id 0000 csum 258d<br>
> TCP 0x12eda4 RX 46673<-80 609cb55f 5ae4dacb..5ae4dacc 0 SYN<br>
> ACK<br>
> TCP 0x12eda4 timer fired in ESTABLISHED for 609cb55f..609cb5c1 5ae4dacc<br>
> TCP 0x12eda4 TX 46673->80 609cb55f..609cb5c1 5ae4dacc 98 PSH<br>
> ACK<br>
> IPv4 TX 10.146.0.10->10.146.0.2 len 150 proto 6 id 0302 csum 6231<br>
> VIRTIO-NET 0xee704 enqueuing iobuf 0x12ec94 on vq 1<br>
> VIRTIO-NET 0xee704 tx complete iobuf 0x12ec94<br>
> .TCP 0x12eda4 timer fired in ESTABLISHED for 609cb55f..609cb5c1 5ae4dacc<br>
> TCP 0x12eda4 TX 46673->80 609cb55f..609cb5c1 5ae4dacc 98 PSH<br>
> ACK<br>
> IPv4 TX 10.146.0.10->10.146.0.2 len 150 proto 6 id 0402 csum 6131<br>
> VIRTIO-NET 0xee704 enqueuing iobuf 0x12ec94 on vq 1<br>
> VIRTIO-NET 0xee704 tx complete iobuf 0x12ec94<br>
> VIRTIO-NET 0xee704 rx complete iobuf 0x1305f4 len 74<br>
> VIRTIO-NET 0xee704 enqueuing iobuf 0x12fdf4 on vq 0<br>
> IPv4 RX 10.146.0.10<-10.146.0.2 len 60 proto 6 id 0000 csum 258d<br>
> TCP 0x12eda4 RX 46673<-80 609cb55f 5ae4dacb..5ae4dacc 0 SYN<br>
> ACK<br>
> ..TCP 0x12eda4 timer fired in ESTABLISHED for 609cb55f..609cb5c1<br>
> 5ae4dacc<br>
> TCP 0x12eda4 TX 46673->80 609cb55f..609cb5c1 5ae4dacc 98 PSH<br>
> ACK<br>
> IPv4 TX 10.146.0.10->10.146.0.2 len 150 proto 6 id 0503 csum 6030<br>
> VIRTIO-NET 0xee704 enqueuing iobuf 0x12ec94 on vq 1<br>
> VIRTIO-NET 0xee704 tx complete iobuf 0x12ec94<br>
> ..VIRTIO-NET 0xee704 rx complete iobuf 0x130df4 len 74<br>
> VIRTIO-NET 0xee704 enqueuing iobuf 0x1305f4 on vq 0<br>
> IPv4 RX 10.146.0.10<-10.146.0.2 len 60 proto 6 id 0000 csum 258d<br>
> TCP 0x12eda4 RX 46673<-80 609cb55f 5ae4dacb..5ae4dacc 0 SYN<br>
> ACK<br>
> .TCP 0x12eda4 timer fired in ESTABLISHED for 609cb55f..609cb5c1 5ae4dacc<br>
> TCP 0x12eda4 TX 46673->80 609cb55f..609cb5c1 5ae4dacc 98 PSH<br>
> ACK<br>
> IPv4 TX 10.146.0.10->10.146.0.2 len 150 proto 6 id 0604 csum 5f2f<br>
> VIRTIO-NET 0xee704 enqueuing iobuf 0x12ec94 on vq 1<br>
> VIRTIO-NET 0xee704 tx complete iobuf 0x12ec94<br>
> ......TCP 0x12eda4 timer expired in ESTABLISHED for 609cb55f..609cb5c1<br>
> 5ae4dacc<br>
> TCP 0x12eda4 transitioned from ESTABLISHED to CLOSED<br>
> TCP 0x12eda4 connection deleted<br>
> Connection timed out (<a href="http://ipxe.org/4c0a6035" rel="noreferrer" target="_blank">http://ipxe.org/4c0a6035</a>)<br>
> iPXE> VIRTIO-NET 0xee704 rx complete iobuf 0x1315f4 len 74<br>
> VIRTIO-NET 0xee704 enqueuing iobuf 0x130df4 on vq 0<br>
> IPv4 RX 10.146.0.10<-10.146.0.2 len 60 proto 6 id 0000 csum 258d<br>
> TCP 0x0 RX 46673<-80 609cb55f 5ae4dacb..5ae4dacc 0 SYN ACK<br>
> IPv4 received packet rejected by stack: The socket is not connected<br>
> (<a href="http://ipxe.org/380a6001" rel="noreferrer" target="_blank">http://ipxe.org/380a6001</a>)<br>
><br>
> iPXE> VIRTIO-NET 0xee704 rx complete iobuf 0x131df4 len 74<br>
> VIRTIO-NET 0xee704 enqueuing iobuf 0x1315f4 on vq 0<br>
> IPv4 RX 10.146.0.10<-10.146.0.2 len 60 proto 6 id 0000 csum 258d<br>
> TCP 0x0 RX 46673<-80 609cb55f 5ae4dacb..5ae4dacc 0 SYN ACK<br>
> IPv4 received packet rejected by stack: The socket is not connected<br>
> (<a href="http://ipxe.org/380a6001" rel="noreferrer" target="_blank">http://ipxe.org/380a6001</a>)<br>
> VIRTIO-NET 0xee704 rx complete iobuf 0x1325f4 len 42<br>
> VIRTIO-NET 0xee704 enqueuing iobuf 0x131df4 on vq 0<br>
> VIRTIO-NET 0xee704 enqueuing iobuf 0x1325f4 on vq 1<br>
> VIRTIO-NET 0xee704 tx complete iobuf 0x1325f4<br>
> VIRTIO-NET 0xee704 rx complete iobuf 0x132df4 len 62<br>
> VIRTIO-NET 0xee704 enqueuing iobuf 0x1325f4 on vq 0<br>
> IPv4 RX 10.146.0.10<-221.204.224.56 len 48 proto 6 id 6aac csum 167b<br>
> TCP 0x0 RX 22<-2281 0a4bb0cf 3368312b..3368312c 0 SYN<br>
> IPv4 received packet rejected by stack: The socket is not connected<br>
> (<a href="http://ipxe.org/380a6001" rel="noreferrer" target="_blank">http://ipxe.org/380a6001</a>)<br>
> VIRTIO-NET 0xee704 rx complete iobuf 0x1335f4 len 42<br>
> VIRTIO-NET 0xee704 enqueuing iobuf 0x132df4 on vq 0<br>
> VIRTIO-NET 0xee704 enqueuing iobuf 0x1335f4 on vq 1<br>
> VIRTIO-NET 0xee704 tx complete iobuf 0x1335f4<br>
><br>
><br>
><br>
> === HTTP server 10.146.0.2 (ws01) ===<br>
><br>
> $ sudo tcpdump host 10.146.0.10<br>
> tcpdump: verbose output suppressed, use -v or -vv for full protocol<br>
> decode<br>
> listening on ens4, link-type EN10MB (Ethernet), capture size 262144<br>
> bytes<br>
> 04:55:43.715483 IP ipxe01.c.suda-main.internal.46<wbr>673 ><br>
> ws01.c.suda-main.internal.http<wbr>: Flags [S], seq 1620882782, win 65532,<br>
> options [nop,nop,TS val 323011 ecr 0,nop,nop,sackOK,nop,wscale 9,mss 1460],<br>
> length 0<br>
> 04:55:43.715549 IP ws01.c.suda-main.internal.http ><br>
> ipxe01.c.suda-main.internal.46<wbr>673: Flags [S.], seq 1524947659, ack<br>
> 1620882783, win 28160, options [mss 1420,sackOK,TS val 77061640 ecr<br>
> 323011,nop,wscale 7], length 0<br>
> 04:55:44.712811 IP ws01.c.suda-main.internal.http ><br>
> ipxe01.c.suda-main.internal.46<wbr>673: Flags [S.], seq 1524947659, ack<br>
> 1620882783, win 28160, options [mss 1420,sackOK,TS val 77061890 ecr<br>
> 323011,nop,wscale 7], length 0<br>
> 04:55:46.712829 IP ws01.c.suda-main.internal.http ><br>
> ipxe01.c.suda-main.internal.46<wbr>673: Flags [S.], seq 1524947659, ack<br>
> 1620882783, win 28160, options [mss 1420,sackOK,TS val 77062390 ecr<br>
> 323011,nop,wscale 7], length 0<br>
> 04:55:50.712864 IP ws01.c.suda-main.internal.http ><br>
> ipxe01.c.suda-main.internal.46<wbr>673: Flags [S.], seq 1524947659, ack<br>
> 1620882783, win 28160, options [mss 1420,sackOK,TS val 77063390 ecr<br>
> 323011,nop,wscale 7], length 0<br>
> 04:55:58.712842 IP ws01.c.suda-main.internal.http ><br>
> ipxe01.c.suda-main.internal.46<wbr>673: Flags [S.], seq 1524947659, ack<br>
> 1620882783, win 28160, options [mss 1420,sackOK,TS val 77065390 ecr<br>
> 323011,nop,wscale 7], length 0<br>
> 04:56:14.712836 IP ws01.c.suda-main.internal.http ><br>
> ipxe01.c.suda-main.internal.46<wbr>673: Flags [S.], seq 1524947659, ack<br>
> 1620882783, win 28160, options [mss 1420,sackOK,TS val 77069390 ecr<br>
> 323011,nop,wscale 7], length 0<br>
<br>
</div></div>Thanks for the logs. Can you please try the following patch? It's a<br>
crazy workaround to reinitialize the driver on each transmitted<br>
packet. If it works, we'll go from there and try something more<br>
sophisticated. Thanks!<br>
<br>
<br>
--- a/src/drivers/net/virtio-net.c<br>
+++ b/src/drivers/net/virtio-net.c<br>
@@ -345,6 +345,8 @@ static void virtnet_close ( struct net_device *netdev ) {<br>
*/<br>
static int virtnet_transmit ( struct net_device *netdev,<br>
struct io_buffer *iobuf ) {<br>
+ virtnet_close ( netdev );<br>
+ virtnet_open ( netdev );<br>
virtnet_enqueue_iob ( netdev, TX_INDEX, iobuf );<br>
return 0;<br>
<div class="m_-3982545736899824438gmail-m_1089263003359519948gmail-m_443485469150077857gmail-m_6255584689611316807m_-1148929363715887492gmail-m_-8888738478967729338gmail-m_-6110462976158404824m_-8764367700832313336gmail-m_600783110557361012HOEnZb"><div class="m_-3982545736899824438gmail-m_1089263003359519948gmail-m_443485469150077857gmail-m_6255584689611316807m_-1148929363715887492gmail-m_-8888738478967729338gmail-m_-6110462976158404824m_-8764367700832313336gmail-m_600783110557361012h5"> }<br>
<br>
<br>
> 2016-12-14 18:17 GMT+09:00 Ladi Prosek <<a href="mailto:lprosek@redhat.com" target="_blank">lprosek@redhat.com</a>>:<br>
>><br>
>> Hi Akihiro,<br>
>><br>
>> On Wed, Dec 14, 2016 at 8:51 AM, Akihiro Suda <<a href="mailto:suda.kyoto@gmail.com" target="_blank">suda.kyoto@gmail.com</a>><br>
>> wrote:<br>
>> > Hi Christian,<br>
>> ><br>
>> > Thank you for the suggestion, I'll try to report this to google.<br>
>> ><br>
>> > I found that the cause of this issue is because GCE's<br>
>> > VIRTIO_PCI_QUEUE_NUM<br>
>> > is 4096, which is larger than iPXE's MAX_QUEUE_NUM (256).<br>
>> ><br>
>> ><br>
>> > <a href="https://git.ipxe.org/ipxe.git/blob/26050fd4c87c50503d5bd573b2ec91703676e211:/src/drivers/bus/virtio-pci.c#l43" rel="noreferrer" target="_blank">https://git.ipxe.org/ipxe.git/<wbr>blob/26050fd4c87c50503d5bd573b<wbr>2ec91703676e211:/src/drivers/b<wbr>us/virtio-pci.c#l43</a><br>
>> ><br>
>> > <a href="https://git.ipxe.org/ipxe.git/blob/26050fd4c87c50503d5bd573b2ec91703676e211:/src/include/ipxe/virtio-ring.h#l24" rel="noreferrer" target="_blank">https://git.ipxe.org/ipxe.git/<wbr>blob/26050fd4c87c50503d5bd573b<wbr>2ec91703676e211:/src/include/i<wbr>pxe/virtio-ring.h#l24</a><br>
>> ><br>
>> ><br>
>> > IIUC this issue will be resolved if GCE supports virtio 1.0, which would<br>
>> > allow iPXE to use custom queue num smaller than the virtio device's max<br>
>> > value.<br>
>><br>
>> iPXE virtio 1.0 code does not cap the queue size to MAX_QUEUE_NUM. In<br>
>> fact, it does not check it against any maximum and still uses<br>
>> fixed-sized virtqueue data structures, just like legacy. This is a<br>
>> bug.<br>
>><br>
>> > However, it would be great if iPXE can support setting MAX_QUEUE_NUM to<br>
>> > 4096<br>
>> > (via `config/general.h`, probably).<br>
>><br>
>> Even better, iPXE should try to use the host provided size in legacy<br>
>> mode, i.e. no MAX_QUEUE_NUM needed. The few extra dynamic allocations<br>
>> shouldn't hurt anything.<br>
>><br>
>> > Actually I tried to set MAX_QUEUE_NUM to 4096 and got<br>
>> > virtnet_open_legacy()<br>
>> > succeeded, but it still doesn't work:<br>
>> ><br>
>> > iPXE> dhcp<br>
>> > DHCP 0x120e44 entering discovery state<br>
>> > Configuring (net0 42:01:0a:92:00:0a)...DHCP 0x120e44 DHCPDISCOVER<br>
>> > DHCP 0x120e44 DHCPOFFER from <a href="http://169.254.169.254:67" rel="noreferrer" target="_blank">169.254.169.254:67</a> for 10.146.0.10<br>
>> > .DHCP 0x120e44 DHCPDISCOVER<br>
>> > ..DHCP 0x120e44 entering request state<br>
>> > DHCP 0x120e44 DHCPREQUEST to <a href="http://169.254.169.254:67" rel="noreferrer" target="_blank">169.254.169.254:67</a> for 10.146.0.10<br>
>> > DHCP 0x120e44 DHCPREQUEST to <a href="http://169.254.169.254:67" rel="noreferrer" target="_blank">169.254.169.254:67</a> for 10.146.0.10<br>
>> > .DHCP 0x120e44 DHCPREQUEST to <a href="http://169.254.169.254:67" rel="noreferrer" target="_blank">169.254.169.254:67</a> for 10.146.0.10<br>
>> > .DHCP 0x120e44 DHCPREQUEST to <a href="http://169.254.169.254:67" rel="noreferrer" target="_blank">169.254.169.254:67</a> for 10.146.0.10<br>
>> > ...DHCP 0x120e44 DHCPREQUEST to <a href="http://169.254.169.254:67" rel="noreferrer" target="_blank">169.254.169.254:67</a> for 10.146.0.10<br>
>> > ....... No configuration methods succeeded<br>
>> > (<a href="http://ipxe.org/040ee119" rel="noreferrer" target="_blank">http://ipxe.org/040ee119</a>)<br>
>> > iPXE><br>
>> ><br>
>> > I haven't captured DHCP packets (because I'm unsure how to do so in<br>
>> > GCE's<br>
>> > fully managed DHCP server), but I guess DHCPACK is actually sent from<br>
>> > the<br>
>> > DHCP server but somehow iPXE missed it.<br>
>><br>
>> Do you see any interesting log output with DEBUG=virtio-net,virtio-pci ?<br>
>><br>
>> > I'd appreciate if anyone can support me to set MAX_QUEUE_NUM to 4096.<br>
>><br>
>> I'll look into it. My proposal is to keep using 128 as the cap in<br>
>> virtio 1.0 and support any queue size in legacy, only subject to<br>
>> available memory.<br>
>><br>
>> > 2016-12-14 16:19 GMT+09:00 Christian Nilsson <<a href="mailto:nikize@gmail.com" target="_blank">nikize@gmail.com</a>>:<br>
>> >><br>
>> >> You might want to report this to google as well if you haven't already.<br>
>> >><br>
>> >> On Wed, Dec 14, 2016 at 7:16 AM, Akihiro Suda <<a href="mailto:suda.kyoto@gmail.com" target="_blank">suda.kyoto@gmail.com</a>><br>
>> >> wrote:<br>
>> >> > Hello,<br>
>> >> ><br>
>> >> > I tried to boot iPXE on a Google Compute Engine instance, but the<br>
>> >> > virtio<br>
>> >> > driver doesn't work because virtnet_open_legacy() fails to find<br>
>> >> > virtqueues.<br>
>> >> ><br>
>> >> > Please let me know what I can do for providing further information to<br>
>> >> > debug<br>
>> >> > the issue.<br>
>> >> ><br>
>> >> > The serial console output with "DEBUG=virtio-net" is as follows:<br>
>> >> ><br>
>> >> > iPXE initialising devices...Common virtio capability not found!<br>
>> >> > VIRTIO-NET 0xdf584 busaddr=0000:00:04.0 ioaddr=0xc040 irq=11<br>
>> >> > VIRTIO-NET 0xdf584 mac=42:01:0a:92:00:0a<br>
>> >> > ok<br>
>> >> ><br>
>> >> > iPXE 1.0.0+ (26050) -- Open Source Network Boot Firmware --<br>
>> >> > <a href="http://ipxe.org" rel="noreferrer" target="_blank">http://ipxe.org</a><br>
>> >> > Features: DNS HTTP iSCSI TFTP SRP AoE ELF MBOOT PXE bzImage Menu<br>
>> >> > PXEXT<br>
>> >> > iPXE> ifstat<br>
>> >> > net0: 42:01:0a:92:00:0a using virtio-net on 0000:00:04.0 (closed)<br>
>> >> > [Link:up, TX:0 TXE:0 RX:0 RXE:0]<br>
>> >> > iPXE> dhcp<br>
>> >> > VIRTIO-NET 0xdf584 cannot register queue 0<br>
>> >> > Could not open net0: No such file or directory<br>
>> >> > (<a href="http://ipxe.org/2d5c403b" rel="noreferrer" target="_blank">http://ipxe.org/2d5c403b</a>)<br>
>> >> > iPXE><br>
>> >> ><br>
>> >> ><br>
>> >> > This ENOENT is raised from virtnet_open_legacy():<br>
>> >> ><br>
>> >> ><br>
>> >> > <a href="https://git.ipxe.org/ipxe.git/blob/26050fd4c87c50503d5bd573b2ec91703676e211:/src/drivers/net/virtio-net.c#l205" rel="noreferrer" target="_blank">https://git.ipxe.org/ipxe.git/<wbr>blob/26050fd4c87c50503d5bd573b<wbr>2ec91703676e211:/src/drivers/n<wbr>et/virtio-net.c#l205</a><br>
>> >> ><br>
>> >> ><br>
>> >> > I also tested some previous commits, but all of them failed, so the<br>
>> >> > issue is<br>
>> >> > not a regression in some recent commit:<br>
>> >> ><br>
>> >> > * 26050fd (Dec 8, 2016): the last commit when I tested<br>
>> >> > * 7b499f8 (Apr 11, 2016): the last commit without support for virtio<br>
>> >> > 1.0<br>
>> >> > * e4419ff (Jul 2, 2010): the first commit with the "native iPXE<br>
>> >> > driver"<br>
>> >> > (The commit date is older than 232c208, but actually newer than<br>
>> >> > 232c208)<br>
>> >> > * 232c208 (Jul 11, 2010): the last commit without the "native iPXE<br>
>> >> > driver".<br>
>> >> > The error message is "ERROR: queue size 4096 > 512", "Cannot register<br>
>> >> > queue<br>
>> >> > #0"<br>
>> >> ><br>
>> >> ><br>
>> >> > Regards,<br>
>> >> ><br>
>> >> > Akihiro Suda<br>
>> >> ><br>
>> >> ><br>
>> >> > ______________________________<wbr>_________________<br>
>> >> > ipxe-devel mailing list<br>
>> >> > <a href="mailto:ipxe-devel@lists.ipxe.org" target="_blank">ipxe-devel@lists.ipxe.org</a><br>
>> >> > <a href="https://lists.ipxe.org/mailman/listinfo.cgi/ipxe-devel" rel="noreferrer" target="_blank">https://lists.ipxe.org/mailman<wbr>/listinfo.cgi/ipxe-devel</a><br>
>> >> ><br>
>> ><br>
>> ><br>
>> ><br>
>> > ______________________________<wbr>_________________<br>
>> > ipxe-devel mailing list<br>
>> > <a href="mailto:ipxe-devel@lists.ipxe.org" target="_blank">ipxe-devel@lists.ipxe.org</a><br>
>> > <a href="https://lists.ipxe.org/mailman/listinfo.cgi/ipxe-devel" rel="noreferrer" target="_blank">https://lists.ipxe.org/mailman<wbr>/listinfo.cgi/ipxe-devel</a><br>
>> ><br>
><br>
><br>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div></div></div><br></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div></div></div><br></div></div>
</blockquote></div><br></div>