[ipxe-devel] Fwd: [PATCH] iavf/intelxf restore initial queue count after use
Geert Stappers
stappers at stappers.nl
Fri Oct 9 16:20:06 UTC 2026
Previously in #ipxe:matrix.org
> > A while ago, i sent a Patch-Suggestion to ipxe-devel at lists.ipxe.org
> > as a non-member and it is now waiting for an approval.
> Resent the patch to the iPXE mailinglist,
> this time with a Cc: to stappers at stappers.it.
Now
----- Forwarded message -----
Date: Fri, 9 Oct 2026 16:13:20 +0200 (CEST)
From: L. Blote
To: "ipxe-devel at lists.ipxe.org" <ipxe-devel at lists.ipxe.org>
Cc: Stappers <stappers at stappers.it>
Subject: [PATCH] iavf/intelxf restore initial queue count after use
Dear iPXE-Devs,
Hi all,
we use iPXE as a uEFI Boot ROM for our Intel Virtual Function NIC inside QEMU+KVM to netboot Debian Linux.
Almost everything works flawlessly and directly out-of-the-box!
Our only problem arises from the combination of three individual settings/behaviors:
1) We use multiple queue pairs to improve performance (which is covererd by the "minimal" feature set as per [1] Section 1.1.2)
2) The Intel Ethernet Adaptive Virtual Function (AVF) Host<>Guest Interface distinguishes between the allocation/request of queues (VIRTCHNL_OP_REQUEST_QUEUES) and the configuration of existing queues (VIRTCHNL_OP_CONFIG_VSI_QUEUES). Unfortunately, documentation is sparse ([1] does not list the opcodes for allocation, while still requireing their use in 6.2.1.1). While the Linux and BSD iavf drivers know about these opcodes ([2], [3]), they are currently not used (or have been removed in [4]).
Furthermore, the Linux host ice driver configures the queue count only during the VF creation, but does not fully (re)initialize a VF on VM startup. Therefore, changes to the queue count of an VF from one VM might also affect the next VM using the same VF.
3) iPXE needs and supports only one queue pair and has the ability to request its desired queue count (which is 1, see intelxlvf_admin_request_qps() ). This happens directly in the device probe, therefore every VF which is accessible to iPXE is downgraded to 1 queue.
As a result, every VF which was seen by iPXE is currently downgraded to 1 queue pair and no current OS driver is able to request more queues, even if the required MSI-X vectors are provided by the VF.
For the linux kernel, a possible workaround seems to be reverting the Patch [4] as mentioned in [5].
But this would not fix the issue for BSDs and would require a custom guest kernel to be built and maintained with each update.
Therefore, this Patch proposes 3 changes to iPXE:
1) Move the Queue Request from intelxlvf_probe() to intelxlvf_open() to leave unused VF NICs unaffected. This causes the netdev to be registered with all queues still allocated. But since there will be no traffic until the device is open()ed, this should be safe.
2) Extend the intelxl struct and store the initially allocated queue count during device probe (see patched intelxlvf_admin_get_resources()) as intelxl->vinitial_queues. This might break binary compatibility, but in our tests even chainloading an unmodified iPXE ontop of our patched uEFI ROM/Driver works well. Since only the private device struct (netdev->priv) is modified, a major incompatibility seems unlikely.
2.1) There is one additional guard block in the patched intelxlvf_admin_get_resources(): When the VM is killed/stopped during netboot without closing the device (see point 3 below), it will be downgraded to 1 queue forever. To recover from this condition, the MSI-X vector count is used as a fallback to determine the desired/maximum possible amount of allocateable queues.
3) During device close, (re-)allocate the queues. On a physical PC this would not be needed since the next power cycle or device driver initialization would restore the original state, but not on KVM+QEMU resp. with iavf.
The Patch was developed against iPXE 1.21.1+git20250501.dad20602+dfsg from the debian project with all other quilt patches already applied.
We have been using it for a few months now without any known issues (in the "Upgrade always" b) version, see open points below)
Open Points:
- How should the invocation of the (2.1) recovery mechanism be logged. Is syslog( LOG_WARNING,..) a good way?
- When should the (2.1) recovery happen? Currently, a) recovery only occurs if the queue count is 1 (preserving intentional downgrades like 5 MSI-X and 2 queues). Alternatively, b) the fallback could always upgrade to the maximum possible queue count (MSI-X - 1 Admin Queue). It would also be possible to c) ignore the initial queue count and directly use the MSI-X vector based calculation for the queue count.
For feedback, questions, or suggestions, feel free to reply to this thread or reach out on #ipxe:matrix.org
Thanks in advance &
Best regards,
L. Blote
[1] https://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/ethernet-adaptive-virtual-function-hardware-spec.pdf (Dead link; see archived version via Archive.org’s Wayback Machine)
[2] https://github.com/intel/ethernet-linux-iavf/blob/main/src/virtchnl.h#L149
[3] https://cgit.freebsd.org/src/tree/sys/dev/iavf/virtchnl.h
[4] https://patchwork.ozlabs.org/project/netdev/patch/20200304232136.4172118-3-jeffrey.t.kirsher@intel.com/
[5] https://lore.kernel.org/lkml/aO0eED8yVkgF3FI8@siski.de/
---
src/drivers/net/intelxl.h | 3 +++
src/drivers/net/intelxlvf.c | 44 ++++++++++++++++++++++++++++---------
2 files changed, 37 insertions(+), 10 deletions(-)
diff --git a/src/drivers/net/intelxl.h b/src/drivers/net/intelxl.h
index 4481300d3..d56c6fc3f 100644
--- a/src/drivers/net/intelxl.h
+++ b/src/drivers/net/intelxl.h
@@ -912,6 +912,7 @@ struct intelxl_nic {
unsigned int port;
/** Queue number */
unsigned int queue;
+
/** Virtual Station Interface switching element ID */
unsigned int vsi;
/** Queue set handle */
@@ -934,6 +935,8 @@ struct intelxl_nic {
/** Current VF opcode */
unsigned int vopcode;
+ /** Initial VF Queue Count */
+ unsigned int vinitial_queues;
/** Transmit descriptor ring */
struct intelxl_ring tx;
diff --git a/src/drivers/net/intelxlvf.c b/src/drivers/net/intelxlvf.c
index ab4df4c47..78b5ec066 100644
--- a/src/drivers/net/intelxlvf.c
+++ b/src/drivers/net/intelxlvf.c
@@ -28,6 +28,7 @@ FILE_SECBOOT ( PERMITTED );
#include <unistd.h>
#include <errno.h>
#include <byteswap.h>
+#include <syslog.h>
#include <ipxe/pci.h>
#include <ipxe/netdevice.h>
#include <ipxe/ethernet.h>
@@ -406,9 +407,23 @@ static int intelxlvf_admin_get_resources ( struct net_device *netdev ) {
/* Parse response */
intelxl->caps = le32_to_cpu ( buf->res.caps );
intelxl->vsi = le16_to_cpu ( buf->res.vsi );
+
+ /* Remeber initial queue count */
+ unsigned int queues = le16_to_cpu(buf->res.qps);
+ if (intelxl->vinitial_queues < queues ) {
+ intelxl->vinitial_queues = queues;
+ }
+
+ /* Upgrade to MSI-X Vectors if the initial queue count is not plausbile */
+ unsigned int vectors = le16_to_cpu(buf->res.vectors);
+ if ((2 > queues) && (2 < vectors) && (vectors - 1 > intelxl->vinitial_queues)) {
+ intelxl->vinitial_queues = vectors - 1;
+ syslog( LOG_WARNING , "%s: INTELXLVF Queue Count %#04x does not match MSI-X Count %#04x - falling back to MSI-X Count\n", netdev_addr ( netdev ), queues, vectors);
+ }
+
memcpy ( netdev->hw_addr, buf->res.mac, ETH_ALEN );
- DBGC ( intelxl, "INTELXL %p capabilities %#08x VSI %#04x\n",
- intelxl, intelxl->caps, intelxl->vsi );
+ DBGC ( intelxl, "INTELXL %p capabilities %#08x VSI %#04x QPS %#02x(%#02x)\n",
+ intelxl, intelxl->caps, intelxl->vsi, intelxl->vinitial_queues, queues);
return 0;
}
@@ -468,9 +483,10 @@ static int intelxlvf_admin_stats ( struct net_device *netdev ) {
* Configure number of queue pairs
*
* @v netdev Network device
+ * @v qcnt Number of Queues to request
* @ret rc Return status code
*/
-static int intelxlvf_admin_request_qps ( struct net_device *netdev ) {
+static int intelxlvf_admin_request_qps ( struct net_device *netdev , int qcnt) {
struct intelxl_nic *intelxl = netdev->priv;
struct intelxlvf_admin_descriptor *cmd;
union intelxlvf_admin_buffer *buf;
@@ -483,7 +499,7 @@ static int intelxlvf_admin_request_qps ( struct net_device *netdev ) {
cmd->flags = cpu_to_le16 ( INTELXL_ADMIN_FL_RD | INTELXL_ADMIN_FL_BUF );
cmd->len = cpu_to_le16 ( sizeof ( buf->rqps ) );
buf = intelxlvf_admin_command_buffer ( intelxl );
- buf->rqps.count = cpu_to_le16 ( 1 );
+ buf->rqps.count = cpu_to_le16 ( qcnt );
/* Issue command (which will trigger a reset) */
if ( ( rc = intelxl_admin_command ( intelxl ) ) != 0 )
@@ -646,6 +662,11 @@ static int intelxlvf_open ( struct net_device *netdev ) {
struct intelxl_nic *intelxl = netdev->priv;
int rc;
+ /* Configure number of queue pairs, if applicable */
+ if ( ( intelxl->caps & INTELXLVF_ADMIN_CAP_RQPS ) &&
+ ( ( rc = intelxlvf_admin_request_qps ( netdev, 1 ) ) != 0 ) )
+ goto err_rqps;
+
/* Calculate maximum frame size */
intelxl->mfs = ( ( ETH_HLEN + netdev->mtu + 4 /* CRC */ +
INTELXL_ALIGN - 1 ) & ~( INTELXL_ALIGN - 1 ) );
@@ -689,6 +710,7 @@ static int intelxlvf_open ( struct net_device *netdev ) {
err_alloc_rx:
intelxl_free_ring ( intelxl, &intelxl->tx );
err_alloc_tx:
+ err_rqps:
return rc;
}
@@ -711,6 +733,14 @@ static void intelxlvf_close ( struct net_device *netdev ) {
return;
}
+ /* (Re)Configure number of queue pairs, if applicable and initial queue count > iPXE Queue Count (1)*/
+ if ( ( intelxl->caps & INTELXLVF_ADMIN_CAP_RQPS ) &&
+ ( 1 < intelxl->vinitial_queues ) &&
+ ((rc = intelxlvf_admin_request_qps(netdev, intelxl->vinitial_queues)) != 0)) {
+ DBGC ( intelxl, "INTELXL %p failed reverting queue count to %#04x\n",
+ intelxl, intelxl->vinitial_queues );
+ }
+
/* Free receive descriptor ring */
intelxl_free_ring ( intelxl, &intelxl->rx );
@@ -819,11 +849,6 @@ static int intelxlvf_probe ( struct pci_device *pci ) {
if ( ( rc = intelxlvf_admin_get_resources ( netdev ) ) != 0 )
goto err_get_resources;
- /* Configure number of queue pairs, if applicable */
- if ( ( intelxl->caps & INTELXLVF_ADMIN_CAP_RQPS ) &&
- ( ( rc = intelxlvf_admin_request_qps ( netdev ) ) != 0 ) )
- goto err_rqps;
-
/* Register network device */
if ( ( rc = register_netdev ( netdev ) ) != 0 )
goto err_register_netdev;
@@ -832,7 +857,6 @@ static int intelxlvf_probe ( struct pci_device *pci ) {
unregister_netdev ( netdev );
err_register_netdev:
- err_rqps:
err_get_resources:
err_version:
err_reset_admin:
--
----- End forwarded message -----
> > I know that iPXE is developed by volunteers in their spare time and
> > i really appreciate that ! I'm just unsure that i missed something
> > important.
https://ipxe.org/contact#customers
Groeten
Geert Stappers
--
Silence is hard to parse
More information about the ipxe-devel
mailing list