Hermon Ethernet work queues have more RX than TX entries, unlike most
other drivers. This is possibly the source of some stochastic
deadlocks previously experienced with this driver.
Update the sizes to be in line with other drivers, and make them
slightly larger for better performance. These new queue sizes have
been found to work well with ConnectX-3 hardware.
Signed-off-by: Christian Iversen <ci@iversenit.dk>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Some older versions of the hardware (and/or firmware) do not report an
event when an Infiniband link reaches the INIT state. The driver
works around this missing event by calling ib_smc_update() on each
event queue poll while the link is in the DOWN state.
Commit 6cb12ee ("[hermon] Increase polling rate for command
completions") addressed this by speeding up the time taken to issue
each command invoked by ib_smc_update(). Experimentation shows that
the impact is still significant: for example, in a situation where an
unplugged port is opened, the throughput on the other port can be
reduced by over 99%.
Fix by throttling the rate at which link polling is attempted.
Debugged-by: Christian Iversen <ci@iversenit.dk>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
This change is ported from Flexboot sources. When stopping a Hermon
device, perform hermon_unmap_mpt() which runs HERMON_HCR_HW2SW_MPT to
bring the Memory Protection Table (MPT) back to software control.
Signed-off-by: Christian Iversen <ci@iversenit.dk>
Modified-by: Michael Brown <mcb30@ipxe.org>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The eIPoIB local Ethernet MAC is currently constructed from the port
GUID. Given a base GUID/MAC value of N, Mellanox seems to populate:
Node GUID: N + 0
Port 1 GUID: N + 1
Port 2 GUID: N + 2
and
Port 1 MAC: N + 0
Port 2 MAC: N + 1
This causes a duplicate local MAC address when port 1 is configured as
Infiniband and port 2 as Ethernet, since both will derive their MAC
address as (N + 1).
Fix by using the port's Ethernet MAC as the eIPoIB local EMAC. This
is a behavioural change that could potentially break configurations
that rely on the local EMAC value, such as a DHCP server relying on
the chaddr field for DHCP reservations.
Signed-off-by: Christian Iversen <ci@iversenit.dk>
Modified-by: Michael Brown <mcb30@ipxe.org>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Some older versions of the hardware (and/or firmware) do not report an
event when an Infiniband link reaches the INIT state. The driver
works around this missing event by calling ib_smc_update() on each
event queue poll while the link is in the DOWN state. This results in
a very large number of commands being issued while any open Infiniband
link is in the DOWN state (e.g. unplugged), to the point that the 1ms
delay from waiting for each command to complete will noticeably affect
responsiveness.
Fix by decreasing the command completion polling delay from 1ms to
10us.
Signed-off-by: Christian Iversen <ci@iversenit.dk>
Modified-by: Michael Brown <mcb30@ipxe.org>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Add hermon_dump_eqctx() for dumping the event queue context and
hermon_dump_eqes() for dumping any unconsumed event queue entries.
Originally-implemented-by: Christian Iversen <ci@iversenit.dk>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Require drivers to report the total number of Infiniband ports. This
is necessary to report the correct number of ports on devices with
dynamic port types.
For example, dual-port Mellanox cards configured for (eth, ib) would
be rejected by the subnet manager, because they report using "port 2,
out of 1".
Signed-off-by: Christian Iversen <ci@iversenit.dk>
The malloc_dma() function allocates memory with specified physical
alignment, and is typically (though not exclusively) used to allocate
memory for DMA.
Rename to malloc_phys() to more closely match the functionality, and
to create name space for functions that specifically allocate and map
DMA-capable buffers.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The Hermon driver uses vlan_find() to identify the appropriate VLAN
device for packets that are received with the VLAN tag already
stripped out by the hardware. Generalise this capability and expose
it for use by other network card drivers.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Any underlying errors arising during ib_create_cq() or ib_create_qp()
are lost since the functions simply return NULL on error. This makes
debugging harder, since a debug-enabled build is required to discover
the root cause of the error.
Fix by returning a status code from these functions, thereby allowing
any underlying errors to be propagated.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
When the area to be mapped straddles the 2GB boundary, the expression
(high+size) will overflow on the first loop iteration. Fix by using
(end-size), which cannot underflow.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
The Infiniband specification (volume 1, section 11.4.1.2 "Post Receive
Request") notes that for UD QPs, the GRH will be placed in the first
40 bytes of the receive buffer if present. (If no GRH is present,
which is normal, then the first 40 bytes of the receive buffer will be
unused.)
Mellanox hardware performs this placement automatically: other headers
will be stripped (and their values returned via the CQE), but the
first 40 bytes of the data buffer will be consumed by the (probably
non-existent) GRH.
This does not fit neatly into iPXE's internal abstraction, which
expects the data buffer to represent just the data payload with the
addresses from the GRH (if present) passed as additional parameters to
ib_complete_recv().
The end result of this discrepancy is that attempts to receive
full-sized 2048-byte IPoIB packets on Mellanox hardware will fail.
Fix by allocating a separate ring buffer to hold the received GRHs.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
WinPE has been observed to call PXENV_UNDI_SHUTDOWN but not
PXENV_STOP_UNDI. This means that Hermon hardware is left partially
active (firmware running and one event queue mapped) when WinPE starts
up, which can cause a Blue Screen of Death.
Fix by ensuring that the hardware is left quiescent (with the firmware
stopped) when no interfaces are open.
Reported-by: Itay Gazit <itayg@mellanox.co.il>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
This reverts commit 15c1200 ("[hermon] Work around missing mport
support in current BOFM implementations").
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Current BOFM versions are unable to create entries with mport>1, which
means that only the port 1 MAC address can be explicitly specified.
Work around this by using the provided MAC address as a base address
for all subsequent ports. For example, if BOFM assigns the address
00:1A:64:76:00:09 for port 1
then we will assign the addresses
00:1A:64:76:00:09 for port 1
00:1A:64:76:00:0a for port 2
Future BOFM versions that may correctly support mport will work with
this scheme without modification provided that the BOFM entries are
created in increasing order of mport. Since BOFM tools tend to
generate entries in increasing order (of slot, port, etc), this is not
an unreasonable compromise.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Avoid memory leak of untreated events by having circular event queue
operation.
Signed-off-by: Itay Gazit <itaygazit@gmail.com>
Modified-by: Michael Brown <mcb30@ipxe.org>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Add PRM structures to support Hermon Ethernet devices.
Signed-off-by: Itay Gazit <itaygazit@gmail.com>
Modified-by: Michael Brown <mcb30@ipxe.org>
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Unlike Arbel, port parameters must be applied via a separate call to
SET_PORT, rather than as parameters to INIT_PORT.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Mapping a single page at a time causes a several-second delay at
device initialisation time. Reduce this by mapping multiple pages at
a time, using the largest block sizes possible given the alignment
constraints.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Only port state change events are currently mapped to our event queue,
since those are the only events we are prepared to handle. This
ignores a potentially useful source of diagnostic information in the
case of unexpected failures.
Fix by mapping all events to the event queue; a build with debugging
enabled will therefore at least dump the raw content of the unexpected
events.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
No event is generated upon reaching INIT, so we must poll separately
for link state changes while we remain DOWN.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
ib_smc_update() potentially updates the Infiniband port state, and so
should almost always be followed by a call to ib_link_state_changed().
The one exception is the call made to ib_smc_update() before the
device is registered.
Fix by removing explicit calls to ib_link_state_changed() from drivers
using ib_smc_update(), including a call to ib_link_state_changed()
within ib_smc_update(), and creating a separate ib_smc_init() for use
prior to device registration.
Signed-off-by: Michael Brown <mcb30@ipxe.org>
Christian Iversen
Michael Brown
Itay Gazit