[ath9k-devel] ath9k MSI interupts

[Lamar Hansford]

Configuration:
* NVidia T30
* AR9485 (also AR9460)
* Kernel 4.6.2

Our company is developing a product on an ARM based platform which requires use of MSI interrupts (legacy interrupts do not seem to work). I have updated the ath9k driver to enable MSI interrupts, however I have a few questions:

ISSUE 1:
When I enable MSI interrupts and allocate ISR it appears the interrupt is masked.

Pseudo code as follows:
* REG_WRITE(ah, AR_PCIE_MSI, AR_PCIE_MSI_ENABLE);
* num_irq = pci_msi_vec_count(pdev);
* ret = pci_enable_msi_range(pdev, 1, num_irq);

Resulting in:
lspci -v
01:00.0 Network controller: Qualcomm Atheros AR9485 Wireless Network Adapter (rev 01)
Subsystem: Qualcomm Atheros AR9485 Wireless Network Adapter
Flags: bus master, fast devsel, latency 0, IRQ 391
Memory at 20000000 (64-bit, non-prefetchable) [size=512K]
[virtual] Expansion ROM at 20080000 [disabled] [size=64K]
Capabilities: [40] Power Management version 2
Capabilities: [50] MSI: Enable+ Count=1/4 Maskable+ 64bit+
Capabilities: [70] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Virtual Channel
Capabilities: [160] Device Serial Number 00-00-00-00-00-00-00-00
Kernel driver in use: ath9k
Kernel modules: ath9k

If I dump the MSI entry (msi_desc ) I see the
Aug 1 16:19:17 s02702016 kernel: [ 311.350904] ath: phy0: ATH9K PCI-MSI-LIST-ENTRY: 0
Aug 1 16:19:17 s02702016 kernel: [ 311.352673] ath: phy0: ATH9K PCI-MSI-LIST-IRQ: 391
Aug 1 16:19:17 s02702016 kernel: [ 311.352695] ath: phy0: ATH9K PCI-MSI-LIST-NVEC: 1
Aug 1 16:19:17 s02702016 kernel: [ 311.352714] ath: phy0: ATH9K PCI-MSI-LIST-MASKED: fffffffe
Aug 1 16:19:17 s02702016 kernel: [ 311.352732] ath: phy0: ATH9K PCI-MSI-LIST-ISMSIX: 0
Aug 1 16:19:17 s02702016 kernel: [ 311.352750] ath: phy0: ATH9K PCI-MSI-LIST-MULTI: 0
Aug 1 16:19:17 s02702016 kernel: [ 311.352767] ath: phy0: ATH9K PCI-MSI-LIST-MULTICAP: 2
Aug 1 16:19:17 s02702016 kernel: [ 311.352783] ath: phy0: ATH9K PCI-MSI-LIST-MASK: 1
Aug 1 16:19:17 s02702016 kernel: [ 311.352800] ath: phy0: ATH9K PCI-MSI-LIST-ENTRY: 0


SYSLOG
Jul 29 18:38:33 s02702016 kernel: [ 128.363804] WARNING: CPU: 1 PID: 1138 at kernel/irq/manage.c:511 __enable_irq+0x54/0x80
Jul 29 18:38:33 s02702016 kernel: [ 128.364078] [<c0128d78>] (warn_slowpath_fmt) from [<c01a162c>] (__enable_irq+0x54/0x80)
Jul 29 18:38:33 s02702016 kernel: [ 128.364113] [<c01a162c>] (__enable_irq) from [<c01a169c>] (enable_irq+0x44/0x7c)
Jul 29 18:38:33 s02702016 kernel: [ 128.364393] [<c01a169c>] (enable_irq) from [<bf277b74>] (ath_pci_probe+0x568/0x944 [ath9k])
Jul 29 18:38:33 s02702016 kernel: [ 128.366378] WARNING: CPU: 1 PID: 1138 at kernel/irq/manage.c:511 __enable_irq+0x54/0x80
Jul 29 18:38:33 s02702016 kernel: [ 128.367054] [<c0128d78>] (warn_slowpath_fmt) from [<c01a162c>] (__enable_irq+0x54/0x80)
Jul 29 18:38:33 s02702016 kernel: [ 128.367127] [<c01a162c>] (__enable_irq) from [<c01a169c>] (enable_irq+0x44/0x7c)
Jul 29 18:38:33 s02702016 kernel: [ 128.367345] [<c01a169c>] (enable_irq) from [<bf277d40>] (ath_pci_probe+0x734/0x944 [ath9k])
Jul 29 18:38:33 s02702016 kernel: [ 128.368975] ieee80211 phy0: Atheros AR9462 Rev:2 mem=0xf2000000, irq=391

I suspect that I need to write a register to un-mask the MSI interrupts.

NOTE: I have other peripherals (wired Ethernet) which uses MSI-X interrupts with no issue.

ISSUE 2
The number of vectors supported indicates 4. However when I call:
* pci_enable_msi_range(pdev, 1, num_irq);

It returns that only a single IRQ is allocated.

This seems problematic as I expect that there are separate IRQ for RX/TX/error/control.


Any help would be appreciated.
-Lamar




This email and any attachments may contain private, confidential and privileged material for the sole use of the intended recipient. If you are not the intended recipient, please immediately delete this email and any attachments.

[Lamar Hansford]

Configuration:
* NVidia T30
* AR9485 (also AR9460)
* Kernel 4.6.2

Our company is developing a product on an ARM based platform which requires use of MSI interrupts (legacy interrupts do not seem to work). I have updated the ath9k driver to enable MSI interrupts, however I have a few questions:

ISSUE 1:
When I enable MSI interrupts and allocate ISR it appears the interrupt is masked.

Pseudo code as follows:
* REG_WRITE(ah, AR_PCIE_MSI, AR_PCIE_MSI_ENABLE);
* num_irq = pci_msi_vec_count(pdev);
* ret = pci_enable_msi_range(pdev, 1, num_irq);

Resulting in:
lspci -v
01:00.0 Network controller: Qualcomm Atheros AR9485 Wireless Network Adapter (rev 01)
Subsystem: Qualcomm Atheros AR9485 Wireless Network Adapter
Flags: bus master, fast devsel, latency 0, IRQ 391
Memory at 20000000 (64-bit, non-prefetchable) [size=512K]
[virtual] Expansion ROM at 20080000 [disabled] [size=64K]
Capabilities: [40] Power Management version 2
Capabilities: [50] MSI: Enable+ Count=1/4 Maskable+ 64bit+
Capabilities: [70] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Virtual Channel
Capabilities: [160] Device Serial Number 00-00-00-00-00-00-00-00
Kernel driver in use: ath9k
Kernel modules: ath9k

If I dump the MSI entry (msi_desc ) I see the Aug 1 16:19:17 s02702016 kernel: [ 311.350904] ath: phy0: ATH9K PCI-MSI-LIST-ENTRY: 0 Aug 1 16:19:17 s02702016 kernel: [ 311.352673] ath: phy0: ATH9K PCI-MSI-LIST-IRQ: 391 Aug 1 16:19:17 s02702016 kernel: [ 311.352695] ath: phy0: ATH9K PCI-MSI-LIST-NVEC: 1 Aug 1 16:19:17 s02702016 kernel: [ 311.352714] ath: phy0: ATH9K PCI-MSI-LIST-MASKED: fffffffe Aug 1 16:19:17 s02702016 kernel: [ 311.352732] ath: phy0: ATH9K PCI-MSI-LIST-ISMSIX: 0 Aug 1 16:19:17 s02702016 kernel: [ 311.352750] ath: phy0: ATH9K PCI-MSI-LIST-MULTI: 0 Aug 1 16:19:17 s02702016 kernel: [ 311.352767] ath: phy0: ATH9K PCI-MSI-LIST-MULTICAP: 2 Aug 1 16:19:17 s02702016 kernel: [ 311.352783] ath: phy0: ATH9K PCI-MSI-LIST-MASK: 1 Aug 1 16:19:17 s02702016 kernel: [ 311.352800] ath: phy0: ATH9K PCI-MSI-LIST-ENTRY: 0


SYSLOG
Jul 29 18:38:33 s02702016 kernel: [ 128.363804] WARNING: CPU: 1 PID: 1138 at kernel/irq/manage.c:511 __enable_irq+0x54/0x80 Jul 29 18:38:33 s02702016 kernel: [ 128.364078] [<c0128d78>] (warn_slowpath_fmt) from [<c01a162c>] (__enable_irq+0x54/0x80) Jul 29 18:38:33 s02702016 kernel: [ 128.364113] [<c01a162c>] (__enable_irq) from [<c01a169c>] (enable_irq+0x44/0x7c) Jul 29 18:38:33 s02702016 kernel: [ 128.364393] [<c01a169c>] (enable_irq) from [<bf277b74>] (ath_pci_probe+0x568/0x944 [ath9k]) Jul 29 18:38:33 s02702016 kernel: [ 128.366378] WARNING: CPU: 1 PID: 1138 at kernel/irq/manage.c:511 __enable_irq+0x54/0x80 Jul 29 18:38:33 s02702016 kernel: [ 128.367054] [<c0128d78>] (warn_slowpath_fmt) from [<c01a162c>] (__enable_irq+0x54/0x80) Jul 29 18:38:33 s02702016 kernel: [ 128.367127] [<c01a162c>] (__enable_irq) from [<c01a169c>] (enable_irq+0x44/0x7c) Jul 29 18:38:33 s02702016 kernel: [ 128.367345] [<c01a169c>] (enable_irq) from [<bf277d40>] (ath_pci_probe+0x734/0x944 [ath9k]) Jul 29 18:38:33 s02702016 kernel: [ 128.368975] ieee80211 phy0: Atheros AR9462 Rev:2 mem=0xf2000000, irq=391

I suspect that I need to write a register to un-mask the MSI interrupts.

NOTE: I have other peripherals (wired Ethernet) which uses MSI-X interrupts with no issue.

ISSUE 2
The number of vectors supported indicates 4. However when I call:
* pci_enable_msi_range(pdev, 1, num_irq);

It returns that only a single IRQ is allocated.

This seems problematic as I expect that there are separate IRQ for RX/TX/error/control.


Any help would be appreciated.
-Lamar




This email and any attachments may contain private, confidential and privileged material for the sole use of the intended recipient. If you are not the intended recipient, please immediately delete this email and any attachments.

[Adrian Chadd]

Hi!

Firstly - please post your diffs somewhere so people can start to help. :)


ok, so:

I have an open copy of the ar9380 HAL in freebsd - find a freebsd tree
(eg github.com/freebsd/freebsd), it's in
sys/contrib/dev/ath/ath_hal/ar9300/ . Grep _MSI *.c and you'll see
wha'ts going on.

TL;DR:

* there are different INTCFG register bits for MSI:

ar9300reg.h:#define AR_INTCFG_REQ 0x00000001 // Interrupt
request flag
ar9300reg.h:#define AR_INTCFG_MSI_RXOK 0x00000000 // Rx
interrupt for MSI logic is RXOK
ar9300reg.h:#define AR_INTCFG_MSI_RXINTM 0x00000004 // Rx
interrupt for MSI logic is RXINTM
ar9300reg.h:#define AR_INTCFG_MSI_RXMINTR 0x00000006 // Rx
interrupt for MSI logic is RXMINTR
ar9300reg.h:#define AR_INTCFG_MSI_TXOK 0x00000000 // Rx
interrupt for MSI logic is TXOK
ar9300reg.h:#define AR_INTCFG_MSI_TXINTM 0x00000010 // Rx
interrupt for MSI logic is TXINTM
ar9300reg.h:#define AR_INTCFG_MSI_TXMINTR 0x00000018 // Rx
interrupt for MSI logic is TXMINTR

.. and the register:

ar9300reg.h:#define AR_INTCFG AR_MAC_DMA_OFFSET(MAC_DMA_INTER)

osprey_reg_map.h: volatile u_int32_t MAC_DMA_INTER; /*
0x5c - 0x60 */

.. and it's offset into the register window as 0x0, so it's actually
register 0x5c.

ok, so that's required to get the MSI interrupts to fire.

there's different logic in get_pending_interrupts() based on whether
it's MSI or not.

Ther'es also this:

ar9300.h: u_int32_t ah_msi_reg; /* copy of AR_PCIE_MSI */
ar9300.h: u_int32_t AR_PCIE_MSI;
ar9300reg.h:#define AR_PCIE_MSI_ENABLE 0x00000001
ar9300reg.h:#define AR_PCIE_MSI_HW_DBI_WR_EN 0x02000000
ar9300reg.h:#define AR_PCIE_MSI_HW_INT_PENDING_ADDR
0xFFA0C1FF // bits 8..11: value must be 0x5060
ar9300reg.h:#define AR_PCIE_MSI_HW_INT_PENDING_ADDR_MSI_64
0xFFA0C9FF // bits 8..11: value must be 0x5064

There's code at the end of ar9300_get_pending_interrupts() and
ar9300_set_interrupts() that messes with the MSI interrupt register
above.

So I'd grep the ar9300 HAL code for:

_MSI *c
ah_msi_reg
AR_INTCFG
AR_PCIE_MSI

HTH,



-adrian