[ath9k-devel] AuthSAE rekeying issues with ATH9K

[Stam, Michel [FINT]]

Dear list,

I've been trying to tackle an annoying bug I've been having on a mesh of 2 units with AuthSAE enabled for the past two weeks, but I cannot seem to find what causes it.

It happens when I run an iperf test between the units. At or around the time the SAE lifetime expires, a rekey occurs, after which traffic between the units stops. Wireshark/tcpdump do seem to indicate incoming traffic when observing through a monitor interface.
Sometimes a packets arrive again about a key lifetime later. This does not give a very stable mesh though, as normally the rekey lifetime is 3600 seconds (which means the link is effectively down for an hour).

When the problem occurs, I've observed the "iw dev mesh0 station dump" command returning quickly increasing counters for "rx drop misc" ( NL80211_STA_INFO_RX_DROP_MISC). I can't be certain if it is related, the value also increases on a working link, although it seems slower.

If I leave the link idle (no iperf test, just some pings), then this problem does not seem to occur. This makes me believe it is a race of sorts.

Looking at the debug traces from meshd-nl80211I can find no fault. I also looked at the key material sent down to the ath9k driver (printk's in the kernel driver), but even reading back those registers does not indicate to me that there's a fault. I read back what was written.

Both units use an ath9k Atheros card; One is an AzureWave AR5B95, the other is a Compex WLE200N2-23. I have also observed the problem on Compex WLE350NX cards, so I am guessing this is not hardware related.

I set up both units with this configuration: meshd.txt<https://github.com/cozybit/authsae/files/330064/meshd.txt>. I'm using the latest GIT from AuthSAE.

The kernel I use 4.4.11, but I've seen the same problem with 3.10.49.
The compat-wireless 2016-01-10 driver set used by OpenWRT seems to have the same problem with the old 3.10.34 kernel I run on that system.

The iperf setup is (using 2.0.5):

* One system running iperf -s -u -p 6969 -i 5
* One system running iperf -c -u -p 6969 -i 5 -t 86400 -b 100M

I create the mesh interfaces by:

* iw phy phy0 interface add mesh0 type mp
* ifconfig mesh0 IP MASK up
* meshd-nl80211 -c meshd.txt -i mesh0

Right now the key lifetime is at 60 seconds for problem reproduction, but I have seen the same problem on a link with a key lifetime of 3600 seconds; the link then dies at that time.

Loading ath9k with nohwcrypt=1 solves the problem, but costs more CPU cycles.

Now I've made a patch which calls ath9k_queue_reset every time the key is set. This seems to get rid of the link dying on me, at the cost of a lot of authentication traffic. This is a very heavy-handed approach, and I'm fairly certain this is not gonna work in a production environment. See here for the ugly hack: https://github.com/cozybit/authsae/files/347910/ath9k-install_key-buckshot.diff.txt.

This issue has also been posted as: https://github.com/cozybit/authsae/issues/42

Someone on the AuthSAE github page mentioned that this is apparently this is a long-standing issue with the driver, which was submitted before as https://www.mail-archive.com/ath9k-devel%40lists.ath9k.org/msg13595.html.

Is anyone able to assist me/ give me a couple of pointers?

Regards,

Michel Stam

[Stam, Michel [FINT]]

Anyone who is willing to assist with this?

Kind regards,

Michel Stam

[Adrian Chadd]

Hi!

Ok, so! There's actually some bug in the keycache hardware that the
reference driver calls ATH_SUPPORT_KEYPLUMB_WAR. It waits to see if
there's crypto errors upon RX and if so, it replumbs the key.

I think this is a workaround that needs to be plumbed into the driver.

Bug me about it again soon and I'll go extract the pertinent bits for
someone to go reimplement.



-adrian

[Adrian Chadd]

Also, are you validating both the keycache entry and MIC for TKIP? Do
you have a patch you can share?


-a

Post by Adrian Chadd
Hi!
Ok, so! There's actually some bug in the keycache hardware that the
reference driver calls ATH_SUPPORT_KEYPLUMB_WAR. It waits to see if
there's crypto errors upon RX and if so, it replumbs the key.
I think this is a workaround that needs to be plumbed into the driver.
Bug me about it again soon and I'll go extract the pertinent bits for
someone to go reimplement.
-adrian

[Adrian Chadd]

Right, I just checked. They have some code that reads the key and mic
to compare against what was programmed, and they're calling it each
iteration of ieee80211_timeout_stations().
That happens pretty infrequently (every second, or 15, or 30? not sure
without digging.)

How about doing some periodic re-reading of the keycache to see if
something pops up?

Is it a TX or RX related issue?

If it's an RX only related issue, we could fix it by noticing decrypt
failures for a given key index/MAC and then forcing a replumb.



-adrian

[Stam, Michel [FINT]]

Hi,

Periodically validating the keycache would work as far as I'm concerned, although I'm not sure that a patch for the ath9k would be accepted if that hits the mac80211/cfg80211 layers. This is why I did not look into this, figuring that this should be handled at ath9k level somehow. I did not check if it is possible to access the kernel keycache from within the ath9k driver, this might be an idea.

What I do see when calling iw dev mesh0 station dump is that there's an inactivity timer kept for all stations (=mesh nodes), so it should be possible.

As to TX/RX, I added a monitor interface on the receiving node, and using wireshark I could see packets arriving when the link was not working anymore. That, and the RX_DROP_MISC increasing seems to indicate an RX issue to me. But, if the transmitting end does not use the right key, this would yield the same result, as both ends need to agree on the same key for this to work.
Note that on both nodes I added some debugging to dump the contents of the registers, and that seems to be identical; what is written to it is read back after the write, as well.

Would you mind checking if the patch I sent (replumb_did_not_work.diff) is sound as far as the moment the replumb occurs? Because if that is the case, this would suggest a TX side problem, as this did not seem to fix the issue.

Cheers

Michel

[Stam, Michel [FINT]]

Hi Adrian,

I have a patch which does what you want, but that did not fix the issue. This basically reads back then writes the key material for the MIC and the TKIP.
See replumb_did_not_work.diff, attached.

I tried the replumbing in ath9k_cmn_rx_skb_postprocess(), which is in drivers/net/wireless/ath/ath9k/common.c. That was not very successful either.
Lastly I tried ath_rx_tasklet in drivers/net/wireless/ath/ath9k/recv.c. This function actually invokes the ath9k_cmn_rx_skb_postprocess() function.
I'm guessing that these registers should not be written at a whim, but that needs an understanding of the chips workings that I lack.

I had wanted to keep track of the NL80211_STA_INFO_RX_DROP_MISC, which seems to radically increase when things go wrong (mind, it also increases on a working link sometimes). This variable seems to be derived from flags set in the ath9k driver, which are decoded by the mac80211 layer. Thus keeping track on a per-station level proved to be tricky.

As mentioned before, a patch that did fix the issue was resetting the chip when a key was installed. This caused a lot of packet loss and rekeying to happen.
That's here: https://github.com/cozybit/authsae/files/347910/ath9k-install_key-buckshot.diff.txt.


Thanks,

Michel Stam

[Stam, Michel [FINT]]

Hey Adrian,

Just saw your email, I took a quick look and I was able to find a struct ieee80211_key_conf which contains the key material.
I suppose that the ath_node needs to be extracted from the ieee80211_sta -> drv_priv by looking up the source mac address in the struct ieee80211_hdr somewhere in ath9k_rx_accept()?
The ath_node could then give the ps_key which contains the key cache index. This function also houses decrypt_errors which could be used for detection.

I suppose that ieee80211_find_sta_by_ifaddr( ) with the destination and source mac as arguments would nicely fit the bill.
The struct ath_node could also contain the key material, which could then be used for the replumbing.

Any particular pattern I must follow when writing the key material? Or will ath_key_delete( ) followed by a ath_hw_set_keycache_entry( ) suffice?

Kind regards,

Michel Stam

[Adrian Chadd]

hi,

any news?


-a

Hi,
Yeah, delete and re-add should be fine. Just look at whatever the
locking requirements are so you're serialised with anything else
touching the keycache hardware at the same time.
Hopefully it's not such a terrible bug that you end up doing it like,
hundreds of times a second. :)
-adrian

[Stam, Michel [FINT]]

Hey Adrian,

Apologies, I was out of the office on Friday, so I did not have access to my development system nor my test units.

It is on the top of my todo list, will get to it hopefully today, or tomorrow.

Cheers,

Michel

[Sven Eckelmann]

This post might be inappropriate. Click to display it.