Quick update about my issue. I'd try and turn the lamps "remote access" off with the switch under it's control screen, and I'd get an error. Then I started getting "This device is only available locally" or "This device is not associated with your TP-Link online profile" type messages.
In the end, I deleted all my non-functional devices from Alexa, removed them from TP-Link, factory reset those devices, re-added them to TP-Link, re-discovered them in Alexa's app... and now they all work fine.
I could certainly understand why folks might get frustrated with them, because I feel like I know what I'm doing with wifi devices, and sometimes the switch wouldn't connect to the wifi on my phone without trying a couple of times, etc. But it seems everything is working great now, and I'm really glad I stuck with it and got it setup how I wanted!