Hopefully helps prevent more issues like #3681 and #4008. The
port/address_mode logic is really subtle, and it took me a long time to
diagnose #4008 despite being the one to have addressed the duplicate
issue before! Not to mention I wrote the code! Definitely need to do
something to make it more understandable...