Personally I don't PVP, but in PVE, hands down the most annoying spell for me is spirit armor. When a mob suddenly has 400 more hits than normal it can really put a damper in a deck that's tuned down for farming.
Hmmm good question. In my opinion the most annoying is weakness. Especially when the enemy goes first. So boom you have a lot of tings to power up your attack it should be just enough maybe a little too much. Then they use weakness and you waste all your pips and dont kil anything. it just gets annoying. But yah spirit armor is pretty annoying too. I just dont find it as often as weakness.
I have to admit, Spirit Armor is a pain. On a day to day basis, I hate the spell that Balance Monsters use frequently and that is the one that reduces attack damage by 25%. I don't know how many times I will have already selected a spell that produces enough damage to finish a monster off, only to have that annoying spell cast on me so that I have to go another round.