Hands down Weakness and Tower Shield are the most annoying spells. Although I am Balance and I love handing them out to the enemy. Receiving one is like a hard slap to the face. Tower Shield is worse. In worlds Celestia and up you no longer see elemental or spirit shields. Just Tower Shields and Weakness. And since my strategy for these worlds are to set up and then hit like a truck it's heartbreaking to see your enemy cast a Tower Shield right when your going to attack and end the battle. Unless you want a weak attack you have to get rid of it and taking down all your if not most of your buffs with it, and start ALL over again.
Efreet is super annoying, since you cant ignore the negative blade since it is so high. but still, it can be easily fixed with a cleanse charm. Earthquake is also a pain in the neck since you cant do anything to stop it from removing your blades.
I'll just have to say the sprite. It's annoying when your opponent casts it to heal themselves. Also the fairy/pixie. ( I forgot what it's called. ) When your about to finish someone in pvp, ( or whatsoever, ) they just use that spell and BAM! You know what happens next.