If you had helped a friend complete this quest in the past, the game has already set the flag for that quest to completed. This happened to me as well. I went a third time to check if it was a glitch/bug or a flag setting. The three wizards were not there again.
So it's not really a glitch. Just a flag issue. Nothing to really worry about.
