You won't be weak, GH is side world, and offers some more xp and side questing opportunities. If memory serves I didn't touch GH until I hit DS, but that was long ago. As long as you are completing the storyline and many of your side quests you should be fine hitting GH or not.
Grizzleheim is not needed for Mooshu, but contrary to Ravenlady, I think you do get some good XP which does help in Mooshu. Also, GH is needed to get into Wintertusk, and Wintertusk has some of the best XP leveling rewards, and the best equipment (especially crafting) that you will see in the game. And as Ravenlady & others pointed out, you eventually need to get to Wintertusk for your leveled spell and pet. Also, the prospector zeke quests help with training points, which always are worth the effort in the long run.
But both worlds are fun and colorful, and eventually you will want to do both worlds. So really, the order you do them in is not that important.