As you progress around the areas of Grizzleheim, keep your eyes open for little brown birds. Just like you found all the Smiths in Wizard City, so will you find all the Yardbirds in Grizzleheim.
If you want the prize, you have to build the ladder rung by rung. There are no shortcuts.