Yes, and while we are at it, can we remove the penalty for transferring pets between characters.
The reason for both these limits is to prevent people from doing more than their level will allow them to do. If other wizards were able to cast on a wizard's garden, there would be no limit to what could be grown. Even the hardest plants could be grown in large numbers, and high level characters could support multiple houses on individual wizards just casting the higher level spells and letting the others use the easier spells. One of the key points to gardening is time and resource management. This would be lost if a person had access to multiple store of gardening energy.