I think it would be a cool thing to have the ability - maybe a spell, special treasure card, or piece of equipment that grants this - to turn people who are our friends into temporary assistant gardners.
As assistant gardners, for a period of 24 hours they could cast various plant spells on our plants (but not their own) and gain a small amount of xp for assisting us.
Not practical and no one can go to your house/castle unless you are there. Besides, your friends may have their own gardens to tend. You can't honestly expect them to use their energy to tend your garden. Gardening is work no matter how you look at it.
If you expect to reap the rewards, then you must put forth the effort.
