Citrus tree experts: need advice

Kore!

I planted 3 citrus trees in the spring (meyer lemon, lime, & red grapefruit). They're all 3-4' tall, so still pretty young. Should I be pulling off any fruit that tries to grow until the tree gets more mature? A neighbor mentioned something about it being unhealthy for the tree to grow fruit, because it diverts nutrients that would be used to make the tree grow while it's so young. My lime tree is the tallest (around 4'), and it's got a dozen or so limes growing. The leaves have started falling off, so I'm not sure if it's coincidental, or if the tree is under stress.

WDYT? Should I pull the fruit off for the next year or two? 

~NB~

The reproductive process does "divert nutrients" in both plants and animals, but that doesn't make it "unhealthy".

You CAN redirect resources into green growth if that is your goal, but that defeats the whole point of having a fruit tree. If your goal is delicious citrus fruit, then let the fruit mature, harvest it, and enjoy it. But if foliage is the goal (makes little sense to me, but they aren't my trees), the time to remove the reproductive organs is before they develop. You would remove the FLOWER BUDS before they OPEN, before they're pollinated, and l-o-n-g before they have set fruit. By the time the fruit is set, the most critical "nutrient" you're diverting is water, because the most complicated part is done.

juno

My Eureka lemon has started putting on fruit at 6 ft.  Most of its branches are still thin and delicate, and when the fruit got bigger, many of the branches broke off or split (I think I was able to repair some of it with glue and supports, but it was still a bummer).  I guess it could be considered a nutrient problem if those nutrients would have gone to thickening up those branches?

I haven't had the same problem with my orange, though some of the branches have become sort of deformed from supporting such heavy fruit (same with my young apple).  I'll probably watch the lemon next year and remove blooms from the newer branches and give them a chance to get woody. 

Kore!

Thanks for the advice! My goal is fruit, not foliage; however, still want the tree to mature beyond the puny twig stage, since they're planted in my yard.

My grapefruit tree has a few white flower balls that are about to open. Since that's the smallest of the 3 trees, I may pull off the buds for now to try to get it a little bigger first. It's way too small to support the weight of grapefruit.

~NB~

Also keep in mind that in food crops, where the edible end product is the goal, the plants that we humans create (via grafting) are not fantastic for any other quality other than their crop. Fruit trees with spindly branches are all too common, and have to be pruned differently because of that; sometimes they need artificial support as well.

Fruit trees that grow from seed seldom have such problems, but there's a trade off: sometimes they don't bear fruit for many years. Also, the fruit size might be small and the quality less desirable. Two-foot tall trees heavy laden with fruit, bending over and breaking, are most likely grafted.