I would buy a 4 yo not backed before I'd by a horse that was ridden since 2.

We are only just discovering that littlekids are having to go to chiroprators because they are now lugging around heavy back packs to school. All of them? No. But it's a risk you take. And no crystal ball will tell you whether you are making the right decision. That's a big gamble to ask unfinshed bone growth to carry any additional weight than what they were born with, in my books.