What age people are comfortable with starting their horses tends to depend on what discipline you do and what area you live in. 4 is actually a pretty common age to start a horse around here. Rarely do you see a horse started at 2.

PS: most people don't sell the unbacked 3 yr old, but the green, but competent 5 yr old. Heck, I've seen greenies that were 7 that were started at 6.