well it takes time ,I don't think it takes five hrs a day though
If he was my horse I would let him 'help me around the place' whenever you're out and about doing maintance or cleaning stalls or whatever let him follow you around and hang near the horse trailer a lot go in and out of it all the time make up something to do in there ,things like that and if you aren't freakin out odds are good they aren't going to either
that's what I do I just let them shadow me and touch everything or not pretty soon they get over it
and just don't overreact or pay to much attention to what his mental state is ,he'll notice you're confident and he'll take his cues from you