Somewhere between "calm" and "blind panic" there has got to be a point where he's just a bit tense. I don't like getting into battles to the death with my horse, he's bigger than me (and hot and inexperienced on trails). So I do hacking alone in baby steps, picking routes that take him a bit out of his comfort zone (head comes up a bit, maybe walks slightly quicker, but not head straight up and steps getting tense and short), staying well away from the point where he panics. We do that route until he's happy with it, and then add to it. If at all possible I try to make it a circular route, with no point that we have to turn around and go home, resulting in much more anticipation. 
Going out with another horse would speed this process up tremendously. He has to be a lot braver to go out by himself. I took mine out for the first time at our new barn, with another boarder's quiet horse, the other day. He was interested and happy the whole time, and the next day he tried to lead me past the arena to the trails again Happy. Not bad for an arena-weenie dressage horse.
I also think you need help, however. In order to pull this off you need to be able to read your horse better than your story demonstrates, and have the riding skill to settle that early bit of tension before the horse blows a gasket.

“It ain't what you don't know that gets you into trouble. It's what you know for sure that just ain't so.”                                                                                                                           ~Mark Twain