This horse has your number and no amount of walking him about in hand will change that. Either ride him out with another horse in front of him or have a trainer out to ride him through his shenanigans when he starts. And like someone else recommended, if you are forced to ride him back to the arena make that arena ride the hardest most boring arena lesson of his life. Then see if his perception of trail riding changes any.