I would do a refresher on the longe line. A few of them. And don't be afraid to back up with the whip and really getting after him. Get his respect in all the gaits on the ground then move to riding. See how that works out, make sure he moves off your voice and slight body/whip signals, then get on. It's been two years, it wouldn't hurt to give him the refresher. I didn't read if he works fine on the longe line. If he does... he might be passive aggressive bullying you.

Good luck, keep us posted!