Well I ride a taller horse because I am so tall, on smalle horses I have to be really careful with my upper body or we knock (the horse in my avitar is 14h though). My "assumptions" are
More along the lines of if I see someone in the warmup ring doing a ton of flatwork, I think they must be an eventer or dressage rider. If their horse is REALLY skinny or seems to be borderline out of control I think, great, another out of control wannabe jumper. If I see the horse going really quiet and its not wearing boots I assume they must normally do the hunters lol.