I've come to the conclusion that it depends almost entirely on the horse, yet varies between on the ground/in the school/out hacking.

The times I am most nervous would be any time I am around or on a horse I don't know at all. That said, I can (or think I can) usually get a reading of a horse fairly quickly as to whether I can trust it, although until I get to know it properly I will never feel as safe with a 100% bombproof horse I don't know as I would with a horse I knew well but had a few quirks. If I've never sat on a horse, you can guaruntee that someone telling me of hundreds of past times when it has reared/bucked/bolted/kicked would send me running for the hills because I don't want to take that unknown risk, but if it's a horse I know and have ridden, hearing of one time something happened is more likely to provoke me to think about WHY it did in relation to what I know about the horse and I can predict better not only how to prevent it but how to

For example, with Rupert, most people are scared of him on the ground and I can see why. He has been known to be extremely disrespectful of your space and used to strike out (years ago) from food agression and there is still a risk now he will nip, which is why we don't make a habit of hand feeding him. However, I don't feel unsafe at all around him on the ground, mainly because I've handled him enough that I know when he is starting to think about getting cheeky, whether he is being innocently stupid or deliberately being a brat.

The other factor is size. I'll get on Rupert, who I know has reared and managed to do a pretty impressive act of levitation with me before. But he is under 14hh. I'm tall enough that many people say I should be riding something 16hh+, but you're crazy if you think I'm getting on anything that big if there's a chance it'll play up the same. The taller it is, the better behaved it has to be for me to be entirely comfortable.

But most of all, the situation you put youself in can have a much bigger effect on your condfidence, I think. People can act stupider than horses, sometimes, and being out on a horse I don't know, accompanied by someone sensible who I know will pay attention to us and not act stupid with their mount will fill me with a lot more confidence than riding even a well-known horse with someone I don't trust.