There's always potential, but paying attention to them helps limit it. There's one old grumpy fucker that you can't really love on anymore without him getting bitchy, but otherwise the ones now are all real nice.

Used to have 1 that didn't like being ridden because she was lazy, she would start stamping while riding them try bucking you off, but she was such a fat lazy turd she'd give up bucking. Another one was real skittish. Once his ears started flattening you knew you needed to get off him. Some small animal went zipping out from under him in the grass, scared him, he took off back for the barn, I couldn't get off him in time so I got a foot hung up in the stirrup and got drug most of the way back.

Edit: I'd suggest finding someone you trust that has horses or a reputable trainer to get into riding. There's a connection you get with horses that's hard to understand by anyone who doesn't deal with them. It's really an amazing thing to witness someone who's really connected with a horse.