Same thing happened when I let a neighbor's child have a ride on a little bay mare. Fortunately I had told her to ask her mnother if it was okay first because same thing happened, Beauty started to trot, changed direction and the child fell off, fracturing her elbow.
I did feel pretty terrible about that.