Brokerages that cater to mom-and-pop investors are emerging as an early victim of the intensifying scrutiny of high-speed trading.