Typically flyers can be caused by the tightening of your grip as you pull the trigger. Where they go depends on which hand is tensing up as the hammer falls. You probably already know this. All I wanted to comment on was the fact that your shot groups were much better with your hand loads. Are they light? Or just as hot as factory loads? Good groups either way, either ammo!