What part of your finger is pulling the trigger?
Are your locked in visually on your front sight as you pull the trigger? Can you see the sight move to the left as you pull the trigger?
Are you low and to the left when shooting with both hands?
Single action weapons are much easier to control in this regard. They have a shorter and lighter trigger. A long heavy trigger is much more difficult (but obviously doable).