The pistol's rear sight can be moved left or right, if you do it carefully.
Are you shooting to the left, or are you shooting down and to the left?
Are you right-handed, or left-handed?
At 20 yards, are you shooting off of a rest, or freehand and unsupported?
(The problem may be your technique, and not the gun's sights.)
We need more information.