are you right handed? if so you may be not getting enough finger on the trigger and pushing the weapon a little. Try with an unloaded weapon (make sure to check). Find a spot on the wall and aim to it. Then squeeze the trigger like you have been. and watch the point of aim to see it move.
You might want to get some snap caps and practice dry firing the weapon. The thing is. where on your finger you are putting on the trigger has a lot to do with how you shoot. too much and you can pull, not enough and push. The 5.5 lb pull and longer travel on a Glock you can see a movement that can throw off your POI. Being you are consistently grouping well but in the wrong spot points to improper trigger pull or other reason for jerking or pushing the gun a little.