First, check the sights. Be sure they are tight, if one is not it may move a bit changing point of aim. Next fire a few rounds from a rest off a bench to be sure it is sighted properly. Now with the rifle you have two hands, your shoulder, and cheek to reduce movement of the barrel. While the pistol is less stable being held in one or both of your hands. You also have to deal with a shorter sight length than with the rifle. This being said you have to overcome the less stable platform. Start with trigger control. Even pressure on the trigger till it fires. It's really tempting to pull it fast when the sights are lined up on the target, do it and you throw the shot every time. Also try to be sure you have the same grip on the pistol for each shot. More or less pressure from the thumb or if you hold tighter with you fingers will also change point of impact.
Sounds like a bunch of stuff you have to do to make a pistol hit what you want. But it is only a little different form the rifle. Do it a few time carefully and you'll see what it takes to make it work, like they always say practice makes perfect.
Have fun man.