Which Colt .22?
Woodsman? Ace? Other 1911 type?
Recoil is probably not a factor. Maybe the works are dirty.
If it's one of those "floating chamber" designs, cleanliness is important.
But—what the heck—since the .22 doesn't generate a lot of recoil, and the gun works on recoil, you may not be gripping it tightly enough, or with sufficiently rigid arms, so you may be absorbing some of the recoil it needs.