The .380 ACP cartridge is only slightly more powerful (whatever that means) than is the .32, although the former's slight edge moves it into the "effective" class, while the .32 stays behind as (relatively) ineffective.
However, neither is anything like a "man stopper."
That being the case, my own opinion is that you have no reason to switch to the P3AT.
A better thought would be to practice being very accurate with your .32, over and over again, until your shot placement automatically and perfectly matches wherever you're looking. Doing that will convert your seemingly-ineffective pistol into a relatively effective one, at least at up-close-and-personal distances.