Big though you may be, when you first fire that .454 Casull or .44 Magnum with full-power loads, you will be in for a huge surprise.
It's not a matter of mere size, or even of strength. It's a matter of experience.
In a self-defense situation, your very first shot will not end the fight, regardless of what you're shooting. You need to be able to make quick, accurate follow-up shots too.
The powerful handguns you named present so much recoil and so much displacement in the hand, that follow-up shots will be anything but "quick." The flinch these guns can generate in the relatively-inexperienced user will also preclude your next shot from being accurate.
And then, there's the matter of concealment. Sometimes, the bigger the guy, the more difficult it is to conceal a pistol. The guns you mentioned are really big revolvers, and they don't conceal well.
Try before you buy. (Some shooting ranges rent time on various guns.)
Make sure you're using full-power cartridges.
Try carrying one, within the shooting-range venue, and see what it's like.
Then decide on what to buy.