Gonna depend on which gun you're talking about.
A small frame pistol will balance better with a shorter barrel. A large frame pistol will balance better with a longer barrel. For example, a Keltec P3AT balances nicely with a 2 3/4" barrel, and a 1911 balances nicely with a 5" barrel. However, if you put a 5" barrel on the Keltec and the 2 3/4" barrel on the 1911, the balance is not so nice anymore.
Since you discounted carry, the main reason to go with a longer barrel in a handgun is velocity, and thus power.
The perceived accuracy improvement actually comes from the longer sight picture, not the barrel length.