Carrying without a round in the chamber is definitely not the answer.
A holster covers the trigger guard, preventing anything at all from causing the gun to fire while in the holster. During a draw, your motions go against those that would cause anything to "get in there" and pull the trigger (except for your finger). If anything, something like this could happen when holstering the weapon, which if done properly, this still would never happen.
This is an "issue" that bothers some people, and I say maybe these types of weapons just aren't for you. If a firearm with a manual safety would make you more comfortable, than purchase a different carry weapon. There's no sense in walking around uncomfortable, when there's an easy remedy (purchase a different firearm). The gun won't go bang unless the trigger is pulled. A gun in a holster sitting on a table, in a safe, in a drawer, in your pants or anywhere will not fire unless the gun is removed and the trigger is pulled. Basically, you need to make the decision of whether you're comfortable enough carrying the firearm. If not, you should change it.
P.S. - If you'd feel safer carrying an XD, let me know.