How are you shooting.
1. A continuous string of fire till mag is empty.
2. Shoot a couple then pause and shoot a couple more.
In my experience it can be difficult to hold a long string together shooting at a single spot. Some tend to hold their breath to much and probably lose concentration as the shots roll on.
Two or three followed by a short pause or shifting between multiple targets night help.
If the XDM is anything like the XD's Ive owned the trigger will smooth up with time and use.