From an easily-available parts-breakdown illustration,* it seems that it's the thumb-stud on the magazine's follower, that actuates the pistol's hold-open latch.
So, my first question is: Does the magazine still have its follower's thumb stud?
Next, if the stud is there, does the bottom extension of the hold-open latch properly contact the follower's thumb stud? (Remove the left grip, and watch.)
Third, is the path of the pistol's slide clean and lightly lubricated?
If the hold-open works properly when you retract the slide by hand over an empty magazine, I would suspect either dirt or a too-strong recoil spring.
If the hold-open does not work properly when you retract the slide by hand, I would suspect either that the hold-open latch is broken, or that the magazine is.
In my personal experience, although not with anything like your particular pistol, I have found that difficulties like the one you describe are more often magazine-related, than they are ammunition-related.
*Fir the diagram, click on: http://stevespages.com/ipb-browning-buckmark.html
For the manual, click on: http://stevespages.com/page7b.htm, and look in the "B"s.
(No, I am not that Steve.)