Where I am in NY, you're not supposed to shoot within 500 feet of a dwelling without the owner's permission. Where I grew up, we backed up on some state land as well as a lot of undeveloped but privately owned woodlands. It was just kind of expected that you didn't play in the woods during deer season.
I'd be more concerned whether the hunters were smart enough to know where their missed shots were going. Where they shot the deer wouldn't bother me as much.