And it might be a major pain in your ass if you DON'T. Whether there is no one there (VERY bad) or someone you do not know (just as bad), you would be held liable for neglect if anything were to happen to her. Forget your ass, think about her safety. Even if it's people you know, they don't share custody with you, only the BM does.
So the next time it happens, take her back home with you. When the BM calls and starts pitching a fit, ask her if she would have done the same had it happened to her and you weren't home? And if she still has a problem with it, tell her to take it up with the court and she will need to come to YOUR home to pick dau. up, since you already made the trip to her home and she wasn't there.
So what if she gets her panties in a bunch? What's the worst that can happen? And how much is your daughter's safety worth?