It bothers me. If I rent a house, you don't get to traipse through my yard or let your dog crap on my lawn just because I don't actually own it. If I rent an apartment or hotel room, you don't get to wander in without my permission. My campsite is no different. For the time I've rented it, you can go around unless I've invited you in.
I think that how BLM leaseholders feel about the public being on their leased land.