I think it really comes down to the idea if someone should reasonably want to be in that place. If it's 2 AM and you are sitting in a deserted park on a swing, it's pretty reasonable that you are not going to be taking up a spot smoking that anyone else will want to occupy that they can't otherwise. I don't disagree that it's asshole-ish to demand someone to leave.
Actually, even if it's noon and you are sitting on a swing and smoking a cigarette, it's still probably /asshole-ish/ to demand you leave. But if there are a bunch of families and kids around that are just looking over at you because they'd like to swing but they don't want to be next to your smoke-- You are probably also being an asshole.