I think I would divide it into two categories, and there's a slightly different response for each.
OOC behavior:
This includes how someone speaks to you/treats you on channels, in OOC chatter, pages, and other non-IC communication (though if things in play veer towards the meta that starts to involve this, I'd consider that ooc behavior as well). It also includes things thrust upon your PC that you may or may not agree to (like the Surprise! Incest! stuff that has been talked about elsethread, but it can also mean people imposing offputting subjects onto your play that you're uncomfortable with without asking.)
I think that it is always okay to put up personal boundaries when it comes to how people treat you OOCly. If someone is haranguing you about something on channel, or constantly paging you to do something for them ect, and it is starting to make you uncomfortable, it is okay to ask them to stop, oocly. If someone is constantly negatively venting to you about the game/something, and it is impairing your enjoyment of the game, it's okay to tell them that you need some space from complaints for now. If someone really wants to RP about a subject you're deeply uncomfortable with or have a squick about (my main one is miscarriage or infertility stuff that is overly intimate) then it's okay to tell that person "I'm sorry, but I really don't enjoy that subject, and would prefer to not be involved in those discussions, but I'd love to RP about something else" (if you otherwise enjoy their RP.
IC Behavior:
When it comes to IC canon or worldviews though, I think that you have to be a little more careful, if you are not staff and not a lorekeeper for the game.
If it is GLARINGLY out of theme (Like on Arx, someone playing a character who is super sexist) then I think it's kind if they're a new player to let them know that they're way out of canon and do they know that. And then depending on their response, you could point them towards the right resources if you want to. If they freak out, I would say that's time to just back off and then involve staff, just so they know.
If it's just something that annoys you (like someone who isn't playing a fealty stereotype how you would, someone who gets titles wrong and needs to be reminded frequently) or a playstyle mismatch (spammy poses to your terse ones or vice versa) or just someone whom you oocly don't really care for that much, but they are not harming anything, just annoying you--then honestly I would just let it go and concentrate on other people whom you do enjoy.
I would say making a big ooc deal about annoyance issues rather than truly deep discomfort ones can be a boundary violation in itself, so I understand the caution.
I would always err on the side of speaking up if someone is being rude, disruptive, or mean OOCly. Even if they're getting everything else "right". But if it's just a matter of taste, I would probably err on the side of caution.