In my opinion, this distinction is irrelevant, because rules (either formal or informally developed as a reaction to social pressure) are also part of culture.
Culture, like language, doesn’t really care about rightfulness nor its goal is to conform to what some subgroup of people likes or dislikes. Culture is a fuzzy agglomerate of inherited habits and social behaviors that evolves in many ways, some of these ways include the effect of organized movements that propose rules that some people will dislike and will resist. And that’s part of culture too.
It’s difficult for me to provide practical examples of why “Is X offensive or are we told it should be considered so” is in my opinion a misleading framing of how topics like this one can be discussed, because I’m afraid I could fall into rhetoric fallacies that would invalidate my argument.
But if you observe some profound cultural changes of the past, related to how a class of people or some specific behavior slowly became more accepted, you’ll see that for today standards something has become offensive also because many years ago there were social movements that started to shout that it should be considered offensive.
That’s why I think that “is X offensive or we are just told it is” is a false dichotomy when observed on a large temporal scale: you can’t exclude that “being told to” is a positive instrumental tactic, employed today, to define what will be considered offensive in the culture that will develop a few years from now…
…which nobody has to like.