Is "Only" In There Or Not?

I think we can all agree that black lives matter.  Of course they do, all lives matter and blacks are a subset of all, so it's just mathematics.

Wait!  You can't say "all lives matter".

Why not, it is after all true.  Right? 

Sure, it's true, but when you say it, it's harmful because:---"Reasons" given that require many dubious assumptions...

So, I can't say something which we all agree is true, because it will hurt the feelings of people who are making outlandish assumptions about my motives?  It seems to me as if I am being expected to accept being a second-class citizen.  

The level of aggression given for why you can't say "all lives matter"  convinces me that there is an invisible "only" in front of "black lives matter".

Much like The Emperor's New Clothes , some of us can see it and (well everyone sees it) but some people know it's there but feel as if it's wrong to point it out.