Those who are militant can see the hypocrisy and what the future looks like if people want to travel down that road. And guess what? It's gonna be more divisive, racist, hypocritical and will lead to more tension.
If we start segregating everybody based on who has been oppressed more, it doesn't advance society, it regresses it. The whole point of eliminating racism is so we don't have to see it everywhere, but by constantly dividing people based on what they look like, it'll only end up with more racism.