No, it was perfectly clear that you simply meant "everybody" that disagrees with you. That said, we know a large percentage of the population agrees with the President's immigrant framing because he was elected President based largely on that premise.

So, lots and lots of people are blaming immigrants for all our problems. I'm not sure rich white men need you to defend them to a random on the internet, they are doing just fine.