It’s entirely possible for a Jew whose relatives were killed in the Holocaust to benefit from certain aspects of (for lack of a better term) white privilege. That the Nazis wouldn’t have considered you white doesn’t mean that store clerks, taxi drivers, prospective employers, and others in the contemporary United States won’t accord you the unearned advantages white people, Jewish and otherwise, enjoy. That your ancestors were victims of genocide in a different place and at a different time doesn’t mean you can’t be part of the victimizing caste in your own society, any more than having had impoverished forbears means that you can’t have been born into money. (Not, to be clear, that all Jews are!)
But if you have relatives who were killed for their “race,” killed because powers-that-be didn’t consider them white; if your family and culture were deeply shaped by this fact, and if you’d still be considered Other if you lived on the continent where all of that happened, then I think balking at white-privilege accusations is understandable. There’s an aspect of white privilege that’s about the ability to be sort of carefree about your identity. For your heritage to be a quirk, but not a thing. Through some mix of real and imagined (but understandable, given the historic context) concerns, that version of white privilege isn’t one all Jews see as available to them.