Then let's go to the extreme. Organic farming has its origins in fascism. Scientific breakthroughs in both medicine and space were made by helping extremely questionable if not outright criminal scientists in Op Paperclip. I suppose we should discard those.
This is a rather benign quotation by a certainly debatable if not often controversial politician, but let's leave it at that. There's no reason to make a mountain out of a mole hill.
If someone across the political spectrum says something that's not inflammatory, just take it and run. Otherwise we're going to start crossing a lot of names off lists, including people of interest that you may agree with. There isn't a single person that hasn't said something at least dumb or controversial at least once. If we're talking about politicians, each definitely has more than one "moment;" albeit some more than others.
This isn't one of those times. Leave it.