I think you've hit most of the issues in your post. He has several notable physical damage sources that don't scale with AD or Crit:
- Q base damage [35-115] (but not the ult, it does magic damage)
- Sheen/Triforce proc [110-193]
- Muramana [6% current mana]
- BoRK on-hits, if you build that (but remember his Q has 110% AD scaling when comparing with Bloodthirster or Death's Dance)
The main reason Lethality items became popular on the other marksmen has been to give them earlier power spikes compared to the more expensive crit items. However, because Ezreal needs to build Triforce and Muramana first, the Lethality items come in much later than they do for the standard Lethality marksman build. While this doesn't necessarily make them worthless, especially if you are playing into a no tank comp and people are not building defensive items, it does take away some of the reason people build them instead of/before crit items.
Without that reason to build them, you end up having to decide if it is better to focus on hitting six items faster and optimizing for the spell and proc damages with the Lethality items or further building up the auto you will be using between spells with critical strike items. I think a Last Whipser item fits into either build path, if needed for tank busting.
Usually, I run Lethality in my runes playing Ezreal and go more crit focused after the Triforce/Muramana core.