What do they lose?
Well they tend to have mostly single target damage.
Where as a Garen going into the enemy team and pressign E can take a lot of health from several enemies.
Basically a diver is a champ that is more focused on taking out a single target VS a champ that is more focused on being something that will be doing less damage to more enemies.
Aka Garen with titanic hydra, Black cleaver and the AD from runes and masteries can easily get his E up to 1200+ damage on all targets in range while Irelia will be more focused on putting all of her 2K+ damage into an enemy ADC or mage.
As such Divers are basically assassins that build tanky enough to to survive some hits instead of being like LeBlanc or Zed who have a combo and safe way out.
And if you are talking about Riven she is in another class which builds mostly damage and relies on an ability to take damage. Ever managed to get some good CC on a riven? She issuaually fairly easy to kill if she gets locked down for a couple seconds in a team fight.