I can see his KDA isn't the best, so I'd assume you didn't build much MR because Malz wasn't a huge problem at the start? It was a 40-minute game, though, so you really should've accounted for the fact that they'd have magic damage late game. A spirit visage, in my opinion, is a must-have for a Volibear's final build if the enemy has at least one source of magic damage. You shouldn't be going full armor if they've a magic damage source, especially when it's a late-game threat like Malz.
I can't say exactly how much extra damage you might have taken from the AD threats, but 400 armor is just excessive. It stops making much of an impact after a few hundred, I believe. I'd say it's best to have a balance of 200+ each. You get like two-thirds damage reduction from each source that way, and getting more only raises it to 70%-ish. Not a huge difference compared to the amount of MR you could've had with another dedicated item.
As for what to do against Malz as Volibear, as a Malz main I can tell you: Not much. Like I said Visage is good for Voli's passive and all stats are useful on him. Adaptive Helm is really the best anti-DOT magic damage item, though. Itemization aside, though, like I said there's not a great place for Voli vs. Malz. Best course of action is obviously to have the team get rid of the shield first (Twitch might have been the best for that in your game, but to be honest you didn't have great AoE poke that game), and then Voli's best engage has always been Flash + Q. And regardless of who you flipped, if it wasn't Malz, you'd probably get ulted right after. Even if you explode, you took one for the team and Malz lost his ult to the tank. Of course, if they melt everyone afterward it doesn't matter, but even not lasting long, it's a decent accomplishment to get Malz's ult out of the way. And you still had great follow-up damage potential with a Twitch, Jhin and Ezreal, even if only for 1.8 seconds. Also, obviously if you're going to solo engage without poke, the only thing you can do is Flash + E + Q, to ensure Malz's passive is down first.
But I can tell you, especially after Rylais, unless I've just not faced too many great Bears out there, Malz can kite and peel Voli off his team pretty well. After that it's really up to your team to have enough follow-up to win the team fight. Malz has great damage and decent peel for his team, but you get on top of him and have enough mobility to stay in his face, you can out-DPS/burst him with quite a few champs.