Even for the occasional weekend warrior who only plans to bomb around on US Forest Service roads, and mild off road driving the more sidewall you can get the better. I will not get into airing down, because without bead lock wheels and pretty good sized tires that is not terribly practical for most.
However, that 1” of extra sidewall compliance on the tire will still make a difference in tire longevity in the event you do find it rubbing on rocks or other hazards. Plus the tires are the first part of the vehicle suspension in contact with the ground, those balloons we roll around on are the initial part of the system for soaking up impacts. Taller sidewalks just soak up impacts better, life is easier on the vehicle and the occupants over rough stuff with tires that can take more edge off of whumps, bumps, pot holes, washboard, etc.
One option for the crowd stuck with 20”s would be to go source a set of 17” Toyota steelies and put some 285/70/17 tires on them or even 285/75/17’s. That will soften things up nicely and put those meats on a set of wheels made for abuse. Then put your 20”s with highway tires back on for Monday-Friday duty.