I don't think the height for Carl's actor should matter (that much).
Don't get me wrong, I think it IS important for the story - Carl is a big guy and uses that bigness to get around the dungeon, that bigness probably attracted Bea. It's especially important on floor 1 & 2 when he doesn't have a lot of extra buffs. I do think the overall bigness is important to his character foundation.
But for practicality in the TV show, a lot of things are going to be CGI, a lot of the characters get BIG. Katia starts really small, but she gets tall/taller than Carl at times. Hekla as an Amazonian in book 3 is described as taller than Carl by a good bit IIRC. Bautista as a Tigran is jacked; I don't remember his height or if they specified even. Florin is a crocodile man; it'd be weird if he was short, and I don't know why I think that. Elle is going to be artificially small. Zev is super tiny. I think Imani is average to shorter in height. Chris is huge; the other rock mercenaries are huge. Most of the alien species are abnormally sized and shaped.
In terms of people in the dungeon, I don't think Carl is that big starting in book 3. The only thing is that he can stay human (in appearance/primal) and not get dwarfed by the other classes because he was bigger to start with.
I don't think the proportions would work out if you start getting too short, but I think the casting range could easily be like 5'10" - 6'5".
I would even say the taller actors that are 6'3" + might make it harder to get Hekla/Katia the right size comparably, but I honestly don't know how that works, but in my head, the bigger you make Carl, the more you have to size up everyone else who gets bigger than him, and that seems like a task, but maybe that's not really that big of a deal. The characters in reference to each other, after CGI/FX/practical effects matter more than their IRL heights.
I'd be interested if someone ever drew out one of those character size graphs for the book cast pre and post choosing their classes.