I've watched full games of his, his defense doesn't stand out one way or another. I'm not even saying his defense is a weakness for him, but like I said, stocks is not just an indicator of defense. You will rarely see star players in the NBA with low stocks, and JHS's stocks aren't just low or bad, they are worst in class. Same with with his finishing around the basket and ability to get to the free throw line. These are alarming indicators that he's just not very athletic, and this shows up on film as well. It's hard to ignore things when they are so evident on film + on paper.
Again, I'm not saying this is the perfect end all be all indicator, but to use your own words it's a thing and you can't deny it's a thing. It's something to consider in the large scheme of things and is certainly a massive negative for JHS.