I never said that's all he was, but he definitely goes from 0 to defense if there's a highlight play to be made. And he doesn't move his feet better. I'm consistently surprised at how often it looks like Favors feet are dragging on the perimeter or that he refuses to even attempt to guard stretch 4s. Or how often he doesn't show hard on pick and roll D.
Look, I like Favors, but Favors' biggest impediment to being better than he is is himself. You will find no disagreement from anyone that Favors SHOULD be the defender, whether he actually is or not.
BTW, if you put any stock into on/off court points allowed, 48-minute and pace adjusted, Kanter is the better defender (In order: Kanter, Favors, Millsap, Jefferson).