At defense? Yes, he is a terrible defensive player. Or at least he can't be a center on a good defensive team.
Solid defensive (but not GREAT) centres aren't too hard to find nowadays though. The Bucks have Henson, Plumlee and Monroe, pretty sure they'd be happy to part with 1 without giving up McCollum.