The fact you think I'm saying that referees don't make mistakes makes me wonder if it would be even worth explaining. But let's give it a try.
Some times refs make bad calls. Some of those bad calls will help us, and others will hurt us. Some of the time they will help us at crucial moments, other times they will cost us the game. When the latter happens, most will complain, and that's fine. We're only human. But stupid people will complain about the refs nearly every game. And that's because they're stupid.
Naturally, we react more emotionally to negative things than to the opposite. That's just our animal instinct. Consequently, bad calls that go against us evoke strong emotions, and are thus much more memorable. Those that help us are either filed under "normal ref mistake" or "tough call, could've gone either way". If your animal instinct trumps your higher cognitive functions, then you'll always feel like the refs are cheating you, because that's all you notice. But there are those of us who can transcend that simple-mindedness. Thus, if I was ever ranking people based on their intelligence, I'll take the "number of complaints about refs" criterion over an IQ test every time.