Actually, there have been rumors of prostitutes being shuttled into the White House itself in years past. I could dig up links, but I won't.
That's been true at various points all the way back to at least the Kennedy years, probably farther. We know Eisenhower cheated on Mamie prior to entering the White House after all.
If I had to bet my life on any president in the last 100 years who was totally faithful to his wife, in office or out, I would pick Jimmy Carter. Anyone else and I'm guessing it's a 50/50 prop bet at best.