So is that a yes or a no or a wuss out?
More of a comment that this question is not really relevant to the discussion. To answer the question would require some knowledge of why I hired an adult babysitter, as opposed to the usual pre/early teen version.
Boy Scout leaders are not babysitters. They don't see the kids in a private location, and don't spend large amounts of individual time with the kis in the course of their duties. So, I don't see the relevance your question has to this situation.
I agree that, as a private club, Scouts can reject anyone they choose. The consequence of being a private club (as opposed to a public accomodation) means that they don't get special government considerations.