I was never LDS, but I did grow up Catholic, and stopped believing somewhere around the middle of high school... although I'm not sure I ever really believed in most of it.
The problem with reliance on any kind of religious authority, whether it's the Bible or Book of Mormon or the Quran or whatever, is that it makes doctrinal disagreements between religions insoluble in principle. If both sides regard the authority of their text and/or religious leader as ultimately authoritative and self-authenticating, then both sides can do nothing but talk past each other. One group will insist that their sacred texts describe the ultimate truth, while the other side will insist with equal passion that the first group is mistaken, and propose different texts which they believe describe ultimate reality. Blind reliance on authority closes the door on conversation by providing no further reason for belief beyond the absurd non-reason of "it is true because we say so!" Yet claiming that a person or text is the infallible word of God does not make it so, and the fact that there is more than one group that makes this claim means that at most only one of them can be right (or more likely, none of them).
What is really needed is some criteria or method for judging religious truth when disputes arise. The only real candidate I see for this is reason itself. After all, even when a religious text is regarded as ultimately authoritative and infallible, it's human beings who have judged it to be such. Any decision to subscribe to a faith claim which cannot be argued rationally amounts to the surrender of reason to a supposedly higher authority, but again, given that there is more than one authority which purports to know the truth regarding human life, and that these authorities disagree on at least some matters, such a surrender becomes at best a crap-shoot regarding the faith claim's truth value; at worst, it is a willing self-deception revealing an existential cowardice. Hate to break it to you, but there just isn't any book out there that's going to tell you everything you need to know about life... you have to work at it. Life's not that easy.
Of course, reason has its limits (clearly). The truth is that there can never be a truly neutral vantage point, because we can never completely distance ourselves from our background beliefs and assumptions. Religions are like languages that way... even if you learn a new one, you're always going to speak your original language/religion the best, and in all others you'll likely have an accent. That's the cultural baggage we're all forced to carry. Such baggage isn't necessarily bad per se, as long as we remain conscious of the fact that it necessarily makes us biased. If everyone could just keep in mind that we're all biased, and that no one has all the answers, I think the world would be a much better place.
