Under the law, if you operate a public business, you can't refuse to sell a wedding cake to an inter-racial couple or an inter-religious couple. Since these refusals are now vanishingly rare, you never hear about them.
To me, the same principle is at stake. I see no reason why you should allow a baker to refuse to make a cake for an LGBT wedding, but say they have to treat inter-racial couples or inter-religious couples without bias. Of course, you might think that such prejudice is acceptable for all three situations.
to me, if you don't want to serve the public, don't open a public business or advertise to the public.
This is where I run into issues. Part of me agrees with you, while the other part of me agrees with the signs I see hanging in some people's stores: "We reserve the right to refuse service to anyone." If it's my store, I'll serve whoever I want or don't want. Imagine if a convicted child rapist came into your store and wanted to buy a bunch of kids toys and candy. I am pretty sure I'd tell him to GTFO.