Mormon silence should never be construed to mean "conceding" a point. Rather, it's a statement that the unbelievers are no longer deemed "missionary contacts" but "swine". As in not worthy of having pearls to trample in the mud.
LDS doctrine considers, in relation to the Catholics, whether east/west, Roman/Orthodox, that God just walks away from Church leadership that rejects God's leadership, and starts things over again. Well, except in their own case, since the doctrine of papal infallibility has been resurrected and raised from the dead on steroids in Mormonism. We are the last dispensation, there isn't time to "start over". Instead, Joseph Smith wrote, God will send Jesus in Glory to destroy the wicked. Even Catholics will face this judgment. The New Testament describes conditions of apostasy in that time frame as prevailing prior to the return of Jesus. Some Mormons, who have actually paid attention to the scriptures, have the notion that God will "set the Church in order" first, as well. . . indicating a judgment to be made on LDS leadership.
So yes, it's Mormon doctrine that internal leadership that goes sufficiently wrong must be removed from their office by some act of God at some point sufficiently close to the coming of Christ. Joseph Smith also taught that leadership that goes wrong has no "Priesthood" with God. So it's tautological that when leadership goes wrong, the assertion of infallibility is false.