I wondered about the charisma theory proposed in this thread by a few people, including you, so I looked into it to see if it had any credence. I didn't find any, especially considering the large amount of converts who never met Joseph until after committing to leave their homes and crossing an ocean to start anew. I even read one account of people meeting him and finding him not serious enough to be a prophet so they left.
I also think that the idea/belief that they were following a prophet that talked on behalf of God was more a factor than any "charisma" of the man acting in that capacity, although the 2 things are entangled.
The film being shown at the LDS visitors' center, in the old Hotel Utah part of the operation, about Joseph Smith portrays some people having a positive reaction on meeting "The Prophet", and the message is clearly that people who knew him were influenced by the personal reality. . . . or charisma, as the case may be. . . .
One man characterized keeps his doubts clear up to the time he met the "Prophet", I believe. . . . though I may be mistaken. . . . it's in the part about the family coming up the Mississippi river on the steamboat to Nauvoo just at the time of his martyrdom.
my grandmother told me of her acqauintance with an older generation family member who was at Nauvoo when Joseph Smith was killed by the mob, and who was there when Brigham Young was accepted as the new Prophet. She said Joseph Smith had a chipped tooth, and had a distinctive whistle when he spoke, and that when Brigham Young arose to speak after the martydom of Joseph Smith, he miraculously took on the appearance and the whistle in his speech. . . . . and it was taken as a Divine manifestation to the Church affirming Brigham Young.