Full Idea
It's in virtue of our connection with other speakers in the community, going back to the referent himself, that we refer to a certain man.
Gist of Idea
We refer through the community, going back to the original referent
Source
Saul A. Kripke (Naming and Necessity lectures [1970], Lecture 2)
Book Reference
Kripke,Saul: 'Naming and Necessity' [Blackwell 1980], p.94
A Reaction
There may be two theories of reference getting tangled up here. Going back to the origin is one thing, and relying on the community is another. Do I always know who I am referring to? 'The funniest man in London'.