Full Idea
An embarrassingly simple argument is that most expressions can be used literally but not referentially, no variation in meaning explains this fact, so its meaning is compatible with being non-referential, so no expression is inherently referential.
Gist of Idea
Since most expressions can be used non-referentially, none of them are inherently referential
Source
Kent Bach (What Does It Take to Refer? [2006], 22.2 L2)
Book Reference
'Oxford Handbook of Philosophy of Language', ed/tr. Lepore,E/Smith,B [OUP 2008], p.542
A Reaction
I think I have decided that no expression is 'inherently referential', and that it is all pragmatics.