Full Idea
A very reasonable position would be to accept the integers as primitive entities and then use sets to form higher entities.
Gist of Idea
We could accept the integers as primitive, then use sets to construct the rest
Source
Paul J. Cohen (Set Theory and the Continuum Hypothesis [1966], 5.4), quoted by Oliver,A/Smiley,T - What are Sets and What are they For?
Book Reference
'Metaphysics (Philosophical Perspectives 20)', ed/tr. Hawthorne,John [Blackwell 2006], p.150
A Reaction
I find this very appealing, and the authority of this major mathematician adds support. I would say, though, that the integers are not 'primitive', but pick out (in abstraction) consistent features of the natural world.