Full Idea
Perhaps, instead of objects, numbers are associated with properties of objects. Basing them on objects is strongly empiricist and uses first-order logic, whereas the latter view is somewhat Platonistic, and uses second-order logic.
Clarification
Second-order logic qunatifies over properties as well as objects
Gist of Idea
Empiricists base numbers on objects, Platonists base them on properties
Source
James Robert Brown (Philosophy of Mathematics [1999], Ch. 4)
Book Reference
Brown,James Robert: 'Philosophy of Mathematics' [Routledge 2002], p.49
A Reaction
I don't seem to have a view on this. You can count tomatoes, or you can count red objects, or even 'instances of red'. Numbers refer to whatever can be individuated. No individuation, no arithmetic. (It's also Hume v Armstrong on laws on nature).