Full Idea
There is one hypothetical element in the basis of arithmetic, without which none of it would be true: all the numbers are numbers of the same or of equal units. When we talk of forty horse-power, we assume all horses are of equal strength.
Gist of Idea
Numbers must be assumed to have identical units, as horses are equalised in 'horse-power'
Source
John Stuart Mill (System of Logic [1843], 2.6.3)
Book Reference
Mill,John Stuart: 'System of Logic (9th ed, 2 vols)' [Longmans, Green etc 1875], p.297
A Reaction
Of course, horses are not all of equal strength, so there is a problem here for your hard-line empiricist. Mill needs processes of idealisation and abstraction before his empirical arithmetic can get off the ground.