Full Idea
Zermelo was a reductionist, and believed that theorems purportedly about numbers (cardinal or ordinal) are really about sets, and since Von Neumann's definitions of ordinals and cardinals as sets, this has become common doctrine.
Gist of Idea
Zermelo believed, and Von Neumann seemed to confirm, that numbers are sets
Source
report of Ernst Zermelo (Investigations in the Foundations of Set Theory I [1908]) by Penelope Maddy - Believing the Axioms I §1.8
Book Reference
-: 'Journal of Symbolic Logic' [-], p.489
A Reaction
Frege has a more sophisticated take on this approach. It may just be an updating of the Greek idea that arithmetic is about treating many things as a unit. A set bestows an identity on a group, and that is all that is needed.