Full Idea
If one took the numbers as primitive, one would not be deriving their existence and character from general principles- thus controverting Frege's view of the nature of an a priori subject.
Gist of Idea
For Frege a priori knowledge derives from general principles, so numbers can't be primitive
Source
Gottlob Frege (Grundlagen der Arithmetik (Foundations) [1884]), quoted by Tyler Burge - Frege on Apriority II
Book Reference
'New Essays on the A Priori', ed/tr. Boghossian,P /Peacocke,C [OUP 2000], p.26
A Reaction
He seems to be in tune with Leibniz on this. His view begs the obvious question of where the general principles come from. I would have thought that relationships between concepts might be known a priori, without principles being involved.