Full Idea
That Mary "has all the physical information" is not readily imaginable, so no one bothers. They just imagine she knows lots and lots - perhaps everything known today - but that is just a drop in the bucket.
Gist of Idea
No one bothers to imagine what it would really be like to have ALL the physical information
Source
comment on Frank Jackson (Epiphenomenal Qualia [1982]) by Daniel C. Dennett - Consciousness Explained 12.5
Book Reference
Dennett,Daniel C.: 'Consciousness Explained' [Penguin 1993], p.399
A Reaction
I certainly don't see how we can rule out a priori the possibility that someone who really had all the physical knowledge might be able to infer the phenomenal properties of colour.