Full Idea
It would not carry any implication of falsehood to our simple ideas if by the different structure of our organs it were so ordered that the same object should produce in several men's minds different ideas at the same time (e.g. the colour of a violet).
Gist of Idea
There is nothing illogical about inverted qualia
Source
John Locke (Essay Conc Human Understanding (2nd Ed) [1694], 2.32.15)
Book Reference
Locke,John: 'Essay Concerning Human Understanding', ed/tr. Nidditch,P.H. [OUP 1979], p.389
A Reaction
The vital point here is that it would be based on 'different structures'. I personally cannot see any objection to the possibility that someone's qualia might be inverted - by brain surgery. That is a problem for naïve realists, though.