You have a point but you take it too far.
If you do not believe that what you describe actually does happen then either write a program, or use a word processor to display coarse patterns in stark blue against stark red, or vice versa. Large bold letters will do nicely. If you are feeling adventurous, program the colours to alternate at perhaps two to five times per second. It makes a great screensaver because no one wastes time looking at it for long instead of working. Apart from any aesthetic feelings, the nasty senstion you get is because the wavelengths of the red and blue differ the most among the colours available on an RGB screen.
For more routine examples these effects also occur, though not as sharply. You may notice that an object can be moderately out of focus and still usefully visible, but for fine distinction in testing your focus, optometrists commonly use pure colours; the old black-on-white charts are only for rough work (which is good enough for everyday purposes.)
In any case, remember the retina is neither a mathematical point nor even a plane; there is a bit of leeway. If all our vision had to be perfect all the time we could never see anything clearly.
