So you can always spot a shimmer when you can? This is another example of the Toupee Fallacy. The Toupee Fallacy can be summed up by the following phrase: "All toupees look fake; I've never seen one that I couldn't tell was fake".

Source: http://rationalwiki.org/wiki/Toupee_fallacy

So if you saw a convincing one you wouldn't be able to see that it's a fake toupee. The same applies to the cloak shimmer. The shimmer is only apparent to you as long as you catch it which implies there is also potentially equally many moments when you are unable to see a cloak shimmer especially given how the human eye functions.

Believe it or not, your eyes and mind play tricks on you. I have studied quite a bit on human anatomy so I know a few things about this. The mind and eyes work together to filter out parts of the world you see that the mind thinks is irrelevant to you. Here is a little trick you can do to confirm this. You can do this at home on your free time.

Look at your monitor and set it so that you have a white background. Cover one eye and make a pin-size hole with your index and your thumb to look through while facing the monitor's white background. Now shake that pin-size hole up and down. Notice anything odd emerging? Perhaps you're seeing black lines that look like veins or capillaries on the white background. Believe it or not, those really are your veins you're seeing. In the human eye, the veins cut in front of the light receptors of the retina of your eye (the cylinders and cones). As light passes through the veins it creates a shadow on the recepters. The mind knows this and considers the veins as irrelevant to what the eye is really trying to see. So the mind filters out the shadows produced by the veins by overlaying parts of the scenery that you see over those shadows to compensate. This also explains why you have a blind spot on both of your eyes which can cause you to miss things like the shimmer of a cloak.

Think about it. For every moment you spot a shimmer, there is also an equal (if not greater) number of moments when you didn't see the shimmer. Thus the toupee fallacy comes to mind when someone says "I could see the shimmer just fine".