I did an internship at NASA. What they told me is that anything larger than a golf ball they track while anything smaller than, I think they said a penny, is too small to do damage. The problem is debris that's in between the two. In that case they only get a relatively short warning (it's been a while but I think it was on the order of a couple hours).

The ISS can dodge debris by adjusting the height of its orbit.