When the debris are on our line of sight to the star, it would make the star dim, however, the [individual pieces of] debris cover only a small fraction of the sky, so the chance of being on the line of sight is small.