Eating litter or dirt is often done by cats suffering from liver issues. At least that is what my vet told me as my Brownie does that occasionally.
Is the litter clumping? If that is the case I would be worried about it clumping and swelling in the kitty's belly and causing some major health issues. Did you mention that to the vet at all?
I think a great idea, as Frenchy mentioned, would be to call the vet and take the little one back in. Good luck.