It sounds like a wise move to put the toy away. The puppy is under heavy stress so he just needs time and some TLC until he adjusts to your surroundings.
Talking to him and making a bit of a fuss over him helps a lot too. All you can do is play it by ear.
It will be interesting to hear how you make out with it.