Go to Pot:
What is the reason we say of something that's deteriorated that "it's gone to pot"?
It's because smiths generally keep a pot on hand into which they throw broken pieces of the metal on which they are working. These scraps are ultimately melted down and used again. Anything which has no further use goes into this pot. (Why Do We Say It? pg 106)