Lithium-ion polymer batteries should not be deep discharged. This is a quote from Lithium-ion Polymer wikipedia page
During discharge on load, the load has to be removed as soon as the voltage drops below approximately 3.0 V per cell (used in a series combination), or else the battery will subsequently no longer accept a full charge and may experience problems holding voltage under load. This can be achieved, as with other lithium-ion batteries, also harmed by under- and over-voltage, by circuitry that prevents overcharge and deep discharge.
So discharging batteries below some threshold will make it unable to recharge. The best way to store your battery is to keep it charged around 50%.
On the Apple Batteries iPod page, Apple says that you should:
Use iPod Regularly
For proper maintenance of a lithium-based battery, it’s important to keep the electrons in it moving occasionally. Be sure to go through at least one charge cycle per month. If you use your iPod infrequently (gasp), you might want to add a reminder to your calendar program
So my conclusion is that your usage pattern could be a probable cause for your battery problem.