I have one of these things, I've experienced the same problems. I figure you could try and put some kind of low voltage circuit in their to prevent this or a regulator of some sort to try and keep the battery voltage constant enough to maintain full function.
For you and anyone else interested, a few things I learned is that these things are essentially un-overclockable and I was only able to jump the speed a small amount around 25hz IIRC.
It can be pro-sound modded for anyone who does Chiptune music but I highly recommend against this thing for any type of music work because the audio reproduction isn't accurate to the original Gameboy.
While you can't fit it in an official Gameboy Color housing as stated earlier, you can put an official GBC d-pad in inplace of the really ugly and uncomfortable one that comes standard. I want to say it took a small modification on the plastic but I can't remember, this was a few years ago. I believe I also replaced the rubber contacts under the buttons with those from an official GBC with relaitve ease which made playing it more comfortable (the stock ones are really really stiff.)
also IIRC, I broke the stock speaker and put a DS Lite speaker in and it worked fine with a very small modification the housing, so you could probably get better/louder audio from it if you did this in combination with a small audio amp inside the stock speaker was pretty trash.