If you have an acoustic-electric guitar, all you need is one of these.
You just plug the quarter inch jack into your guitar’s output, and your headphones into the eighth-inch plug and play away.
If it’s an acoustic guitar anyway, it’s not like you’d have trouble hearing it and need this to practice with. And if someone is bothered by your practicing, this won’t help, because they will still be able to hear the acoustic. But at least with a setup like this, you can get an idea how the pickup sounds.