They may not speak English at the airport, train station and shops. They usually speak basic English.
Hotel Service staff and people who are in direct contact with tourist does speak English though, some working adults does too. Some people working in restaurants does too.
School kids (high school) and College students speaks better English than children.
If you are going just for bag packing trip, do your research well there should not be any problem.
When i went to Japan for a week, i survived pretty well, knowing only phrases like:
greetings: (how are you, good morning, nice to meet you)
Does this train goes to..
do you speak english?
Well, Japanese are a friendly bunch of people and they usually try their best to help you.
But usually if you speak a little slower, general public should understand you well enough.
Know that they slightly lengthen the sound of the alphaberts like:
M is Mmmu
L is Elleu
H is Edge-chi
like the movie "death note", my japanese friend understand it as