It's perfectly fine to not want to be spoken to when out doing your errands, however, you should have said something along the lines of "I'm really in a hurry today, take care." & continue that every time. If he hadn't gotten the hint after the 3rd or 4th time, then you get straight forward. "It makes me uncomfortable when you talk to me, I really just want to shop." I think saying he was harassing you was very rude, especially because he probably didn't understand why you would be cruel so suddenly since it seems you haven't ever said anything implicating you wanted to be left alone before. Badly handled & very rude, be more thoughtful of others in the future. He probably is just lonely (you said he was old) & sees you often & thought he could try to befriend you. Think how you would feel if you tried to be nice to someone & they said something very nasty to you!
common sense, being raised to treat others with RESPECT!!!!