How to scrape Amazon categories and their URLs?
I am trying to obtain all the categories and their URLs on Amazon's men's best selling shoes - see the picture attached.
I know how to go into a certain page and obtain all the names of the categories and URLs, for example, the very left of the picture. I can get all the name and URLs from Athletic to Work & safety. I also know how to click on the URL of each of these categories and repeat the same process.
The problems is, i have no idea how to deal with whether or not the category has reached the end. Also, under the same main category, some sub-category may or may not have sub-categories of its own.
All i need are the name of the categories and their URLs, and please excuse me if my writing is not clear to you. Ts.
Be the first to answer this question.