No, not if you don't have any health problems that make a high heart rate unsafe. It is true that walking, gardening, house work, etc. does count as exercise. Many people from many cultures use daily activity as their exercise and are able to maintain cardiovascular health and bone density.
The only way to improve endurance, strength, cardiovascular capacity, etc. is to stress the system. Recent dat suggest that short, high-intensity exercise is more effective than moderate exercise in this way. So, it is better to do a HIIT 20 minute workout on a treadmill (running at max speed and a slower speed for short, alternating intervals) than it is to take a one hour walk. But that is provided you are healthy enough for that level of exertion.