Not considering the HEDT platform because that's best suited for workstations. I'm considering the mainstream consumer platform.
Your point about NOT needing more cores is moot because Intel finally started to increase core counts with Coffee Lake which came out a little more than a year ago and programs have not had a chance to catch up. All of these core count increases are thanks to AMD's 8-core, 16-thread Ryzen 7 processor which costs $300. Once 8 core processors get cheaper, simpler programs like web browsers will take advantage of more threads as these programs become more robust. While there have been a few PC games that can take advantage of more than 4 threads for some time, games that use 6 or 8 threads are becoming more common.
Take a look at benchmarks between the 3rd Generation Core i5 and a 7th Generation Core i5, the spread between the 2 isn't that great because processing power has not come a long way since 2011. The IPC (efficiency) gains have been meager at best. The only noteworthy difference is top of the line HEDT processors have far more cores and clock speeds have become somewhat higher across the board.
Because of these meager gains and the ability to overclock the CPU, many people who run complex programs are still holding on to their 3rd and 4th Generation parts. There has been no reason for these people to upgrade unless they are running highly complex programs or peripherals like 144hz displays. These are people who want the Best of the Best.
It's funny that you'd mention a 3rd Gen part because I just delidded a 3770k and as I'm writing this I'm waiting for the RTV sealant to cure. Judging by the die size of a 3rd and 4th Gen Intel desktop CPU, Intel could have easily made the 3rd and 4th Generation mainstream i7 and i5 processors 6-core parts if they wanted to. IMHO, the 6700k should have at least been a 6-core CPU.
I'd suggest checking back in 2 or 3 years with this question.