The difference is in the way the circuits on the motherboard are designed. Normally a desktop workload is no more than casual surfing, may be a bit of office suite tools or some developmental activity. All this pertaining to One physical user at a time. And not many services running on the system, which means the load on the system resources such as CPU, RAM & I/O is not very demanding. Hence the different interconnects between CPU, Mem & I/O are generally slower. Similarly the CPUs are generally designed to handle the compute requirements of a single user computing needs as mentioned above.
But a server is altogether different. It has to support multiple users. A server could be a web server, App server, DB server which require a lot of processes to be running in the background. This puts a lot of load on the CPU, RAM and I/O. Which means the bus(es) interconnecting the various peripherals have to be fast. There should be no bottlenecks. Also there should be provision to support multiple processors to cope with increasing work loads. Hence the bus(es) are designed to support this. Since the server is a critical, it needs certain Reliability. availability and serviceability (RAS) features, such as circuit redundancy built-in, remote management etc. All this require additional circuitary which needs to be incorporated on to the board.
On a broad level these are the fundamental technical differences between a desktop board and server board.
Hope that helps.