You start by reading up a LOT on the functions of an O/S including how to handle device activity, how to manage the memory mapping hardware (a.k.a. memory management or virtual memory system), task scheduling, access coordination (users, privileges, roles, etc.) Then you design something that handles every one of those features.
At some point you purchase language software that produces TRUE code - not emulated or interpretive code. The assembler is always the most efficient option for the code, but for the programmer, some other language like C or Pascal would be better for the higher-level functions that would be tedious as all heck to code in assembler. Doing it by yourself will make it likely that you'll have your O/S ready in several years.