It's unethical, and we can lose our licenses if we violate that.
And, fyi, it's also unethical for teachers to date students, and frequently, for bosses to date subordinates.
From the AMA:
"Sexual contact that occurs concurrent with the patient-physician relationship constitutes sexual misconduct. Sexual or romantic interactions between physicians and patients detract from the goals of the physician-patient relationship, may exploit the vulnerability of the patient, may obscure the physician’s objective judgment concerning the patient’s health care, and ultimately may be detrimental to the patient’s well-being. If a physician has reason to believe that non-sexual contact with a patient may be perceived as or may lead to sexual contact, then he or she should avoid the non-sexual contact. At a minimum, a physician’s ethical duties include terminating the physician-patient relationship before initiating a dating, romantic, or sexual relationship with a patient. Sexual or romantic relationships between a physician and a former patient may be unduly influenced by the previous physician-patient relationship. Sexual or romantic relationships with former patients are unethical if the physician uses or exploits trust, knowledge, emotions, or influence derived from the previous professional relationship. (I, II, IV) Issued December 1989; Updated March 1992 based on the report "Sexual Misconduct in the Practice of Medicine," adopted December 1990 (JAMA. 1991; 266: 2741-2745)."