Yes, a direct connection to your cable or fiber modem will give you very slightly better performance overall.
(The router introduces latency: one or two stages through the Ethernet switch, and some processing for NAT.)
It will be measurable on an Internet speed test. But I don't think you'd notice it in use, even if you're a gamer.
Having a router really increases your capabilities, and security, so I would not recommend that you eliminate it. Make sure that you've got a gigabit Ethernet connection between your router and your modem, though.