Your score will go up a little, not drop, and you will save yourself some interest.
A car loan is for a fixed term. It is NOT a revolving account like a credit card account is. And it's not paying down a credit card that hurts your score, that helps your score and saves you interest. It's CLOSING the credit card account after paying it down that hurts you, because you are losing that available credit. You want a lot of available credit and very little actual use of it. That gives you a low utilization rate and it improves your score. When you close an account and lose that available credit, it can make your utilization rate go UP - and THAT hurts you.
Car loans do not involve this at all. If you have the money, pay it off. That loan will stay on your report as paid for at least three years before it drops off due to age.
In the meantime, start reading about what goes into making a good credit score.