You can stop doing the very things that are affecting your self-confidence: "constantly reject myself and put myself down." You cannot develop self-confidence if you are constantly self-sabotaging yourself. Self-criticism has its place and everyone who is earnest about improving themselves should do so from time to time. But overdoing is self-defeating. Here's an illustration of what I mean: man goes to doctor complaining about trouble walking because of the constant pain in his knees. Doctor asks why are his knees so bruised. Man replies he hits them everyday with a hammer. Doctor ask, " Why?" Man said, "To toughen them so he can walk better."
Essentially, what has happen is that you focused on only the physical aspect of martial art training and ignored the mental (or spiritual) side of the training. You concentrated on how to defeat others, but you forget your ultimate and worst enemy is yourself. How many times has your own decision or choice put you in a bad situation? How many times have you done things you knew you should not have or not do things that you know you should?
The real purpose of martial arts in the modern world is not really the fighting. Obviously, the techniques exist for that purpose, but the true utility of martial arts today is about cultivating your mind for among other things how to be disciplined and focus such that you are in control and not your whim or emotion such as what you stated.
Think this is something that can't be done? What do you call that aspect of yourself that made you trained for the last 3-4 years? What made you practice and sweat, and maybe even bled some? Could it be that thing call self-discipline? And since you obviously used it to reach a level that makes you confident in your fighting abilities, what makes you think that same discipline cannot be applied to other things in your life?
The world "self" is there in "self-confidence" because you have to cultivate it. No one and no things are going to give it to you. You train it like you trained muscles in the gym or techniques in martial arts. It takes conscious and constant effort just as it took constant training and effort to each a level that makes you confident when it comes to a fight.