This is getting into basic compilers and interpreters. The first part you'll generally want to do is make it solve simple things such as add, subtract, multiply and divide. However, there is a lot more to it than that.
Possibly the easiest to understand is Crenshaw's compiler tutorial. It uses Pascal as the programming language and I think Motorola 68000 machine code instructions in the example, but go ahead and convert it to C/C++ and work with it. Rather than have it generate machine or assembly code, you can also change it to interpret a given formula input as a text string. I know it will work since I had it generating C source code from Qbasic source using Qbasic as the programming language and some special C libraries.
I won't attempt to try to explain it all, but the tutorial does a very nice job and shows you how to do some things in very small pieces.
Something of an expert in C/C++, Basic and even a bit of Pascal when I need it.