Hard to say...
Physics is basically equation understanding and equation memorization. Mostly it's about how well you know your equations and if you know the right one(s) to use in each problems. Physics is usually more easy to visualize, especially in mechanics, and it has a lot to do with how well you can relate equations into concept, I suppose.
Chemistry has more memorization in general. It depends on what you are learning, but half of chemistry is about understanding concept and explaining (i.e. hybridization, lewis dot structure, chemical equations, etc.), while the other half is basic algebra that can get fairly complicated if you do not understand the equations and the problems thoroughly enough. (i.e. ideal gas law, acid and base equilibrium, titration, rate law, stoichiometry etc.) You almost never see the actual theory in action since they are in atomic levels, and you rely a lot on computer graphics and your own imagination.
And yes, physics has harder math as you move on further. But technically it's hard to tell which one you will like better. In my opinion if you liked biology, you would like chemistry; if you liked math, you would like physics.