When you need physics in a game, the main topic you'll need to learn about is called "kinematics", which is part of a larger topic called "mechanics". In a class, the other important thing (more important than the physics for any graphical programming, I think) is the mathematics of vectors, which you need to describe what's going on and to solve problems.
Kinematics is basically the mathematics of motion, both linear and rotational. Mechanics includes that, but adds forces (gravity and friction, mostly) and Newton's laws.
You really don't need to worry too much about that until you get to 3D, and 3D game development is not really a "for fun, or as a hobby" kind of activity. If you can solve algebra word problems about time and motion, you probably have most of the math you need. (And, if you can't, then you're going to find physics really frustrating, I think. :^)
Much of the "physics" in a 2D game is done with sprite animation. If you know that (distance traveled) = (speed) * (elapsed time) then you know how to move the sprite between events; and changing the sprite animation sequence (like to an explosion when colliding with the wrong thing) lets you visually represent what happened to stop that smooth, regular motion.
A lot of that is true in 3D as well, but rotations, lighting, textures, etc. all add much more complexity than "just adding one extra dimension" would suggest. Take a look at the credits on your favorite 3D game to get an idea of how much work is involved.