This is a bit like asking: "Which is better in a car, a seat belt, or an airbag?" They both provide protection, but do it in different ways. And they provide the best protection when both are used together.
Alarm systems can detect an intrusion, notify the police and neighbors that an intrusion is occurring, and possibly scare the intruder off before he does much damage.
Camera systems are generally used to provide documentation of what occurred "after-the-fact". For example, if someone broke into your home, you could look back upon the recorded images and possibly be able to identify the thief. If properly placed, cameras can also be used while you are home to observe what is going on.
In my opinion, the average homeowner would be far better off with an alarm system than a camera system. There are many sophisticated things that can be done with cameras (remote monitoring of cameras from a security station, video motion detection, etc.) but these types of things cost far more than most home owners are willing to spend.
35 years security industry experience.