You can purchase a variety of different systems, ranging from a few hundred dollars to a few thousand. You probably don't need the most expensive systems (unless your house is huge). A good option would be to place external security cameras in areas that a thief would most likely pass through before he entered your home. You can then attach the security cameras to a DVR so that they only record when there is activity.
However, I think its important to point out that while security cameras are great, they are only one (optional) component of home security. Your first goal should be a home security system. You can purchase these online and install them yourself. They are the same models as you would receive from ADT or Brinks, but at a fraction of the cost, since you do the actual installation (which is not that difficult). The site below is a good resource for home security systems and camera systems. Good luck.