The people that are being called out for stolen valor are wearing complete uniforms with rank, ribbons, unit patches, etc and are trying to pass themselves off as members of the military.
Wearing a jacket without all of that stuff on it is completely different.
Of course, it also depends on what type of jacket you are talking about. If it is a PT uniform jacket, old field jacket, or Gortex parka, no one will care. If for some reason you are looking into buying a service dress jacket, you would just look stupid.