No, with some exceptions. Remember, that person who calls you is just working a job, sometimes the only job they're able to get and at really shitty pay, and they're only following the instructions and policies set by the company they work for. I once had a job in a credit card call center (taking calls by people who received the junk mail, not making cold calls), and it's a really unpleasant job. You can't blame the person who took the job because they needed a job.
Now, when the person continues to be pushy and won't take no for an answer, you can start getting annoyed, but there again that person is only doing the job they are told to do, and very often their pay is directly related to the number of positive calls they make that result in a sale or something. All too often they get fired if they don't meet a quota. You can get mad at the company they work for, but always tell the person you are pissed at their company, not the caller as an individual. And a good reason to get pissed at the company is if you're on the do-not-call list yet they call you anyway.
And of course, if it's an obvious scam, yell all you want. The people calling know it's a scam so they're criminals no less than the criminals running the operation.
The one time I got the most pissed and yelled the most was when I got a call from a "charity", the "Fraternal Order of Police" (it's not an actual charity). What pissed me off was that apparently the caller put ME on mute while he's talking through his monologue so he can't hear my objections that would otherwise cut him off. I was screaming into the phone and he never heard me until he was done talking and took me back off mute. When he could hear me again I tore into him hard because of that.