If you have refunds coming for those three years, no. IRS interest and penalties are based on the amount owed. If the amount owed is zero, then there are no penalties or interest. However, if you wait over three years, you lose your refund. Note: A few states have failure to file penalties that are a flat amount so you may get hit with them whether you owe or have a refund coming.
If you owe, the IRS will charge you interest and penalties back to the original due date of the return. Ironically, the "Failure to File Penalty" is much higher than the "Failure to Pay Penalty" so you should always file your returns if you owe even if you may not have the money to pay by 4/15.