The difference between different types of flour is primarily in its gluten content. Bread flours tend to have higher amounts of gluten whereas cake/pastry flours have lower gluten contents. All-purpose flours are about in the middle.
The consequences, depending on what you are doing, can be anything from slightly tougher texture to shrinking pastry shells.
The general rule of thumb is that if you're working a dough for pastry, let it rest as long as you can in the refrigerator. By using a pastry flour for pastry doughs, it needs to rest less than if you were to use all-purpose flour.
As far as cakes, for the most part it won't have much of an effect -- so long as you don't over-beat the batter. If you simply fold the batter to the point that everything is well combined than you shouldn't notice too much of a problem.
Just remember that the more you work a dough/batter the tougher the end result will be if you don't let it rest. Make sure, however, that you don't let cake batters rest otherwise you will lose the fluffy crumb (that's why you add baking soda/powder).
I'm a professional chef.