Q. "Does not using margarine or butter (and which do you prefer) make a noticeable difference .... "
A. Hell YES!
In making mashed potatoes, the flavor is different depending on what and how much is put in -- butter, milk, cream, sour cream, salt, pepper, garlic powder, sautéed onion, sharp cheddar cheese -- and that's what I will use.
I do not use margarine (80% oil by U.S. law) or "spread" (less than 80% oil, have seen under 50% oil with it mostly water.) There is a noticeable flavor between butter and not-butter.