In the words of Lynn Redgrave in "House Calls", "I'm English. YOU have an accent." Remember that those of us who are English come from a country that invented the language, and we've been speaking it far longer than Americans. Of course Americans have accents - American ones.
Everyone has an accent. We have hundreds of them just within the UK, and often I can tell roughly where someone comes from just as soon as they start talking. The variety is far wider than in the US, in fact.
So my concern is more to do with whether I can understand what they're saying.
JOHN G makes a good point - American English is not British English. What on earth is cilantro, or a skillet, or a broiler, or a sidewalk, just to name a few words we don't use? There are dozens of others. Even worse, there are words used in different ways. I don't understand the same thing by chips or gas or a subway as you do.