according to information in Russian language, this surname is of German origin, derives from perl (which means "pearl" in English)+mann/man. Either it was seller of pearls (or jeweler), or person with mother who's name was Pearl.
Was used preliminary in Jewish community of late medieval Germany and from there moved to the east, including Russia.
As for ancestors in Latvia, who knows? May be, yes, may be, no.