In my opinion the previous answers were not very objective.
1 Fees are typically lowest with the most efficient brokers.
2. Rates are typically lowest with the most efficient banks.
Branches represent 50% of a banks operating costs. Therefore a bank choosing to have zero branches and operate via the internet will have significant advantages in having the ability to offer the lowest rates.
In general every industry that has been "touched" by silicon valley and utilized technology and innovation has significantly increased productivity leading to lower prices, more services, at less cost.
Now if a Nigerian Prince asks you to refinance via email, think twice, instead pick a brand that resonates with you. Join their facebook fan page, interact with them and their customers, use some of their free products....like savings accounts, online bill payment, et cetera......then refinance with the brand you like the best.
The following 3 banks have relationships with mortgage brokers, so a homeowner may find it advantageous to pay a broker, typically 1%-2% in fees to find the best rate/terms. If a homeowner chooses to apply to all 3 online banks directly make sure to do so within 30 days of each other, in other words, all at the same time. FICO scores are recalculated each time a score is requested except when applying with several mortgage companies at the same time.
Currently there are only 3 online banks in the USA that offer mortgages. There are others but they are just traditional banks with websites or with the word "Direct" in their name.
Bank of Internet: is a small bank out of San Diego specializing in multi family loans.
First Bank of the Internet of Indiana, is a small bank. They do a small number of residential loans.
ING Direct USA is a large savings bank, and portfolio lender, in Wilmington Delaware. They are a low margin/ high volume lender.
I am not affiliated with either of these lenders, or any lender for that matter.