As many as half of all homeowners have never refinanced their mortgages, despite mortgage rates falling to historic lows over the past three years, according to a new study by Fannie Mae.

That's even though many, if not most of these, could qualify for refinancing, either on their own or through an initiative such as the Home Affordable Refinance Program (HARP).

Closing costs, lack of savings cited

What's holding them back? Fear of high closing costs, for one. Other factors include concerns that refinancing would not reduce their monthly payments enough, as well as not wishing to extend their current loan term.

Some borrowers also expressed concern over having to bring additional money to closing to compensate for declining property values since they bought their home, even though they would likely be eligible for HARP, which eliminates that need. A lack of adequate savings was a top factor cited by those who had not yet refinanced.

The study suggests other factors might include include the relative size of the original mortgage to the remaining balance, along with the number of years the borrower expects to remain in the home, according to Fannie Mae economic researcher Dr. Li-Ning Huang, the study's author.

Most non-refinancers have never sought to do so

The vast majority of homeowners who haven't refinanced have never sought to do so, according to the study. The survey found that 39 percent of all homeowners said they've never tried to refinance, while only 7 percent had tried and been unable to do so, while the other 54 percent had refinanced at some point.

The fact that such a high percentage of homeowners haven't refinanced despite falling rates isn't uncommon. A 2001 study by the Federal Reserve also found that only about half of homeowners had not refinanced their mortgages, although that was a more stable rate environment than has been seen the past three years.

The study found that only 25-30 percent of homeowners have refinanced during the past three years, a period of steadily falling interest rates. Looking ahead, of those who haven't refinanced or originally obtained their mortgage in the past three years, only one out of 10 homeowners described themselves as "very likely" to refinance.

Link between refinancing, education

Borrowers who had refinanced their loans tended to be older, more highly educated and have owned their homes for longer than those who had not, suggesting a link between financial literacy and refinancing.

"Findings suggest that a better awareness of one's financial situation could encourage consumers to consider refinancing and to take action," said Dr. Huang. "In addition, resources and tools that help build financial literacy and awareness could lead to higher rates of refinancing."

Looking ahead, borrowers who said they're likely to refinance their mortgages in the next year were motivated by several factors. One was a fear of being unable to keep up with their current mortgage payments, so they were seeking to reduce them.

Conversely, another frequently cited factor was optimism that their financial situation was about to improve. In fact, borrowers with higher incomes tended to be more likely to expect to refinance than those earning less money.

Results were based on a telephone survey of some 1,400 homeowners with mortgages conducted from January-March 2013.

Published on November 5, 2011