We saw a mixed bag of financial results produced by the lead bank stocks for fourth quarter of 2010. Those bank stock earnings including a big loss from Bank of America (NYSE: BAC) and a big  miss from Citigroup (NYSE: C), and left investors wondering whether bank stocks are safe or unsound. But that uncertainty in financial stocks and big banks and prompted a “wait and see” approach for most investors.

As such, financial sector and bank stock analysts agree that this year’s Q1 performance will largely determine the market’s feelings for banks moving ahead.

On one hand, there are big possibilities for financial stocks. A return to attractive dividend distributions and a more definitive signal of renewed earnings growth are what investors and analysts will be looking for. JPMorgan Chase (NYSE: JPM) CEO Jamie Dimon told CNBC that the financial company will soon increase its annual dividend to the range of 75 cents to $1 per share, most likely in the second quarter of 2011.

However, the real litmus test for the financial sector and bank stocks will be to show sustainable top-line growth in new business. Tthe more favourable returns at the end of last year were largely propped up by the release of loan loss reserves, and the ability of the banks to overcome regulatory constraints and costs is crucial to the success of major financials.

Achieving the first — business growth — will very much depend on the latter — regulatory impact — according to Wall Street bank analysts.

“It’s really a case of which will win — the impact of the new regulations on the banks versus [recovery of] the economy,” says Burt White, chief investment officer at LPL Financial. There is no question that the new regulatory constraints on the banks under the Dodd-Frank Act will have a major impact on earnings, particularly for the smaller entities which remain under capital pressure, he adds. The only question is how much.

The very large banks face a different regulatory hurdle than their less powerful counterparts such as smaller regional banks Bancorp South (NYSE: BXS) or Zions Bancorporation (NASDAQ:ZION), White notes. The big players have to contend with the Volcker Rule and Basel III, whilst being hobbled by the restrictions on proprietary trading (banks can now only invest up to 3% of tier-1 capital in private equity and hedge funds), he explains, whereas those banks dependent on consumer lending and service fees stand to lose a significant chunk of profitable business under Dodd-Frank.

White, however, is bullish of the growth prospects for both the large and mid-sized banks based on a strong rebound in the economy. He believes this year’s 1-Q returns of the banks will show a significant turnaround in loan growth: “This is the ‘sweet spot’ of the business cycle for the banks. We are now starting to see two trends, consumers are starting to releverage while businesses, particularly in the Midwest, are benefiting from higher commodity prices. I expect to see the banking sector producing a ROE in the 4.5% range, with quarterly growth of about 6%.”

Erik Oja, U.S. banking analyst at Standard & Poor’s Equity Research Services, concurs that the regulatory restrictions on proprietary trading will affect only a handful of banks, many of which have already “cleaned house” on their more risky trading operations. However, he is less optimistic to whether the banks will be able to replace the highly lucrative fee-based income which Frank-Dodd has effectively eliminated. “The 4-Q results showed signs of the negative impact on fee income by the new regulations – everything is now starting to hit in terms of the regulations on fee income. This is a major concern of mine, the banks will have to find a way to replace this income,” he adds.

The banks are now restricted in terms of fees earned from over-draft accounts and debit interchange charges. A limited charge of 12 cents per debit transaction — which should come into effect this July — will have a significant impact on the income of the banks, Oja notes, specifically for the smaller community entities.

Oja expects the banking sector will see top-line revenue grow by 1% to 1.5% this year on the back of loan growth and interest-based earnings.

“We’re neutral on the banks at the moment…the next few months will provide better indication of where the sector is going and what impact the new regulations will have on earnings and revenue growth,” says Todd Salamone, director of research at Schaeffer’s Investment Research.

As of this writing, Sean van Zyl did not own a position in any of the stocks named here.