Anyone who follows the financial media — or who simply watched yesterday’s price action — could be forgiven for assuming that Federal Reserve stimulus is the most important factor driving the price of gold.

The Fed’s announcement that there will be no immediate taper of quantitative easing sparked a rally of 4.4% in SPDR Gold Shares (GLD) on Wednesday, providing some relief to investors who have suffered through gold’s rough ride in 2013.

Still, it’s worth asking just how much QE has impacted the price of gold. A look at the historical record shows the relationship might not be as strong as investors have been led to believe.

The QE-Gold Timeline

A look at how GLD has performed at different points along the timeline of QE announcements provides some insight into the imperfect relationship between the two.

The first quantitative easing policy was announced on Nov. 25, 2008, and it wrapped up on March 31, 2010. During that time, the QE-gold link was firmly in place as GLD roared to a gain of 34.7% amid the across-the-board gains in the financial markets.

After March 31, the markets hit an air pocket of a no-QE environment until Aug. 27, when Ben Bernanke dropped the hint of a second QE at the 2010 meeting at Jackson Hole, Wyo. Stocks weakened considerably in this period, falling nearly 9% as gauged by the S&P 500 Index, but gold continued its rally.

From March 31 through Aug. 27, GLD rose 11.1%.

The second round of quantitative easing ran until June 30, 2011, and gold indeed benefited: From the first hint of QE2 at Jackson Hole through the program’s conclusion, GLD climbed 20.7%.

With such strong gains in the books, it would be reasonable to expect that gold would have lost ground once QE2 wrapped up. But this proved not to be the case: From June 30, 2011, through Bernanke’s first mention of QE3 (Aug. 31, 2012) gold again rallied, rising 9.9% without the benefit of QE. Notably, traders had five different opportunities to capitalize of short-term gains of 10% or more.

Given the size of QE3 and the general consensus that gold and QE are linked, it would stand to reason that gold would have continued its rally once the third round was initiated. Again, however, the facts don’t stand up to the conventional wisdom.

From the Aug. 31, 2012, hint through Tuesday, Sept. 17, GLD fell 23%. Notably, the vast majority of this downturn — 19 percentage points, to be exact — occurred before Bernanke first uttered the word “taper” on May 22. While gold indeed weakened somewhat in the post-taper market disruptions, it was being hit hard even before investors grew concerned about the potential for a less-aggressive Fed.

In total, this timeline shows that gold failed to perform as expected in three of the five phases of QE.

What Does This Mean for Investors?

For years, we’ve been hearing the mantra, “With the Fed printing money, gold can only go up.” And on a net basis, it has: In the full extent of the “QE Era” — Nov. 25, 2008, to the present — gold has delivered an average annual return of 10.7%. While short of the SPDR S&P 500 ETF (SPY), which has averaged a gain of 15.6% in that time, investors have nonetheless benefited from adopting a buy-and-hold approach with gold.

Despite this, those looking at gold for shorter holding periods need to be careful in drawing a direct line from the current QE policy to the direction of the gold price. While this would-be link has indeed been in place during certain intervals, the actual record shows that it has been far from a perfect indicator.

Those considering a plunge into GLD solely on the basis of yesterday’s non-taper should think twice.

As of this writing, Daniel Putnam did not hold a position in any of the aforementioned securities.