"how do we explain an increase in optimism? "

Here's another take by Lawrence H. White on Casey Mulligan's notion of Optimism causing the Housing Bubble:

"If I understand him rightly, I don’t much disagree with Professor Mulligan. We agree that Federal Reserve policy acted to promote the housing price boom by lowering real interest rates. The difficult question is: what share of the boom can we attribute to monetary policy, and what share to other independent sources? Applying Professor Mulligan’s way of computing the impact of lower real interest rates alone on the present discounted values of houses, correct anticipation in 2002 of real T-bill rates — which were about to go 200 basis points lower for the next three years — can account for only around a six percent rise in house prices. Thus the milder-discounting effect by itself accounts for only a fraction of the actual run-up in prices observed, assuming correct anticipations. The present-value calculation is straightforward.

We can get a bit more impact out of lower interest rates by noting that the lowering of mortgage lending standards implied an even larger drop in risk-adjusted mortgage rates than in risk-free Treasury rates. Market participants did not have any clear basis in historical time series for anticipating that this drop would reverse itself soon.

Still, I agree that the joint hypothesis “real interest rate anticipations were correct and they alone fully explain the rise in house prices” is untenable. Of course, we already knew that anticipations of house prices could not have been correct, given that nobody would pay $300,000 for a house in 2006 that he knew would be worth only $200,000 two years later. "

I agree with this. No Spigot Theory.

"Professor Mulligan reasonably proposes to attribute the bulk of the rise in house prices to some kind of ex-post-mistaken (but not necessarily irrationally exuberant) anticipations, offering the hypothesis that “it was optimism that raised housing prices, not much of anything tangible during the boom. Whether it was optimism about future interest rates, future tastes, or future technology is more of a quibble.” Optimism about “tastes” here includes optimism about the future growth of demand in particular local housing markets. Something like that would seem to be required to explain why the house price boom was so highly concentrated in a few states. We can’t explain such concentration by appealing only to optimism about technology or national economic policy variables.

An appeal to optimism, of course, doesn’t really explain events but simply gives us a reframed question: how do we explain an increase in optimism? I suggest that optimism (regarding whatever) during this period was not independent of the rising rate of aggregate nominal income growth that was being fueled by Fed policy. Expansionary monetary policy may have (at least cyclically) effects on relative prices and real variables, like the real demand for houses, through income channels, not only through its effect on the real interest rate. I anticipate, and agree, with Professor Mulligan’s likely response that more needs to be done to quantify these other effects."

There seems good reason to believe that their was such an overabundance and overly magnified aspect of Wishful Thinking in this current situation. However, for the explanation, we need to understand the presuppositions, assumptions, and context of this explosion of Wishful Thinking. I believe that a lot of it comes down to an overestimation of what government can do, and simply thinking of the actors in this drama as free market adherents misses the true nature and assumptions of their belief system, which includes plenty of government intervention when it's in their interest.