OS X V10.7 Lion :: Switch User Freezing Mac

I have 2 users set up on my MBP.  About 30% of the time when I try to switch users, by clicking on the current user name in the top right of the screen then choosing either the other user's name or choosing Login Sreen... I get the spinning ball and it's then difficult to do anything.

If I've clicked on the other user's name, the screen fades and I get the password prompt for that other user. I enter the password, and the spinning wheel underneath that spins (as it normally does briefly while validating the password) indefinitely. I can't cancel out of it as I can't click on the cancel button with the spinning ball. The only way I can continue to use the Mac is if I was using a full screen app and can swipe left or right with four fingers to another "space" (not sure this is the right word). That space then works correctly. If I go back to the previous space, the screen is still dimmed with the password prompt and the spinning wheel and can't do anything with it.

It won't let me log off by doing command+shift+Q. It won't let me shut down. So the only way to get out of this is to hold the power button until the Mac turns off.

Info:MacBookPro, Mac OS X (10.7.3)