I am pretty positive that this is an apple bug in mission control. I disabled the "Displays have separate Spaces" setting in mission control, logged out/in and the problem goes away. This was a bit extreme for me though because you loose the menu bar on the second display when doing this. I then tried changing the "When switching to an application, switch to a Space with open windows for the application" setting. This didn't work until I logged out/in (even though the setting doesn't tell you to log out). Hopefully this works for everyone.
