So far typing this I have counted it happening seven times. It is getting to the point where it is too annoying to ignore. The screen just keeps taking me back to the first one. (I run a few apps full screen, and usually keep the first desktop open for any quick work or apps I will use, if I will have an app open for a while that doesn't have full screen, I create a desktop/space for it and run it off on the side somewhere). When it continued to happen I quit all open apps and restarted the computer. After the first few times this happened I pressed ⌘ Cmd- ⌥ Opt- ⎋ Esc and restarted finder. There are no running apps on that desktop (except finder).
My screen (desktop/spaces) keeps jumping to the first 'desktop'.