If you already read Parts I through III and want to skip the preface, you can jump to the beginning of Part IV.
I am publishing this story somewhat hastily in response to the hue and cry on the Above Top Secret forums. Apologies in advance for those areas that I would have improved or omitted entirely had I spent more time thinking about what others wanted to hear and less time thinking about what I wanted to say. I attempted, where it occurred to me, to explain the terms and workings of the things I mention, but I may have forgotten to explain some items which could cause confusion and/or Google searches. I will attempt to update this document in the coming weeks as I have time, to improve the clarity and eliminate elements which may unnecessary or uninteresting to others.
This began and remains a personal journey, an attempt to reconcile and understand my own history and work. If you find it interesting, great. If you don’t, oh well.
The experience I’m about to relate marked the beginning of the end of my NASA work, the end of the beginning of my awakening. The transparent and noble institution I believed I knew was hiding something.
For those who want a synopsis, I’ll try to post one in a few days. Most of you already know parts of it, and I felt it most important that I first get down as many of the specific details as I could to ensure that people wouldn’t accuse me of the greater crime of making extraordinary claims without providing context and specifics.
It is fitting that I publish this today, the 9th anniversary of the launch of Mars Odyssey, the orbiter that made possible so much of the Mars rovers’ good science (in spite of all who have sought to limit it).
P.S. – I hastily inserted chapter headings to try to break up the text logically and make it more readable. These don’t correspond to the original posting in parts.
Chapter 1: Spirit’s Missing Time
Last January I was at JPL on a project when a colleague of mine in the unrelated MER program was working a Spirit glitch. Spirit is one of the two rovers still operating on Mars. Spirit’s expected mission was for a little over 90 days long, but six years later the rover is still operating. Spirit has performed well beyond expectations, but even the best robot is going to have its off days. One of these off days was in late January 2009. My friend and colleague, who I’ll call Rich, was a senior software engineer on the team tasked with maintaining the mobility flight software, the code that controlled the rover’s movements and experiments.
Spirit had refused an instruction. It had been sent instructions to move, it acknowledged receipt of those instructions, but it did not move. In and of itself that was not highly unusual, the rover is given license to ignore move requests it does not believe it can fulfill successfully or safely. But not only did Spirit not move, its non-volatile flash memory was missing data about its motionless hours. Through a separate subsystem they were able to estimate that it had been awake for at least an hour during the gap, but what it had been doing or why it had decided not to move was unknown. Imaging before and after the event showed no change, cameras, IDD, suspension values, terrain; all were the same. This was the mystery my colleague and his team were engaged in solving. Why did the rover ignore its command to move? Why did the rover record nothing? Presumably the one question would answer the other.
They ran a battery of system checks and all appeared nominal. A refusal to comply can have its origins in the rover having lost track of its orientation, so they attempted to recover this with an on-board program that uses the panoramic camera and accelerometers to locate the sun and determine its own orientation from that. After an initial unrelated failure in this procedure (the accelerometer package was off) they were able to reacquire orientation. After a bit more testing and investigation, with more nominal results, the rover driver (RP) once again prepared move orders that were vetted and ultimately uplinked. This time the rover moved as expected, and recorded its activities to flash.
The most probable explanation for the initial faults (the failure to move, memory loss, loss of orientation) was cosmic rays. Cosmic rays are energetic particles, usually protons, capable of disrupting electronic systems they come in contact with. With all systems once again performing as they should, it would be hard to explain any other way. And this can happen, cosmic rays have been suspected in the transitory failure of other satellites, orbiters, and landers. The potential for problems has been amply demonstrated in labs on earth, and as a result Spirit’s electronics are somewhat hardened, to the degree reasonable for its expected exposure.
As part of working the problem, before Spirit had reacquired orientation, Rich had done a few things to dig deep into the mystery. He had dumped portions of Spirit’s memory to see what he might find through a forensic examination back on Earth.