Wednesday, November 18, 2015

Rocket active stabilization system debugging update


Debugging the system on a cheap logic analyzer clone (review coming up soon)
        Ever since I implemented the new control code this system has been plagued with a pair of bugs that frustrated me to no end which I have dubbed "Harlem Shake" and "Mr Freeze." The "Harlem Shake" bug occurs when the IMU outputs seemingly random YPR values suggesting that it is tumbling even when the sensor is kept perfectly still. "Mr Freeze" as the name suggests, is when the system locks up out of nowhere and refuses to respond unless reset which does not prevent the glitch manifesting. After mucking around with the code and a string of long nights with quite a bit of coffee trying to figure out what the heck was wrong with the system, I finally fixed the problem. The following is just inferences based on my limited knowledge at the moment, if anyone has a better theory I'd be glad to hear it. Reading the serial logs led me to believe that the freezing glitch was caused by a FIFO (First In First Out) overflow due to the appearance of that message in the terminal just before the system locked up. Initially, I had messed with the FIFO data speed configuration from 200Khz to 400Khz to no avail before inferring that the data was coming in at a much higher rate than it was being output, being a problem in library implementation. Using a logic analyzer had not showed anything anomalous in the I2C data bus which would have signaled a problem with either the wiring or packet structure. My assumption is that the transcoding of the data  internally was taking up too much processor power or was out of sync causing it to glitch thus causing the system to grind to a halt. This is a library-level issue that I'll be investigating and attempting to fix. Googling around proved library wonkiness to be the case and the fix was to increase the baudrate to the maximum of 115200. After applying this single line fix, the system ran for 10 hours straight without any bugs or freezing at all. I have no clue as to what causes the "Harlem Shake" glitch but it has not appeared since the freezing fix was applied so I'm not going to complain, though it has me curious as to what the core of the problem is.