Wednesday, November 18, 2015

Rocket active stabilization system debugging update

Debugging the system on a cheap logic analyzer clone (review coming up soon)
        Ever since I implemented the new control code this system has been plagued with a pair of bugs that frustrated me to no end which I have dubbed "Harlem Shake" and "Mr Freeze." The "Harlem Shake" bug occurs when the IMU outputs seemingly random YPR values suggesting that it is tumbling even when the sensor is kept perfectly still. "Mr Freeze" as the name suggests, is when the system locks up out of nowhere and refuses to respond unless reset which does not prevent the glitch manifesting. After mucking around with the code and a string of long nights with quite a bit of coffee trying to figure out what the heck was wrong with the system, I finally fixed the problem. The following is just inferences based on my limited knowledge at the moment, if anyone has a better theory I'd be glad to hear it. Reading the serial logs led me to believe that the freezing glitch was caused by a FIFO (First In First Out) overflow due to the appearance of that message in the terminal just before the system locked up. Initially, I had messed with the FIFO data speed configuration from 200Khz to 400Khz to no avail before inferring that the data was coming in at a much higher rate than it was being output, being a problem in library implementation. Using a logic analyzer had not showed anything anomalous in the I2C data bus which would have signaled a problem with either the wiring or packet structure. My assumption is that the transcoding of the data  internally was taking up too much processor power or was out of sync causing it to glitch thus causing the system to grind to a halt. This is a library-level issue that I'll be investigating and attempting to fix. Googling around proved library wonkiness to be the case and the fix was to increase the baudrate to the maximum of 115200. After applying this single line fix, the system ran for 10 hours straight without any bugs or freezing at all. I have no clue as to what causes the "Harlem Shake" glitch but it has not appeared since the freezing fix was applied so I'm not going to complain, though it has me curious as to what the core of the problem is.

System debugging setup

Harlem shake glitch: Exhibit A

Harlem Shake glitch: Exhibit B

Nominal terminal output

     A few months ago I wrote a request on the open member board at TechShop DC asking if anyone had ideas on how to get permission to use a runway for kinetic testing of a system, much to my surprise I got a response. None other than the owner of the Washington Executive Airport offered to let me use his 3000 foot runway and taxiways for system testing. I am more than likely heading up there sometime in late November to check out the airport and start doing test operation planning and figuring out where to put cameras for video capture. The current plan is to build a cage that holds the rocket test article, balanced from the COG while still retaining freedom of movement to verify control authority in a testing environment. The cage would be mounted to either the middle doors of the test vehicle (a suburban) or the roof racks with the middle seats being used as a command and data gathering center. Someone else would be driving the car down the runway while I sit in the command center gathering data and monitoring the test. Data would be extracted via either a serial cable or a wireless link that connects to computers in the command center in addition to multiple cameras recording the fin and complete system movements. Post-testing the RPY and fin angle values would be sent to a 3D model showing the movement of the test article for editing into video. This also is going to be time for the test rig to be built and I'll be needing to start fuselage integration sometime in the next few months as a completed test article is needed.

     Again, I apologize for the lack of updates as school is a much higher priority than my projects. The addition of being a temporary contracted engineer to a friend's company developing a secret project is only making my overall stress levels go crazy. Updates will be coming along from time to time until the semester ends on Dec 18, at which point they might pick up. 


  1. Hey, no worries on the slow updates. These things take time. Glad you got those issues debugged.

    1. Thanks, finals have been killing me as of late. Holiday break starts next thursday and I'll have more updates around then.