Performance:

By this point I had everything working well enough, but there were two glaring issues. The first, my Arduino crashed if it was sent too many messages in a short period of time. The second, I was experiencing data loss on my SocketIO website listener when data was being sent too quickly.   

Debugging NodeJS and Arduino for increased speed:

  Initially, my Arduino code received a string via serial “#AABBCC”, split the string into 3 parts and converted it from Hex to Ints. Example: FF would be 255. This way of processing data posed a big issue. The rate at which I could send messages to the Arduino (10 messages in 5ms) was  faster than the 9600 baud rate could allow. When I tested spamming data from the website I caused the Arduino to crash. While not receiving an error message I believe I ran out of dynamic memory, an issue that can happen when dealing with Strings.    In order to fix this, I first implemented a queuing system on the Express routing server using Bull. This implements a Redis queue, to which I pushed data when a Post request was received. When I received a new job from the queue I emitted the job data on the colormessages SocketIO channel and paused the processing for two seconds. Eventually I would bring that delay to one second then to 500ms.    This queue system is something that I had hoped I wouldn’t have to implement when beginning this project, but it was very simple and increases the ability of the application to scale when there are 10,000 users trying to communicate with the app.    Next, I moved onto the Arduino. First I pushed the baud rate from 9,600 to 115,200. Next I followed this guide on manipulating data using char arrays with a potential fixed length. This makes it easy to know that I won’t ever hit the max memory limit of the Arduino. I implemented the new schema for this new char array idea: R:G:B implementing a  colon as a separator character. Unfortunately, I forgot I appended a /r/n line ending to my serial write abstract class. When I failed to account for that increase in length in long strings like 255:255:255, the program would read this as Red/Green/Blue = 255, BUT, then it would re-read the string as Red = 0, Green/Blue = 255. After increasing the max array length, this worked perfectly.     Note: the code I used for this was very slow, and would be rewritten in Performance v2.   

Moving to Redis Pub/Sub and Queue:

I decided to increase the speed of my app and make it easier to maintain by removing the ExpressJS/SocketIO routing server, and replacing it with Redis Publisher / Subscriber model, and bypassing Bull to create a Redis queue directly. So now the external listener components Website, Twitch, and Youtube push their messages to a Redis queue named “ExternalMessages”. The Redis queue has it’s first element removed and published to the “InternalMessages” channel. The Arduino and MongoDB internal listeners subscribe to that channel.   

Async + Promises:

NodeJS works placing all of your tasks into this thing called the “event loop” and, if you have some code that slows down that loop, it can cause it to block javascript execution, like blocking my SocketIO listener from reading in new data. How do you fix this? By placing functions into the microtasks queue with Async and Promises.    I broke down all of my CPU heavy tasks into async functions. From there I tried to reduce the number of operations they needed to complete to get what I wanted from them. Then I used Promise.all to execute a few related functions at the same time, by pushing them to the microtask queue. After doing this to Web Server and the SocketIO listener, I tested it by clicking two different buttons at the same time, and it successfully handled two messages at once. Prior to this fix, it would not be able to handle two functions, only responding to the first message.    Next I took this concept to the Twitch listener.   

Architecture at this point: