Gcode Manipulator

I spent quite a lot of time on this today, and the code is now "memcheck clean" which means that Valgrind reports no memory leaks. Hooray! I think this is pretty well optimized now.

About a week ago I was working on another piece of software that uses this library (which I may release if it ever works) and I needed to make a shape fit in a very small box (< 1 unit square) because I hadn't set up the camera yet. (That's an OpenGL technicality.) So, I decided that instead of scaling an SVG and using CAM on that, I would instead tell the CAM to use twice as many pixels per inch in order to halve the size of the model, theoretically. As it turns out, that causes the output file to be huge. I was trying to process it when I saw that the program had been running for five minutes with no results. Confused, I looked at the file with Atom only to see that it also took forever to open it. When I looked at the file size I noticed that it was 35MB, which would explain why everything involving it was so slow. I wanted to fix that, because the parser got through 1.6x10^6 lines before crashing with an unfortunate segmentation fault. Thus, multithreading support was under way.---------- more ----------In order to understand why this is so difficult, one must understand something about how computers, and program optimization works. Generally speaking, as I do not want to go into incredible detail, the rule is that a program can be fully optimized for CPU efficiency, or memory efficiency, but not both. Compromises must be made. There are two "models" if you will, for these optimizations.

Stream Model:

The stream model is memory efficient, but CPU intensive. Rather than store anything in memory, this model works on one item at a time, using only as much memory as is absolutely needed to work on the input. There is no input buffer, and no output buffer, and therefore compute time is based solely on CPU power and other intrinsic factors (ie bus speed, MMU latency, etc.) Before multithreading was added, this was the model used by this program.

Block Model:

The block model is CPU efficient, but memory intensive. Basically, it does the opposite of the stream model, and it buffers everything needed in memory, which allows the program to do things "concurrently" with an actor/event model, as well as switch states and resume without the need to complete one task before moving to the next. Many, many programs use this model because it is easier than the stream model.

Since multithreading is CPU efficient, it must make use of the block model. Unfortunately, that means that this program/library was unable to be multithreaded without an overhaul.

The Solution:
I opted for what most people would consider an overly complicated re-write because rather than going about it with a "get 'er done" attitude a la Larry the Cable Guy, I worked with an idealistic approach with the hopes of allowing the original goals of the program and now library I wrote to be met while using multithreading to take advantage of modern computer systems. I split the original RS274 class into three new classes, RS274Tokenizer, RS274Worker, and RS274. Each one builds on the previous one, where RS274Tokenizer is just like the old RS274 class, which is stream-based, and contains only extra data, namely the regex strings needed to do the search and parse operation. It is extremely lightweight. RS274Worker contains the tokenizer, plus a buffer for data to work on, and utility functions to set up and run. Finally, RS274 now contains only automation, making loading, creating threads, moving data, parsing, reconstruction, and finally cleanup a simple process.

The functionality remains the same, and the per-thread parsing speed is unchanged so far as I can tell - 125,000 lines of gcode in 33 seconds or so on a Sandybridge 3.1GHz processor. Now, the program can saturate system resources (processor wise) and use all available threads to work in parallel. Watching it work on a 2.6GHz Ivybridge 8-core i7 is amazing.

Most of the bugs are squashed at this point, I think. Now I'm just trying to make it "memcheck clean."

I was about to write about the re-write I just did, except before I published, I decided to do one last check to make sure it was really working. Good thing I did, because it was not working. I'll write the rest once I get the chance and the energy, but the gist is, RS274 got split into several classes and structs and is now fully multithreading capable. As in, if you feed it a 35MB text file, it will saturate your system resources until it's done. It's great, memory consumption is approximately linear!

As it turns out, there is a surprising amount of unwritten information in RS274 that can be synthesized via analysis. I've split this program into a CLI frontend and a library, and the library, named RS274, now supports doing some of this analysis. It now synthesizes the following meta-data from each file as it parses:

• Modality of lines

• This is used to tell whether a line is under a command that is in effect in the system. As it turns out, this is necessary for use later, so this datapoint is stored with each line.

• It enables lines like the following, where only one command is given for several lines.

G0 X0 Y0 Z0
    X10 Y5
    X5 Z2

• Command type

• Can be motion or non-motion, currently. Does not differentiate between linear/polar motion.

This information is useful for another project I'm working on, which is why I spent the time implementing it. The logic for doing so is very tricky when trying to use the least system resources possible, which is one of the goals of this project.

Lastly, ternary statements are awesome:

(isLineModal & isCommandMotion) ? isMotionBlock = true : (isLineModal | motionMatch[0] == "") ? isMotionBlock = true : isMotionBlock = false;

Bugs in code tend to come together, even worse, they cascade, one leading into another. In this case, there were two problems. First of all,

if (0 == NULL) {
  /*This will run*/
}

The second problem is:

atof(""); /* return: 0.0 */
atof("Hello World"); /* return: 0.0 */

Therefore, the code has a new hack: AFTER a line is parsed and "converted" to a number, a check is run to see if the variable that was used to set the coordinates is empty. If it is, then std::nan() is used to overwrite that section of the parsed line struct and later on, when it is important to know whether the variable was empty or not, std::isnan() is used to check. What a mess.
I updated the Github repository with the latest changes, all seems to be well now. I'm working on compiled binaries to put on this page, because that 1Gb of free space on Hackaday.io is tempting me!

Can anyone test this out? It should work on all platforms, I tested it on Linux and Mac OSX.

I have had no issues with the latest version in the repository, it produces no errors that I can see.
I hope this helps some of you out! Thanks!