SonarPen - earphone >> smart pen

I like digital drawing, and it would be cool if I can draw on the go. I want to develop a smart stylus that has all the standard features and a selling price that undercut everyone by at least 50%. (66% less compare to an Apple Pencil)

The de facto standard of a smart pen is to have the following functions (if not all) :

1. pressure sensing
2. palm rejection
3. shortcut button

So my smart pen must have all of the above and remain affordable. The earphone solution took away the all expensive electronics, controlling circuit, Bluetooth, rechargeable battery and replaced them with a circuit that communicates with an iPad through the standard audio channel. To detect the pressure, the iPad sends a standard wave form to the circuit, the wave form pass through a variable resistor which changes according to the pressure applied on it. The modulated wave form is then sent back to the iPad for comparison.

Other than cost, the earphone circuit is better in terms of:

- Almost all the iOS devices have the connector (the lightning to 3.5mm converter should works)
- Protocol (driver) will be simple, almost like how a modem works.
- Best of all, it is powered by the iPad itself, so no charging ever.

When I knew Apple is going to add a stylus for the iPad, I was pretty happy that It confirmed the need of a stylus for the iPad. However, selling it at US$99 and limited the compatibility to iPad Pro only, make the cost of owner ship to be too high for regular people.

If you are interested in the Making of the SonarPen, don't forget to visit the logs.

Project Logs

Collapse

Arduino as testing tool
Elton • 08/29/2017 at 15:50 • 0 comments

A wise old man once said, "A project without using Arduino or Raspberry pi is not a true Hackaday project." And since I really like SonarPen to be featured on the front page, there's no exception for my project. :)

However, in the case of SonarPen, Arduino was used as a development tool instead of the product itself or as a prototype.
At one point of the development process, I need to compare several different designs to find out which one is the most responsive. I created 5 prototypes, but then realized I didn't have 5 iPad to test the pens. The tests could be done individually but that would take too long and might not be consistent. Then I remembered I have read an article about using Arduino to read several soil moisture sensors at the same time. A soil moisture sensor is just a resistor, which is also what the pressure sensor is. I google the code and circuit. Within 30 mins, I had a pressure monitoring tool that could deal with 6 styli at the same time. The Arduino read all 6 styli and fed the values back to the PC through the serial monitor. It was a big improvement, but the running numbers of the screen were to fast to make sense. Lucky, the ARDUINO IDE 1.6.6 (2015.11.03) has added a very easy to use serial plotter. Without adding any additional code, the pressure feedback values can be plotted on the screen. It made testing and comparing 6 styli at the same time a piece of cake.

Connecting 5 stylus to the monitor.

Arduino IDE's built-in Serial Plotter, running numbers become easy to understand graph.
Choice of platform
Elton • 08/10/2017 at 16:34 • 0 comments

Choosing the iOS devices as the primary support platform is a no-brainer. It is the most popular platform and the audio hardware in different models are common (iPhone has taken away the 3.5mm jack :( ). On the other hand, Android and Windows will be much harder to support because of their huge hardware variation. The decision is to begin the development of the driver for these platforms after the sales of SonarPen has reached a certain level. That also means the hardware design of the stylus must compatible with all the platforms.

A good example would be the iOS detects the existent of a 1k ohm resistor between the MIC and Ground line to tell if the earphone has a MIC on it. Similarly, a Samsung Android device needs the resistor to be 1.4k ohm to confirm a MIC on an earphone. So my stylus has a 1.5k ohm resistor on its circuit for broader compatibility.

A little thing worth mention is all iOS devices sold in EU countries have a lower maximum earphone volume than the ones sold outside. This limitation forced the stylus driver must be able to self-calibrate instead of hardcoded to work at a certain volume. Drivers are needed to rewritten for different platforms, but the experience of creating a self-calibrate driver will help a lot during the development for Android and Windows devices.
Getting apps support
Elton • 08/09/2017 at 15:42 • 0 comments

A smart pen without a wide range of app support is only as good as a dumb one (maybe worse). Having my own drawing app as a companion would the best scenario to launch the product, because users will enjoy the full potential of the stylus out of the box. However, with limited resource, I would rather develop a good piece of hardware accompanied with an easy to use driver for third party apps developer to add support for my pen.

Despite I have a very good partner, Water, who has created a wonderful driver for the SonarPen. I still find getting app developers to support my smart stylus is the hardest task of all.

App developers are usually very busy. Adding support for a new stylus has very low priority in the task list. What makes the situation worse for a new stylus is some developers pick a stylus to support solely base of its popularity. That mentality put SonarPen in a chicken and egg situation. When there are not enough users, the apps won't support the stylus. And when there is no app support, users won't buy the stylus. That why I am extremely grateful for the trust of the developers of GoodNotes. They believe in SonarPen, an affordable but full featured stylus will success, because many iPad users don't want to spend $99 nor $70 for a smart pen.

I am still looking for more iOS apps to support my stylus. If you are a developer or you know a developer that may support SonarPen. Please contact me.

The following videos show the same hardware running in-house demo and GoodNotes. Quick demo VS. professional.

In-house demo app (early 2016).
The lacking is quite serious.
GoodNotes 4 wtih SonarPen (early 2017).
Almost NO lack, width of a stoke is easy to control.
Adding a shortcut button
Elton • 08/08/2017 at 12:35 • 0 comments

At the summer of 2016, I have made a bunch of evaluation sample and starting to reach out to app developers for their opinion. At the meeting with GoodNotes' developers, the kind gentlemen shared their view on the smart stylus and how I could improve mine. When they found my stylus has no shortcut button, they expressed concern as it is a common feature used by many notes taking app users. I had yet figure out how to add a programmable button to the stylus without complicating the existing circuit. But I knew it is an essential feature that must be added, so I told the developers that it will be added in the next revision. On the way back to the office, I listened to music as usual. A call came in, I naturally reached out to the earphone remote to answer it. As soon as I pressed on to the answer button, I realized the simplest way to add a shortcut button to my stylus has always been there, I have just overlooked it. The answer (play) button on the remote is the perfect shortcut button. All I needed to do is to put a micro switch between the MIC and Ground. Since then the three standard smart stylus features are supported.
Play/Pause button is used as programmable shortcut button on the smart stylus
Manage the cable (cont.)
Elton • 08/07/2017 at 16:31 • 0 comments

Various designs have been explored to deal with the cable. The method should:

- Be able to roll up a 45cm long cable
- Have low learning curve
- BE self-explanatory
- Not block the user's grip
- NOT be the weakest link of the stylus (most important!)

And here are the major Pros and Cons of different designs:

Removable/replaceable cable - Pro: cable will never be the weakest link. Con: cable is not managed.

Spring equipped retractable system - Pro: simplest. Con: too much fun to play with, invites people to mess with it which leads product failure.

Cable wraps around the pen - Pro: easiest to manufacture, easy to perform. Con: looks amateur, blocks the grip when cable is NOT released.

Sept 2016 marks the turning point of SonarPen. I realize if the cable wraps around the top part of pen vertically, the design will look tidier than around the pen. It will meet the above requirements, and very easy to manufacture. Based on the same concept, a few more revisions with a different way to wrap were developed. The one that has the lowest learning curve and parts count made final.

Early prototype with retractable cable

In 1 and 2, cable is wrapped around the pen when it is not in use.
In 3, cable is rolling onto a wheel at the grip.

No moving part, low learning curve cable management system

The soft logo tag is doubled as a handle to release the cable.
Choice of 3D printing technology
Elton • 08/05/2017 at 09:25 • 0 comments

3D printing is the greatest gift for Makers. However, if you are serious about developing your own hardware, invest in an SLA 3D printer or send your prints to a lab with SLA equipment. It is especially true when the design has a lot of fine details.

SLA printed parts has very fine surface. Most of the time, I can just spray paint it directly. Also, SLA has much tighter tolerance than ordinary FDM printer. When printed parts needed to tight fit together, SLA will save you a lot of time to get rid of that extra 0.3mm layer of plastic.

It is very important for the SonarPen prototypes to have minimal manual adjustment because the adjustment is hard to repeat. When the team members have prototypes that are different and behave differently, the works produced are useless. Thanks to SLA technology, my team didn't have such issue.

Spray painted SLA printed part(above) vs FDM printed one(below)
Choice of pressure sensor
Elton • 08/05/2017 at 03:30 • 0 comments

The idea of a super affordable smart pen was born when I was playing with the earphone volume slider while listening to music. I realized my force was applying to the volume slider and music was changing accordingly. This force (pressure) could be measured by the iPad if:

1. A housing is holding the slider vertically
2. A sliding rod is linking the nib and the little handle on the slider
3. A standard waveform is played instead of music
4. The volume slider modified waveform is sent back to the iPad through the MIC channel, for a comparison

As a proof of concept, the volume slider is the perfect pressure sensor. Despite the Taiwan volume slider factory is so helpful and made me batches of free sample. I have to switch to a Force Sensing Resistor, FSR because volume slider has a tolerance that is too big to be compensated by the driver and the resolution of a volume slider is NOT fine enough to reflect the light stroke performed by an artist.

Although the made in USA FSR costs 30 times more than the slider, its performance is professional. It’s worth it.

Volume slider provided by the Taiwan factory

Proof of concept 3D printed housing and drawing
A hybrid smart stylus
Elton • 08/04/2017 at 16:38 • 0 comments

Conventional smart styli are self-contained. They all have a controller to detect the applied pressure and the XY location of the pen on the touch screen. Through Bluetooth, this information is sent to the iPad and to be used in an app. This 100% hardware solution leaves very little for the iPad to do.

The draw back of this design is, it can only be used on supporting apps. (except Apple Pencil)

SonarPen is a hybrid design. The detection of XY location of the pen is handled by the touch screen, like any dumb stylus. The pressure sensor is read by the audio circuit of the iPad. A piece of code runs on the iPad to combine the XY and pressure readings to provide palm rejection function.

For SonarPen to work, the iPad needs to do more than one thing (the app and SonarPen) at a time. In our test, even a 2012 released iPad gen 3 is powerful enough to handle the extra SonarPen.

The cool thing is, this hybrid design allows SonarPen to work on apps, even if it is not written to support the stylus.
Choice of cable
Elton • 08/04/2017 at 07:59 • 0 comments

Cable plays a major role in my smart stylus project. At the very beginning, thin woven cable was the preferred choice as it gives the product a cozy feeling. But when I tested it against the demo app on an iPad, the pressure reading was very unstable. The issue got worst when I am moving the pen in the air. It took me days to realize it was the cross talk from the 4 Enameled wires. Woven cable leave NO space between the wires. When the wires rubbing each other or bend in different angles, the resistant of the cable changes = jumping pressure readings.
At the end, the thin flat "noodle" like silicon cable is the choice because its wires are separated = good stability.
Some of the cables that we have used in prototypes. Round, thin, woven, color, thick and flat ones.
Manage the cable
Elton • 08/04/2017 at 04:54 • 0 comments

Through out the 2.5 years, more than 24 major prototype designs and countless minor modifications are created.

As a wired stylus, a dangling cable will drive away the users. Adding a mean of cable management make a lot of sense but making it not too ugly is hard.

Final design, anyone can manage the cable in 5 seconds.