Dan MaloneyHi Raj, welcome aboard! We'll get started in a minute or two
Hello and welcome back to the Hack Chat. I'm Dan, and I'll be moderating today along with Dusan as we welcome Raj Nakaraja for a chat about Open-Source AR.
Hi Raj, welcome. Care to start us off with a little about what you do?
Hey Dan! Excited to get started :)
Thanks for the warm welcome! I'm Raj, head of Engineering at Brilliant Labs. We're a startup making open-source AR/heads up display glasses, and my job is leading the hardware and firmware design
So, as you can probably tell from the way I promoted this, I tend to think of AR as being able to see other parts of the EM spectrum as an overlay on the regular view of the world. I suspect that's not all there is to it, though. How far off the mark am I?
Nto gonna lie -- kinda want a Geordi LaForge visor ;-)
Haha, that's a pretty cool idea honestly. I imagine myself being bombarded with colors as soon as a microwave turns on, or the phone rings. Well, there are tons of use cases for AR glasses. One favorite of mine was a startup a while ago that was doing a circuit overlay on top of PCBs. The idea was that you could see the current and voltages in realtime while you're probing your board
I think they got bought up by one of the big PCB software companies and I never heard of them again. We want to avoid that happening by being open source and building a community around our devices
Yeah, that company was started by a guy we had on the chat a couple of times. It was a really good idea that just sort of disappeared. Forgot all about it, actually.
If anyone wants to recreate it, they can try it out on our Monocle!
Can you go into Monocle a bit? I'm particularly interested in the optics -- having a big goofy thing stuck to your face always seemed like a barrier to general AR, but Monocle is much more streamlined.
Brilliant Labs
Pocket-sized AR for imaginative hackers. At Brilliant Labs, we're building an open-source ecosystem to support developers and creatives reimagining the future.
For sure! Well firstly, here's the full datasheet: https://docs.brilliant.xyz/monocle/monocle/ There's a brief description down the page on the optics, but in short it's a tiny micro OLED which projects down to a beam-splitter. The image then appears as a screen floating a couple of meters in front of you
The reason I ask is for those of us with "old eyes". Seems like we'd have a hard time focusing up close, but maybe that's not how it works, since the apparent image is so far off
Exactly. It looks like a tablet screen at arm's length basically
The display is driven by an FPGA which allows for some nice graphics acceleration, and Bluetooth networking is done via an nRF52. The nRF runs MicroPython, so actually, Monocle is totally python driven
Additionally, there's a 5MP camera, microphone, leds, and a couple of touch sensors
Would you want to use this on your dominant eye? Or would you put it on the weak eye and leave the dominant eye free to see the real world?
This is how it looks in real life. You can compare the text size here to my laptop screen
What is the experience like I mean I've always had a hard time seeing through a microscope where the instinct is to close the other eye but you are really supposed to mentally turn off the eye not looking through the microscope?
I personally put it on my weaker eye. It's not really something you have to focus on, in fact, the focal distance is technically at infinity, so it's always in focus. We've put the display below the horizon so it's like looking down at a book or your phone
I'm curious what you have found most useful to use your Monicle for day to day, are you focused on any particular use cases?
Text requires a bit of focusing I find, but for shapes and icons, it's not so much of a problem. There it can become more of a background thing
Is there some kind of focus for nearsightedness as the focus at infinity is a blur w/o corrective lenses?
There are quite a few things popping up. One example which went sort of viral was a Stanford student who connected it to ChatGPT. He'd use voice to ask it a question, and then it would print the response on the screen. I can find the tweet if you're interested. Another guy made a gym training gadget with timers and things. A lot of dev tools popping up were folks wanted info about their PC or network or something like that. All just running from python scripts over BLE from their PC
As for use cases, I keep coming back to "Wouldn't it be cool for surgery?" On the one hand, I don't want a surgeon who needs help finding his or her way around the surgical field. OTOH, I want the best possible outcomes from surgery. Tough to say where AR would fit-- maybe for training?
Or maybe it overlays the patient's vital signs?
For sure! It's still quite early for this type of stuff, and I think we're days from things going crazy when Apple makes an announcement next week. There will be a lot of amazing immersive AR stuff coming, but we're starting off pretty simple. Monocle doesn't take up your whole vision, isn't 3D, and doens't track the environment in any way. However, as this industry evolves, we're going to see better displays and better controllers where it's possible to really overlay things onto the real world smoothly and without bulky hardware
A worry with AI is when it hallucinates, what if in the middle of heart surgery it decides you are Vulcan "The insides of a Vulcan are completely different from a human's insides. They must be, if a Vulcan's heart is right around where a human's liver would be. This is great if a Vulcan were to get wounded in the chest region. But this is not so good if a Vulcan were to get wounded on the right side of their abdomen"!
On that journey, our ideas of AR experiences will evolve. Who would have imagined all the smartphone apps we have today, before the smartphone arrived. For sure there'll be a lot of cool ideas
Yeah, the same probably goes for AI. It's still so early and we probably have to be a bit careful in what we connect the AIs to. As they evolve though, maybe they will become mature tools that can genuinely offload the gruntwork we do every day, and let us focus on getting more done quicker
I definitely think AI and AR will become intertwined though. That's partly why we added the camera. We hope someone figures out how to implement interesting AI models on the FPGA that make use of the camera
What's the battery life like? Seems like this might chew through batteries pretty fast
This is the GPT app on Monocle https://twitter.com/bryanhpchiang/status/1639830383616487426?s=20 It was with quite early firmware and he threw it together in a weekend. Most the processing is happening on his laptop, but we're working on a real version of this app that goes directly via a phone up to openAI
Will general acceptance in public be a problem as the Google Glass found out or is a matter of desensitization takes place it will become less weird just like people talking to themselves in public has become commonplace with the advent of the smartphone.
Also, people just point their phone's camera at everything, everywhere and anytime. Why do people have different feeling about a smartphone and smart glasses when they are capable of the same?
The normal runtime without the camera running is 45 mins, and with the camera it's half that. Monocle comes with a charging case however that has a battery inside. You can get 6 charges out of the case
Not that bad actually, all things considered.
Yeah, I think cameras have become so common now that most folks aren't all that bothered. Google Glass was so early in that regard, it almost feels like a totally different era back then
However, I do think people talking to themselves and pointing at things in the air probably won't be the way forward. That's a really tough UX problem. In our next product (Frame) we'll have a control ring that will work as a scroll wheel. You'll be able to look around and then click/scroll from the ring to interact much more subtly
Back to the optics for a second: that seems like it would be a tough thing to build. What's the manufacturing process like?
Yeah, manufacturing is not easy, but out of all the different display technologies, it's still giving us great yields. We worked with an amazing optics partner who did tons of science on this for us, and we even got a patent on it. Mostly to prevent patent trolls. There's a lot of science in the patent which you can find online. The optics are two different types of plastic that are bonded with optical glue. The display is also bonded. Each plastic piece is simply injection molded and assembled in a jig
Sounds like a cool process, I'll have to read the patent.
I remember looking at pages of documents they sent over with fringing diagrams, distortion, ghosting, and tons of stuff like that. Each was some trade-off for another. Then they were like. "Pick the one you prefer"
I recall doing an article on injection molded lenses, and seem to recall it was pretty challenging for the home-gamer. But not impossible
The science is way over my head, but our CEO Bobak has become a real expert in this stuff. If you want to know more, you can jump into our discord and ask him. We also have the optics partner in there who pitch in sometimes
Yeah the battery life for small wearables that are environmentally interactive is still a drawback, a possible solution would incorporate some type of wireless charging or even powering from something like a "docking" baseball cap with a larger capacity reserve.
Yeah for sure. We had to go to a pretty special manufacturer to get the precision needed for these
There's one guy in our discord who's made a couple of 3D-printed charging cradles. You couple probably extended it into a wearable battery-type thing. The charging pads on Monocle simply take 5V as an input
*you could
https://github.com/jdc-cunningham/bl-monocle-stuff There's one down the bottom
OK then, just noticed it's after 1:00, so we have to let Raj get back to work. Me too, actually ;-)
Raj, thanks for stopping by today, this was very interesting and I appreciate your time. And thanks to everyone for coming along for the ride!
Thanks for the opportunity, and great talking to you all :) If you're interested to learn more about Monocle (and our future stuff), be sure to check out the docs: https://docs.brilliant.xyz, our website, and the Discord where we hang out. Catch ya round!
Thanks Raj!
Discussions
Become a Hackaday.io Member
Create an account to leave a comment. Already have an account? Log In.