holoplayer one: an interactive midair 3D display

Lemme start by showing you where we are today:

You know that scene from every sci-fi movie you ever watched? Where someone sits down at a fancy table, a 3D world floats up out of the table, and they reach into the world and start designing stuff, or poking around, or doing something else that feels futuristic and wonderful.

I grew up watching those movies, and whenever that scene came into my head, the engineer/inventor part of my brain would pipe up and say, "yeah, but how are they *doing* that?" That little voice never went away. It just stuck around, doodling ideas about oscillating swept-volume POV displays in my french notebooks in high school, tinkering with ideas about pigmented volumetric displays in my college hackerspace, making more spinning swept-volume stuff during internships, and so on and so forth. It just felt funny -- people had been imagining this stuff in fiction for ages, but nobody was talking about how to actually make this actually happen.

As luck would have it, about four years ago, my friend Shawn and myself found ourselves in an unexpected moment of unemployment. Facing the void will make you do weird things. In our case, it made us start a company to make the sci-fi world of holograms a reality.

Trying to make holograms real is a weird, nebulous problem. Over the past four years, we've invented and built about a dozen different technological approaches, all addressing various aspects of what the zeitgeist calls holograms: volumetric prints, swept-volume volumetric displays, LED cubes, projected volumetric displays and 2D aerial displays, to name a few. This buildlog isn't really about those technologies, though -- this is focused on our development of an interactive aerial display. If you want to check out earlier work, dig the historical link barf below

volumetric prints: https://www.kickstarter.com/projects/lookingglass/looking-glass-hologram-20

swept-volume volumetric displays:

LED cubes: https://www.kickstarter.com/projects/lookingglass/l3d-cube-the-3d-led-cube-from-the-future

projected volumetric displays: https://www.facebook.com/gizmodo/videos/10154388795908967/?video_source=pages_finch_main_video

Hey, you made it through the link catacombs! Great! Well, let's get down to brass tacks -- how to make things float, and how to interact with things that aren't there. This video is where it all began, some years ago, with a parlor trick that makes some LEDs appear to float in midair. Here's our first prototype amazing my partner Shawn's kids.

the rest? The rest is in the build logs.

Project Logs

Collapse

interaction
alex • 11/21/2017 at 13:17 • 0 comments

So now we've got floating 2D images. That's some sci-fi stuff right there. How can we touch them?
We need some way to track finger motion in midair. We'd worked with Leap Motions on a previous product, so they were a good choice for a first experiment. We set one up near an aerial display, and we were able to use it to track hand movement near the image. Enough for a quick demo.
It wasn't great for an actual product, though -- Leaps need a fair amount of processing power (like a modern macbook), which ruled out small, cheap computers driving the display. The cameras would freak out if they say an object in their field of view, or heaven forbid got a glimpse of retroreflective material. And finally, Leaps are utter magic 90% of time time, when they work, and are super frustrating the 10% of the time that they don't work. How would you feel if your mouse just didn't work 10% of the time? I would feel bad, too.
We had a serendipitous stroke of luck when we found a device called an airbar (https://air.bar/)
Airbar is a thin little bar designed to sit underneath your laptop screen, plug into your laptop's usb port, and turn a non-touchscreen laptop screen into a touchscreen. They work with a really clever technique where they have a 1D array of IR emitters and sensors and by scanning through all their emitters, they can piece together an image of what's in the plane of the sensor. As an added bonus, they don't actually need to be up against a screen to work -- they effectively turn a 2D plane of air into a touchscreen, and they look like a mouse to a computer. They were perfect for this!
We got super stoked about this idea, and we started rushing around making midair interface demos. Basic stuff like translate-rotate-scale viewers for 3D models. Midair painting, games, and more. It was super exciting!
Except for one nagging problem: when we interact with cell phones and tablet screens, we're using a 2D planar interface to interact with a 2D planar screen. We were doing the same thing here, except our screen was blurrier, dimmer, bulkier and harder to interact with. Why did it need to float?
Why did it need to float?
Many points of light
alex • 11/21/2017 at 12:46 • 0 comments

If the aerial trick works on one point of light, it'll work on many, right? If one point is a pixel, let's show a screen!
That was a pretty easy leap. Problem was, it didn't work so great. LCD screens are roughly one hundred billion times dimmer than an LED (not sure I did my math right there, check me). The cheap retroreflective fabric actually has some deliberate imperfections that make rays of light retroreflect, +/- a couple degrees. That inaccuracy is great if the fabric is on your jogging shorts and you want to reflect the car's headlights into the oncoming driver's eyes, but it sucks when you're trying to form a coherent image -- everything is blurry. The further the rays travel before converging at the virtual image, the blurrier each virtual pixel.
I don't have many images of this stage of experimentation. We were super frustrated.
We found a fix for the blurriness, eventually -- use better quality retroreflectors. They're more expensive, but at least now you don't have a holographic myopia simulator. Here's a later prototype aerializing a phone screen:
We found a smart trick to boost the brightness, too -- in the previous log, I described what happens when you're making the aerial illusion with a regular, clear piece of acrylic. You lose an awful lot of light -- anecdotally, I'd say that you lose at least 90% of the original light coming from the screen, but it's hard to say precisely because the actual measurement is dependent on the viewing angle. Suffice to say, it's a lot, and it makes the aerial image really hard to see.
A lot of that light is lost because the acrylic doesn't know which rays to reflect and which to transmit, so it just reflects some and transmits some other rays. But what if we could tell the acrylic when to reflect and when to transmit?
The key here is that light coming off LCD screens is polarized. When those polarized rays hit the acrylic, we want to reflect them -- 100% of them. The reflected rays then go down, hit the retroreflector and come right back. This time, we want to transmit all the rays. The retroreflector by itself doesn't change the polarity of the light, so we need to add something that'll flip the polarity from when the rays bounce off the acrylic to when they hit the acrylic a second time, and that something is a little film called a quarter-wave plate. A quarter-wave plate is a film that very precisely retards the E or B field of a passing photon, rotating its polarization by 45 degrees. If we shine some linearly-polarized light through a quarter-wave plate, we get circularly-polarized light on the other side. Put in another quarter-wave plate, and we're back to linear polarization, rotated 90 degrees from the original light. Cool, right?
So we put a quarter-wave plate in front of the retroreflector. Linearly polarized light passes through on the way down to the retroreflector, rotates 45 degrees, bounces off the retroreflector and then rotates another 45 degrees on the way back up. And just like that, we've made an optical way to distinguish between the light we want the acrylic to reflect and the light that we want it to transmit.
So how do we make the acrylic change its behavior? Well, it turns out that there's another material called a reflective polarizer that reflects light at one polarization and transmits it if it's another. You can get it in a stick-on film, so we just stick that to the bottom of the acrylic and we're good to go.
This weird old trick not only reduces belly fat, it also increases the brightness of the aerial image by about 4x above our initial approach.
So now we've got a crisp, bright aerial image. What's next?
We want to touch it.
making things float
alex • 11/21/2017 at 12:21 • 0 comments

I call this the beer bottle trick. Great at parties.
It's the simplest way to make the retroreflective aerial illusion. You'll need:
++ some retroreflective material. I like 3M's scotchlite because it's dirt cheap, easy to get on ebay (http://www.ebay.com/bhp/3m-reflective-fabric) , and you can crumple it up and wad it in your pocket without feeling bad
++ some flat, clear material. Glass or acrylic is good
++ a spacer. Here, I'm using beer bottles
++ an LED or other bright light source
Here's how it works: Put down the retroreflective material, face-up, on a table. Beer bottles go on top, and your acrylic lays on top of the beer bottles. Someone holds an LED in-between the acrylic sheet and the retroreflective material, and whooahhhh, a virtual image of the LED appears *above* the acrylic. You wiggle it around a bit, and you realize that it's like the acrylic is a mirror, and you are on the other side of the mirror, hanging out in the same space as the virtual image in the mirror.
Crazy, right? What's going on here?
Think of the LED as a point source of light -- that means that rays of light are coming out of it in every direction. Some of those rays(1) come up and hit the acrylic sheet. Some of the rays that hit the acrylic (2) get reflected back down off the underside of the acrylic. The reflected rays go down until they hit the retroreflective fabric. The fabric is retroreflective, which means that if rays of light hit it, they bounce right back to whence they came (3), and some of *those* rays pass through the acrylic, continue to pass through a point on the other side of the acrylic that happens to be at the exactly mirror position of the LED, and continue on into oblivion.
One light ray doesn't look like much, but consider what happens when you have a bunch of rays all following the same geometry:
All those rays intersect at the same point above the acrylic and then keep on going. We can't see the point where they intersect, but we *can* see them as they keep going and hit our eyes. When you get different images in each eye, you're getting a stereo depth image that your mind reads as an object at the intersection of those two images. In other words, you see an LED floating above the acrylic. Weird!
And it's not just you! Anyone in the same room would see the same illusion (the visible limits of the illusion are based on the size of your retroreflector, acrylic and beer bottles). If you all try to touch the floating LED, all your fingers would meet at the same spot.
Here's my partner Shawn walking you through that same illusion on video, with eight floating LEDs.
So, this feels really interesting. This is a cheap trick that's easy to set up and can make a point of light appear to float in midair. What next?
Obviously, many points of light.