How do you make a Virtual Reality Glove? - Smarter Every Day 191

Hey, it's me, Destin. Welcome back to Smarter Every Day. I want this video to be long, and I want it to get down into the weeds and just air out and let me get as technical as I want to. In the last episode of Smarter Every Day, you got to see me interact with the Haptic Glove, and it changed my mind about VR (Virtual Reality).

Here's the deal: when you're a start-up company, it's a double-edged sword, right? You want people to know that you're doing these great wonderful things so you can get investors and stuff like that. But conversely, you don't want them to know the "secret sauce" because you gotta maintain your competitive edge, right?

So, the fun thing about this is when I got down to talking to the engineers at like the fundamental Haptic Pixel level on how this thing works, you can see their eyes dartin' around the room; they freak out. The reason they freak out is because they haven't been permitted to talk about how this glove works in forever apparently. Then this dude walks in with a camera, sticks a camera in his face, and asks him very technical questions. It's really fun to see their reaction!

But I was able to sign some paperwork, and we were able to work out a deal so that I can take you right up to the edge of proprietary information, but not cross that line so that they can still maintain their edge. So after I met the founder Jake and the co-founder Bob, they turned me over to the head of engineering ops, Lathe, to ask questions.

In the last video, we went through what the experience was like. This time, I just wanted to focus on how everything works. I decided to just go for it and ask Keenan what kind of actuators they used to simulate touch.

“So, I am going to wear a glove under the glove.”
“Correct, it's just for sanitary purposes.”
“I am really excited right now.”
“So the question I have is, if...”
“If I touch on my hand right here and I am supposed to feel the presence of something there.”
“In order to have an actuator in that location, you have to have a ton of different valves, for example, if you’re using pneumatics.”
“Or, you know, linear actuators if you're using electro-mechanical actuators.”
“Yes.”
“So how do you get around that problem?”
“We describe them as like Haptic Pixels, and each Pixel would be its own... pneumatic actuator, and they are arrayed over your hand.”
“Okay, I'm ready for it. Yeah.”
“Okay.”
“How many of these exist?”
“Umm...”
“I don't think I can share that.”
“Okay.” (both laugh)
“Cool.”
“So we'll just pull this down and I'll help you.”
“Put these on.”
“Ah, so you're aligning it to my finger. Oh, wow.”
“Yup.”
“Well, what are these?”
“So, we...”
“We, uh..., ahh...”
“They don't want to talk about anything!”
“Okay, so we developed a custom motion tracking solution that uses both optical and magnetic tracking to get really accurate positions of each one of your fingertips.”
“Yes.”
“And then we do some crazy stuff in the software, and using all that information, we can figure out where all your fingers are in space.”

“Yes, and what I learned from my class I'm taking at school is, there is a lot of coordinate transformations happening in my hand right now. Right? Are they happening in my hand or are they happening off my hand in the processor?”
“They're happening in the computer. I know the math, and that's awesome!”
“Alright, let's do it!”

After running the simulation, I learned one of the most important concepts. It's called the two-point threshold. You had to have done a study on this. You figured out the specific bladder density that you need in order to trick the brain, knowing that it's not gonna go photorealistic and feel like complete immersion. Is that true?
“Yeah. There's actually a medically known metric called the two-point threshold. Which is basically if you take two pencils and poke somebody on the skin, how close together do those get before they can't tell that it's two separate points?”
“I keep calling them bladders; what's the actual word?”
“Ah...we call them tactors.”
“So the spatial resolution of the tactors just has to be inside of the two-point resolution of the human hand. It's a little more complicated than that. I mean, the two-point threshold is one of very many aspects of touch, and I am not the most qualified person to talk to you about that. And we do have a couple of folks here that can talk for a very long time about it.”

“So I'm Destin.”
“Hey, I'm Adam.”
“So how do you develop software for something that has never been done like this before? Like do you create your own language for the glove?”
“So...the way the software works is that it takes information from the physics engine, and then we have to figure out how to render that. And you know, it's pneumatic. We have to figure out how to open and close valves in the right order, and all that sort of thing. So the software does that.”
“So you have 150-something valves or whatever it is you have to control, and you have to control it almost real-time. You know your latency, don't you?”
“Yeah.”
“Yeah. So, and you're not gonna tell me that I'm assuming.”
“I'm not talking about that.” (both laugh)

“So you have to... It has to be quick enough where a human can think that they're actually engaged with it. So there is a threshold there. There is a threshold, and they've done studies on it. They found that anything longer than a quarter of a second is too long. You lose the temporal connection between the things.”
“But if you go below like...you know...half that, then the brain...”
“You're fine?”
“Yeah, your brain doesn't know.”

“They're discussing about whether or not I get to see a tactor. I really wanna see a tactor.” (laughs) “Who do I talk to to see a tactor?”
“Do you think... do you think Lathe will let me see-?”
“Ok.”
“We're looking at the website for that.”
“They're looking at the website. Because that's one of the rules we were given. If it's on the website, then we can show you.”
“Gotcha. Ok cool.”

“Have you ever met someone that you just click with? You're on the same wavelength instantly? This is Mark. The other engineers call him 'Mr. Glove.' This is the guy that knows all the things. The fun part about this conversation is if you watch Mark's eyes, it's clear that he just wants to 'geek out,' but he can't, because someone set up this proprietary borderline that he can't cross. Anyway, this is one of the most enjoyable conversations I've had in a really long time.”

“So that's Mark.”
“Hi.”
“So...this is it, huh?”
“Yeah, and they're really pretty simple. As you said, they're pneumatic; as you guessed, they're a bladder. So right here I'm looking at the end.”
“So I'm seeing...how many is that, 12?”
“Mhm.”
“What do you call- so this is a tactor. So what are these individual dots called?”
“We actually call the individual dots tactors and the whole thing is a panel. So we group a bunch of tactors together on a panel; it makes it easier to route air to them, makes it easier to support them, and we can control each of them individually.”

“Look, I know we're in the weeds here, but Mark's about to describe a bunch of neat little things about your hands. So if you'll check out your hands and do these little activities as he's describing them, you'll understand things that you've never even thought about before. Like the squish of your skin as you touch a hard object. If you will use your hands to do them as he's talking, you will understand the sense of touch on a much greater level.”

“So this is the spot where the research stuff that you have to get into kind of moves away from the traditional electrical mechanical engineering and kind of moves into how the brain works. It moves into ergonomics, it moves into human factors, and it moves into psychometrics.”
“Psychometrics...I don't even know what that means.”
“It's how people process things. It's how your brain works for sensation.”

“I don't know if you've ever seen something called the sensory homunculus. It's a picture of a person scaled relative to how many nerve endings they have in different parts of their bodies. It's basically a surface density of your tactile receptors.”
“Is that a good way to say it?”
“It's a very good way of saying it.”
“These panels, we think a lot about the huge hands part of that. They're incredibly sensitive. And so figuring out how to design a piece of equipment that is high fidelity and capable enough of tricking those sense organs is a lot of what our challenge is.”

“Okay, so you're inflating this particular tactor here, right?”
“Right.”
“It's like a balloon, right?”
“Mhm.”
“If I were to touch this table with my hand right now, okay?”
“Touch the table.”
“So it's not a step function; it's a curve, but it's not linear either.”
“Right.”
“Because as you press in on something, there's thickness to my skin, and so it starts as a line and then it ramps up, and then it gets harder and harder and harder, right?”
“Absolutely.”
“Is it an exponential curve? Do you know what that curve is? You don't have to tell me what it is.”
“You're hitting on a great question, which is what that approach and that curve of displacement looks like. Remembering that your fingers are sense organs. One of the ways they sense is by stretching, right? They're deformed. And you actually feel that deformation.”

“So there's 'what is your skin doing?' Which is, it actually gets stiffer as you push on it harder and harder...”
“And that's part of why we have such good...”
“Dexterity?”
“Right. The other part of it is 'what's the material you're pushing into?' In some cases, if that material is soft, well that's gonna deform as well. So you can have slightly different approach.”
“Yeah.”

“Like we're sacks of watery meat stuff, and so the relative difference between the hardness of the object you're touching is going to affect the attack time.”
“I don't know what term you used, but I would say attack time...”
“Rise time.”
“Rise time?”
“Mhm.”
“This is all making sense. There's also little things like how well does your glove fit? That's why when I put the glove on, it felt like there was a physical thimble that goes on the end of my finger is because you wanted a tactor to be aligned with my finger just like that, right?”
“Right.”
“Is that because you have to have something to react against?”
“Correct.”
“Is that why you did that?”
“Yep.”

“When you're touching something, it's not just the feedback you feel on your skin, but also the feedback you feel in your joints, your tendons, as you push against it. So you want to make sure to combine both of those. You're absolutely right. And I've never thought about that at all. If I grab something...if I grab this. And I squeeze it, I can tell how big it is because of where my tendons are. And I can tell how hard it is because of how hard I'm squeezing it and it's deflecting my skin between my bones, and all that goes into... it rolls up into one transfer function...to give you the answer of what object is in your hand.”
“Right. And that's where things get complicated. Your world revolves around boundary conditions, doesn't it? Like mathematical boundary conditions.”
“Pretty much, yeah. Boundary conditions and contact mechanics.”

“So what is this right here? This tape that I'm seeing? It looks like a tape measure that's pulling back.”
“That's the force feedback system. That's what stops your hand from closing when you grab an object.”
“It's a boundary condition.”
“Yes. It's a boundary condition. So instead of pushing on this side of the hand, you're actually pulling from the other side of the hand, and that's what stops the hand from moving.”
“Correct.”

“And so you have to have some kind of braking mechanism back here... So, can you crush things? Like if you have something virtually, can you set a resistance? Like if I were to touch something hard versus something soft, can you vary that force feedback?”
“(nervous sound) Yes.”
“Okay. Yes. Okay.” (laughter)
“These guys are like: 'everything's proprietary, just wear the glove, stupid boy.'”
“I think that one was just a complicated answer.”
“That was a complicated answer? That one is really complicated.”

“So something that matters here is, the entire volume... There's a time it takes to do that compression.”
“Right.”
“So there's a lag associated with... It's basically the elasticity of the volume, right?”
“What's the word? Well, there's a question of 'how fast you can push air through to this guy?' and that gives you how fast you can run that tactor. So it's this balance between having enough pressure to activate the system in fast time, granted you're going through a small tube.”

“So the effusion rate of the gas is important.”
“So you’ve got all that figured out. And there's mechanical lag associated with the effusion rate, or the head pressure is what it's called.”
“Yeah. With the velocity of the fluid.”
“The feedback diagram, or the block diagram of how perfecting this looks... it's gonna look like a plus and a minus between you two guys, right? And include some electrical engineers in there for good measure.”
“We don't—nobody cares about electrical engineers, let's be honest.” (laughter)
“Just pretend I didn't say that. I'm with Jeffery, an electrical engineer.”

“So Jeffery, I know how the tactors work. Or at least I've seen them. I think I figured it out.”
“You felt them?”
“I felt them.”
“I make sure we have good information about where the fingertips and the other parts of the hand are, so that we can turn on tactors at the right times to the right pressures.”
“Dude, that's awesome. I mean those valves that you're controlling, can you talk about that at all?”
“I can't talk about the type of valve we use, but the timing and the hydraulics problem that you mentioned are things we work on pretty frequently. It's important to get them inflated the right amounts at the right times. We kind of get a pass on how accurate that is just because of how slow sensation is.”

“Right.”
“You know people's perception of touch especially is not nearly as fast as things like vision or sound.”
“Especially thermally. I'm assuming that's something you guys are...”
“That's particularly slow.”
“Ed was telling me about the thermal component to this. Both of these objects are 73 degrees, but that metal feels way colder.”
“Yes, when you're touching these objects, you're not just feeling the temperature of the objects, but you're feeling the heat transfer between them and your hand. And that's because...The reason the metal feels colder is because it has a higher thermal conductivity than the wood does.”

“So it's like sapping more heat out of your hand.”
“That's exactly right, yeah. You're gonna model different types of objects by making more water flow or float a different temperature to pull more heat out of your fingers; that's basically what you're doing.”
“Yeah, that’s exactly right, and it's important that we don't just flow water at the right temperature, but we flow it at the temperature that you makes you think you're touching the real objects with different thermal conductivities.”

“Adam showed me another demonstrator where a dragon flies across the room and breathes fire on my hand, which I can feel both thermally and like touch, and then it breathes ice on my hand, which felt cold. Now what's clear to me is there they've got to be using fluid at this point, because that's the only way you can remove that much heat. So, I think there's a tactor for touch and a pad for thermal, but I think they're separated within the two-point threshold. They wouldn't tell me. I wasn't permitted to look.”

“But one thing that was really interesting, If you switch from hot to cold, it (snaps fingers) happened really fast. So they have to dump the hot from the lines and get cold in there fast. So there's a lot going on here, and I want to know mechanically how they do this. This is a heat exchange problem.”
“This was number two?”
“This is number two; you tried number three.”
“Awesome, so this is second generation. Ok, so it's just one hand stationary in location.”
“Oh God, that makes so much sense. If I were to develop this, I would have started at moving the hand in space and then add the haptics to that, but instead, you started with the haptics.”
“We knew people could do motion capture -- that's been done before -- so we wanted to start with the thing that hadn't been done.”

“Clever, keep your thumb beside your hand as you go in.”
“How are you doing that? I would expect that that's fluid in there or that's actually a solid that I'm touching.”
“You're not allowed to tell me? I'm looking at Lathe right now just to see.”
“(offscreen) It's hot and cold fluid.”
“Gotcha, it's a chunk of snow, it's a snowball; oh, it's a dragon! Oh, it's gonna burn my hand!”
“I don't think it likes being poked.”
“Oh, shit! That's awesome! Okay, that's good.”

“So you guys like Game of Thrones? So it's a combination of the tactors firing and temperature control. So you haven't implemented temperature control into--”
“Not in the glove; the glove does not have thermal.”
“Is that a goal?”
“I don't think I can talk about that.” (Laughter)
“That's awesome! So, what we've learned is it's how far the tactors are fired, the rate at which they are fired, so it's not just the rise time. The timing is important, not just the rise time, but the impact timing is important, the heat transfer with fluid, which is something obviously they don't want to talk about. Obviously, they're working on it. That's amazing that all that stuff actually tricks your brain. It does a really good job of tricking your brain, which I did not expect and I did not like VR up at this point. Seriously, that's pretty cool; we're living in the stinking future right now, and it reminds me of when I was a kid and used to watch Beyond 2000 with my dad. I feel like I'm getting to make that show now, which is cool.”

“Anyway, please consider going to 23andme.com/Smarter. That's support Smarter Every Day. Please consider subscribing if you enjoyed this video. If not, that's no big deal. Thanks for coming along on the ride with me.”

“Anyway, I'm Destin, you're getting smarter every day. Have a good one, bye!”
“So I feel like I sufficiently understand at least a little bit of the system. Yeah, thank you very much!”
“It's Mark, right?”
“Yeah.”
“So I have found the wizard for the glove.”