Meet the Guy Who's Bringing Defense Industry VR Patents to the Commercial Sector

Richard Boyd has been working on virtual reality since the 90s, back when virtual reality promised the world but failed to follow through. While working on "desktop VR" at the Virtus Corporation, he helped found Red Storm Entertainment with author Tom Clancy, which went on to become a huge name in video games with titles like Rainbow Six.

In 2001, he also co-founded 3Dsolve, which Lockheed Martin acquired in 2007 to work on several virtual reality and simulation technologies. During his time there, he had over 100 patent disclosures around virtual reality, which he's now trying to bring to the commercial sector.

Boyd left Lockheed in 2013, and today is CEO of Szl, a tech startup building AI-powered mobile apps.

Similar to our interview with VR pioneer Jacquelyn Ford Morie, Motherboard caught up with Boyd to find out what his vast experience in virtual reality can teach us about its current resurgence.

Motherboard: How did virtual reality become a big part of your career?

Boyd: When I first joined the Virtus Corporation we were working for James Cameron, who was filming the underwater science fiction film The Abyss. They were building this massive underwater set at an abandoned nuclear reactor in South Carolina. What we did, on a black and white Macintosh in real time 3D, is model the entire set and work with the cinematographer to walk through all the shots that were going to be in the film. We found out that a third of the set was never going to be on the screen, saving them a couple million bucks. Cameron and I have been friends ever since.

After we sold Red Storm to Ubisoft, I started 3Dsolve. I wanted to take gaming technologies and apply it to solving problems like education, health care, energy, space travel—the existential problems of our day.

What kind of work did you do at 3DSolve?

We did the first 3D simulation software that was validated the US army training and doctrine command (TRADOC). The military used simulation software in the Air Force before but they've never done it for small unit tactical training and that kind of thing. We pioneered that there until Lockheed bought the company in 2007. They were already interested in energy and space and I dragged them into health care as well.

While I was there I created a group called Virtual World Labs, which was an unusual thing. We had our own logo, we had our own website, we were operating like this little skunk works group within Lockheed. We had over 100 patent disclosures around virtual reality, augmented reality, different kind of input devices you can wear on your hands, including an operating system that ties all that stuff together.

I left after six years, and what we're doing now is licensing those technologies out of Lockheed because they only care about the government and military world, and we want to bring it into the commercial sector.

It looks like a lot of the new technologies we're seeing around virtual reality today—body tracking, different types of input devices—already existed in some form at Lockheed years ago, before the current VR boom. Where is all that tech?

It's frustrating. I left Lockheed at the end of 2013 and now it's 2015 and we're still going through the paperwork to get some of that stuff. There's a very thorough review for everything developed within the defense industry. You have to remember that everything developed there is using taxpayer dollars and there's always a heavy review of, "does this technology provide some kind of strategic value to the country? Should there be export controls on it so other countries don't get access to it?"

Even if you read the patent and think "oh this should be open to everybody, there's no reason to even go through the process," they have to go through the process anyway because it's just the way the defense industry works. It's slow, and thorough, but the pipeline is there.

One of the patents I co-authored was a holodeck. Some of this comes from when I was on the set of Avatar, where people were acting in this environment with hundreds of sensors that were capturing all their motions. I had a patent on how you do something like that [detailed, accurate motion capture] for the size of a football field and even be able to do it outside in broad daylight.

Some of those technologies are very interesting and I want to get them into the health care and education environment. I want kids to be going on virtual world field trips, to Mars and inside the human body. The technologies exists, we just have to get them out into a commercial environment.

The benefit of simulation training has been understood in the military for a long time. It's the shortest path to mastery and yet in industry and education we're really slow to apply a lot of these basic things.

Boyd at the 1994 Meckler VR conference.

What are the current limitation you see with virtual reality?

VR is cool for games, but still for the foreseeable future it's going to be for short sessions. If you ever tried to play a game, especially a fast first-person shooter, I've been doing this forever and even I don't want to do it for more than 30 minutes. I don't want to play Counter-Strike with stereoscopic display. I don't even want to play it on a 100-inch home theater setup at home. it make me sick.

I think that's going to be a problem until we upgrade the human brain.

Another emerging problem with virtual reality, now that the headsets have made so much progress, is input methods. What do you think about the hand tracking and motion controls solutions that Valve and Oculus have shown so far?

We actually worked with Alan Kay, who invented the Windows interface and laptop computer, at Virtual World Labs. One of the things he said is that one reason the mouse was so successful is that it translated small motions into big maneuvers on the screen. All those people who want to do

the Minority Report thing

, just try to do that. It's why the

data glove

didn't work in the early 90s. It looks cool but try to practically work that way for 30 minutes. your arms get tired. It's dumb. You could be moving your thumb ever so slightly to do the same.