Please use Chrome or Firefox. IE explorer will encounter some format issue.


Welcome To next ar!

Next AR is the current project team I am working with right now. I work with two designers, one producer, and one other engineer, to create a passthrough AR experience by using Unreal 4. This is the group picture of my team, and I am the rightmost guy. 

What is NEXT AR project?

Next AR is an R & D project sponsored both by Carnegie Mellon University and Electronic Arts in Redwood City, California. Its purpose is to create a passthrough AR experience with social events. Players in different physical locations use hand gestures to interact with other players and objects through the Internet. I am building the basic tech stacks right now, and detailed information is provided down the page.  Here is a demo video on the tech stacks I implemented so far:



Carl Rosendahl

Carl Rosendahl

Carl Rosendahl

The Director of ETC's Silicon Valley campus. Creator of the Pacific Data Images.


Melanie Lam

Carl Rosendahl

Carl Rosendahl

A game producer, designer. Shipped the Sims Franchise and its expansion pack series in EA.



HTC Vive Pro

HTC Vive Pro

HTC Vive Pro

Has stable tracking ability.


ZED Camera

HTC Vive Pro

HTC Vive Pro

The depth camera mounted on Vive.


Leap Motion

HTC Vive Pro

Leap Motion

Hand gestures detector on Vive.

My progress

1) Setting up Hardware Environment

To incorporate both ZED camera and Leap Motion on Vive, I compiled a customized version of Unreal 4.21, which allowed ZED camera to use its rendering pipeline. During this phase, I spent plenty of time testing with the hardware and their APIs. To the left is a short video of me presenting hardware setup to EA faculties. 

2) Finished the first single player demo

I created a simple interactive single-player demo after the hardware had been dealt with. In this demo, players use their hands to grab and throw cubes. Also, I coded the finger drawing feature, which is popular in VR and AR. To the right is a short video of the single player demo. 

3) Adding network feature for multiplayer mode

Since having social elements is one of the client's requirements, our group's next step is to add a networking feature. I tried to use the default client-server network model from Unreal 4, and it worked. However, it is laggy for the moment. You can find the demo video to the left.


4) Fixed network lagging issue

 Fixing the laggy network became my next priority. Instead of using the default client-server model, I forced some network logics to run on client-side and inform the changes to the server. In this case, an individual client did not need to wait for the server to respond as logic was calculated locally. The server then broadcasted to all other clients about the changes.

5) Adding realistic physics

The half milestone was closing, and we want to add realistic physics to our experience to foster social activities with two people. Instead of using fixed constrains for grabbing objects, I integrated a free physics plugin made for controllers and integrated it onto Leap Motion. Plugin website is here:

To the left is the brand-new physics demo video.

6) Half milestone play test

During the half milestone, people from Electronic Arts headquarter play-tested the project demo. They provided precious feedback and were surprised to see the network feature implemented. To the right is the video of play-testing footage.


Working on AR features right now

Up till now, I have been building the tech stacks. Currently, I am discussing with the team on how we will utilize the AR feature and tech stacks to create a social AR experience. Challenge waits ahead.