Gaze serves as a fundamental method of targeting in the realm of mixed reality. Through the gaze cursor, users can direct their attention in both the physical world (especially when spatial mapping is employed) and interact with holograms. Complementing this, users can engage with their environment through gestures. The key avenues for user actions encompass gestures, motion controllers, and voice commands. Among these, the principal gestures include the air tap, double air tap, hold, and drag; moreover, users also have the option to devise custom gestures to suit their specific needs.
This guide is designed to help you build a HoloLens application from the ground up, providing hands-on experience with the gaze cursor and various gestures. Throughout the development process, we will leverage the Unity platform for building the application.By following this guide, you will gain practical knowledge and proficiency in utilizing the gaze cursor and implementing gesture interactions within your HoloLens applications. This hands-on approach will empower you to create immersive mixed reality experiences.
Create a new Unity project and add a scene named ‘GazeGesture’ to the project. Initially, you need to make some adjustments to enable and build the HoloLens application successfully.Navigate to ‘Edit’ -> ‘Project Settings’ -> ‘Player.’ On the ‘XR Settings’ tab, check the ‘Virtual Reality Supported’ checkbox, and add ‘Windows Virtual Reality’ to the list.To configure the project, go to ‘File’ -> ‘Build Settings.’ In the ‘Universal Windows Platform’ tab, select ‘HoloLens’ as the device and ‘D3D’ as the build type.
It’s crucial to configure the camera settings in your scene correctly. Set the camera projection to ‘Perspective’ and position it as shown in the accompanying picture. The camera should be situated at coordinates (0, -0.02, 0) with a rotation of (0, 0, 0) and a scale of (1, 1, 1). This camera’s placement determines the point from which all other objects in your virtual space will be initialized.
private MeshRenderer meshRenderer;
void Start ()
{
meshRenderer = this.gameObject.GetComponentInChildren<MeshRenderer>();
}
In the Update method of the ‘GlobalCursor.cs’ script, implement a check to determine if the user’s gaze or pointer is currently intersecting with an object. If there is an intersection, ensure that the MeshRenderer of the cursor is displayed on the HoloLens. If no intersection occurs, hide the MeshRenderer of the cursor.”
var headPosition = Camera.main.transform.position;
var gazeDirection = Camera.main.transform.forward;
RaycastHit hitInfo;
if (Physics.Raycast(headPosition, gazeDirection, out hitInfo))
{
FocusedObject = hitInfo.collider.gameObject;
}
else
{
FocusedObject = null;
}
if (FocusedObject != oldFocusObject)
{
recognizer.CancelGestures();
recognizer.StartCapturingGestures();
}
Create a new GameObject named ‘Cube’ and attach a ‘Box Collider’ component to it.
Now, you can export the project to Visual Studio and deploy it to your HoloLens device. When you look at the ‘Cube’ object you created, the cursor should be displayed.If you add the ‘Spatial Mapping’ component to your Unity project, you’ll notice the cursor appearing over all objects in your real-world environment.To enhance your HoloLens experience, let’s implement three gestures: air tap, hold, and cursor over. You can assign the corresponding event names to the mixed reality cube for each gesture.
First, add the event names to a class named ‘Constants.cs’.
public class Constants {
public static string CURSOR_OVER = "Cursor over";
public static string TAP = "Tap";
public static string DOUBLE_TAP = "Double Tap";
public static string HOLD_START = "Hold start";
public static string HOLD_COMPLETED = "Hold completed";
public static string CLEAR = string.Empty;
}
Add the C# scripts ‘GazeGestureManager.cs’ and ‘TextMessage.cs’ to a new GameObject named ‘GazeGestureManager.’ Also, add the ‘cubeGestures.cs’ script to the ‘Cube’ GameObject. In ‘GazeGestureManager.cs,’ initialize the events. When any of the events becomes active, display text messages on the ‘TextMessageCursorOver’ and ‘TextMessageGestures’ GameObjects.
GazeGestureManager.cs file
public class GazeGestureManager : MonoBehaviour {
public static GazeGestureManager Instance { get; private set; }
public GameObject FocusedObject { get; private set; }
public GestureRecognizer recognizer;
private bool updateFocusedObject = true;
void Awake()
{
Instance = this;
recognizer = new GestureRecognizer();
recognizer.TappedEvent += (source, tapCount, ray) =>
{
if (FocusedObject != null)
{
updateFocusedObject = true;
FocusedObject.SendMessageUpwards("OnSelect");
}
};
recognizer.HoldStartedEvent += (source, ray) =>
{
if (FocusedObject != null)
{
updateFocusedObject = true;
FocusedObject.SendMessageUpwards("OnHoldStart");
}
};
recognizer.HoldCompletedEvent += (source, ray) =>
{
if (FocusedObject != null)
{
updateFocusedObject = true;
FocusedObject.SendMessageUpwards("OnHoldCompleted");
}
};
recognizer.HoldCanceledEvent += (source, ray) =>
{
if (FocusedObject != null)
{
updateFocusedObject = true;
FocusedObject.SendMessageUpwards("OnHoldCompleted");
}
};
recognizer.StartCapturingGestures();
}
void Update()
{
if (!updateFocusedObject)
{
return;
}
GameObject oldFocusObject = FocusedObject;
var headPosition = Camera.main.transform.position;
var gazeDirection = Camera.main.transform.forward;
RaycastHit hitInfo;
if (Physics.Raycast(headPosition, gazeDirection, out hitInfo))
{
FocusedObject = hitInfo.collider.gameObject;
}
else
{
FocusedObject = null;
}
if (FocusedObject != oldFocusObject)
{
recognizer.CancelGestures();
recognizer.StartCapturingGestures();
}
}
}
TextMessage.cs file
public class TextMessage : MonoBehaviour {
public static TextMessage Instance { get; private set; }
public GameObject textMessageCursorOverGmObj;
public GameObject textMessageGesturesGmObj;
void Awake()
{
Instance = this;
}
public void ChangeTextMessage_CursorOver(string textMessage)
{
textMessageCursorOverGmObj.GetComponent()<TextMesh>.text = textMessage;
}
public void ChangeTextMessage_Gestures(string textMessage)
{
textMessageGesturesGmObj.GetComponent()<TextMesh>.text = textMessage;
}
}
GlobalCursor.cs file
...
this.transform.rotation = Quaternion.FromToRotation(Vector3.up, hitInfo.normal);
TextMessage.Instance.ChangeTextMessage_CursorOver(Constants.CURSOR_OVER);
}
else
{
meshRenderer.enabled = false;
TextMessage.Instance.ChangeTextMessage_CursorOver(Constants.CLEAR);
TextMessage.Instance.ChangeTextMessage_Gestures(Constants.CLEAR);
}
...
CubeGestures.cs file
public class CubeGestures : MonoBehaviour {
private bool holding = false;
void OnSelect()
{
TextMessage.Instance.ChangeTextMessage_Gestures(Constants.TAP);
}
void OnHoldStart()
{
holding = true;
TextMessage.Instance.ChangeTextMessage_Gestures(Constants.HOLD_START);
}
void OnHoldCompleted()
{
TextMessage.Instance.ChangeTextMessage_Gestures(Constants.HOLD_COMPLETED);
holding = false;
}
}
You can find this sample and download it from GitHub at this link https://github.com/gntakakis/Hololens-Gaze-Gestures-Unity
GNtakakis's Blog 