Gaze is a primary form of targeting within mixed reality. With the gaze cursor the user is looking in the real world (if you use spatial mapping) and in the holograms and with the use of gestures they can interact with their environment. The primary way for a user to take action are via gestures, motion controllers and voice. The main gestures are air tap, double air tap, hold, drag and you can create your own gestures.
In this guide you can create a HoloLens application from scratch in order to learn to use gaze cursor and some gestures. For the development of the application we use the unity platform.
Create a new unity project and into the project new scene with the name “GazeGesture”. Initially, you have to make some adjustments in order to use and build successfully the HoloLens application. Choose Edit -> Project Settings -> Player. On the XR Settings tab, check the Virtual Reality Supported checkbox and add “Windows Virtual Reality” to list. To extract the project, go to File -> Build Settings …, there choose the Universal Window Platform tab and set Device to HoloLens and Build Type to D3D.
It is important to set in the Camera of the scene the projection to Perspective and to be in the right position as shown in the picture. Position (0, -0.02, 0), Rotation (0, 0, 0), Scale (1, 1, 1). Essentially, the point where you choose to display the camera is the point based on this which the other objects in space will be initialized.
private MeshRenderer meshRenderer;
void Start ()
{
meshRenderer = this.gameObject.GetComponentInChildren<MeshRenderer>();
}
In the update method, check if the user’s point is an object otherwise it does not display the MeshRenderer of the cursor in the HoloLens.
var headPosition = Camera.main.transform.position;
var gazeDirection = Camera.main.transform.forward;
RaycastHit hitInfo;
if (Physics.Raycast(headPosition, gazeDirection, out hitInfo))
{
FocusedObject = hitInfo.collider.gameObject;
}
else
{
FocusedObject = null;
}
if (FocusedObject != oldFocusObject)
{
recognizer.CancelGestures();
recognizer.StartCapturingGestures();
}
Add a new GameObject cube with the name “Cube” which must contain a Box Collider component.
Now you can export the project to Visual Studio and from there to run on HoloLens. See that if our look is in the box you created, the cursor is displayed. If you add spatial mapping component in the unity project you can see the cursor appear in all of the objects of our real world. Now, let’s add the gestures air tap, hold and cursor over and you ‘ll put the name of the event you use in the mixed reality cube.
First, add the names from the events to the class named “Constants.cs”
public class Constants {
public static string CURSOR_OVER = "Cursor over";
public static string TAP = "Tap";
public static string DOUBLE_TAP = "Double Tap";
public static string HOLD_START = "Hold start";
public static string HOLD_COMPLETED = "Hold completed";
public static string CLEAR = string.Empty;
}
Add the c# scripts GazeGestureManager.cs, TextMessage.cs to a new GameObject named “GazeGestureManager” and cubeGestures.cs on the GameObject “Cube”. In the GazeGestureManager.cs you have to initialiaze the events. When some of the events is active it show text message on GameObjects “TextMessageCursorOver” and “TextMessageGestures”.
GazeGestureManager.cs file
public class GazeGestureManager : MonoBehaviour {
public static GazeGestureManager Instance { get; private set; }
public GameObject FocusedObject { get; private set; }
public GestureRecognizer recognizer;
private bool updateFocusedObject = true;
void Awake()
{
Instance = this;
recognizer = new GestureRecognizer();
recognizer.TappedEvent += (source, tapCount, ray) =>
{
if (FocusedObject != null)
{
updateFocusedObject = true;
FocusedObject.SendMessageUpwards("OnSelect");
}
};
recognizer.HoldStartedEvent += (source, ray) =>
{
if (FocusedObject != null)
{
updateFocusedObject = true;
FocusedObject.SendMessageUpwards("OnHoldStart");
}
};
recognizer.HoldCompletedEvent += (source, ray) =>
{
if (FocusedObject != null)
{
updateFocusedObject = true;
FocusedObject.SendMessageUpwards("OnHoldCompleted");
}
};
recognizer.HoldCanceledEvent += (source, ray) =>
{
if (FocusedObject != null)
{
updateFocusedObject = true;
FocusedObject.SendMessageUpwards("OnHoldCompleted");
}
};
recognizer.StartCapturingGestures();
}
void Update()
{
if (!updateFocusedObject)
{
return;
}
GameObject oldFocusObject = FocusedObject;
var headPosition = Camera.main.transform.position;
var gazeDirection = Camera.main.transform.forward;
RaycastHit hitInfo;
if (Physics.Raycast(headPosition, gazeDirection, out hitInfo))
{
FocusedObject = hitInfo.collider.gameObject;
}
else
{
FocusedObject = null;
}
if (FocusedObject != oldFocusObject)
{
recognizer.CancelGestures();
recognizer.StartCapturingGestures();
}
}
}
TextMessage.cs file
public class TextMessage : MonoBehaviour {
public static TextMessage Instance { get; private set; }
public GameObject textMessageCursorOverGmObj;
public GameObject textMessageGesturesGmObj;
void Awake()
{
Instance = this;
}
public void ChangeTextMessage_CursorOver(string textMessage)
{
textMessageCursorOverGmObj.GetComponent()<TextMesh>.text = textMessage;
}
public void ChangeTextMessage_Gestures(string textMessage)
{
textMessageGesturesGmObj.GetComponent()<TextMesh>.text = textMessage;
}
}
GlobalCursor.cs file
...
this.transform.rotation = Quaternion.FromToRotation(Vector3.up, hitInfo.normal);
TextMessage.Instance.ChangeTextMessage_CursorOver(Constants.CURSOR_OVER);
}
else
{
meshRenderer.enabled = false;
TextMessage.Instance.ChangeTextMessage_CursorOver(Constants.CLEAR);
TextMessage.Instance.ChangeTextMessage_Gestures(Constants.CLEAR);
}
...
CubeGestures.cs file
public class CubeGestures : MonoBehaviour {
private bool holding = false;
void OnSelect()
{
TextMessage.Instance.ChangeTextMessage_Gestures(Constants.TAP);
}
void OnHoldStart()
{
holding = true;
TextMessage.Instance.ChangeTextMessage_Gestures(Constants.HOLD_START);
}
void OnHoldCompleted()
{
TextMessage.Instance.ChangeTextMessage_Gestures(Constants.HOLD_COMPLETED);
holding = false;
}
}
You can find this sample and download it from github to this link https://github.com/gntakakis/Hololens-Gaze-Gestures-Unity
GNtakakis's Blog 