Gaze and gestures in the HoloLens

Gaze is a primary form of targeting within mixed reality. With the gaze cursor the user is looking in the real world (if you use spatial mapping) and in the holograms and with the use of gestures they can interact with their environment. The primary way for a user to take action are via gestures, motion controllers and voice. The main gestures are air tap, double air tap, hold, drag and you can create your own gestures.

In this guide you can create a HoloLens application from scratch in order to learn to use gaze cursor and some gestures. For the development of the application we use the unity platform.

Create a new unity project and into the project new scene with the name “GazeGesture”. Initially, you have to make some adjustments in order to use and build successfully the HoloLens application. Choose Edit -> Project Settings -> Player. On the XR Settings tab, check the Virtual Reality Supported checkbox and add “Windows Virtual Reality” to list. To extract the project, go to File -> Build Settings …, there choose the Universal Window Platform tab and set Device to HoloLens and Build Type to D3D.


It is important to set in the Camera of the scene the projection to Perspective and to be in the right position as shown in the picture. Position (0, -0.02, 0), Rotation (0, 0, 0), Scale (1, 1, 1). Essentially, the point where you choose to display the camera is the point based on this which the other objects in space will be initialized.

CameraInspector
To create the cursor, you have to add a new GameObject with the name “GlobalCursor” with Scale (1, 0.5, 1) and as a child of this GameObject another GameObject with a name “GlobalCursorMesh” which must include the components Mesh and MeshRenderer. You can choose Mesh from Microsoft WindowsMixedReality Toolkit.
mixedrealitytoolkit.png

Add a new c# script called GlobalCursor.cs into to GameObject “GlobalCursor”. Next, initialize the MeshRenderer of the Cursor.
private MeshRenderer meshRenderer;

void Start ()
{
     meshRenderer = this.gameObject.GetComponentInChildren<MeshRenderer>();
}

In the update method, check if the user’s point is an object otherwise it does not display the MeshRenderer of the cursor in the HoloLens.


        var headPosition = Camera.main.transform.position;
        var gazeDirection = Camera.main.transform.forward;

        RaycastHit hitInfo;
        if (Physics.Raycast(headPosition, gazeDirection, out hitInfo))
        {
            FocusedObject = hitInfo.collider.gameObject;
        }   
        else
        {
            FocusedObject = null;
        }
        
        if (FocusedObject != oldFocusObject)
        {
            recognizer.CancelGestures();
            recognizer.StartCapturingGestures();
        }

Add a new GameObject cube with the name “Cube” which must contain a Box Collider component.

CameraInspector.png

Now you can export the project to Visual Studio and from there to run on HoloLens. See that if our look is in the box you created, the cursor is displayed. If you add spatial mapping component in the unity project you can see the cursor appear in all of the objects of our real world. Now, let’s add the gestures air tap, hold and cursor over and you ‘ll put the name of the event you use in the mixed reality cube.

First, add the names from the events to the class named “Constants.cs”

public class Constants {

    public static string CURSOR_OVER = "Cursor over";
    public static string TAP = "Tap";
    public static string DOUBLE_TAP = "Double Tap";
    public static string HOLD_START = "Hold start";
    public static string HOLD_COMPLETED = "Hold completed";
    public static string CLEAR = string.Empty;
}

Add the c# scripts GazeGestureManager.cs, TextMessage.cs to a new GameObject named “GazeGestureManager” and cubeGestures.cs on the GameObject “Cube”. In the GazeGestureManager.cs you have to initialiaze the events. When some of the events is active it show text message on GameObjects “TextMessageCursorOver” and “TextMessageGestures”.

GazeGestureManager.cs file

public class GazeGestureManager : MonoBehaviour {

    public static GazeGestureManager Instance { get; private set; }

    public GameObject FocusedObject { get; private set; }
    public GestureRecognizer recognizer;
    private bool updateFocusedObject = true;
    
    void Awake()
    {
        Instance = this;
        
        recognizer = new GestureRecognizer();
        recognizer.TappedEvent += (source, tapCount, ray) =>
        {
            if (FocusedObject != null)
            {
                updateFocusedObject = true;
                FocusedObject.SendMessageUpwards("OnSelect");
            }
        };
        recognizer.HoldStartedEvent += (source, ray) =>
        {
            if (FocusedObject != null)
            {
                updateFocusedObject = true;
                FocusedObject.SendMessageUpwards("OnHoldStart");
            }
        };
        recognizer.HoldCompletedEvent += (source, ray) =>
        {
            if (FocusedObject != null)
            {
                updateFocusedObject = true;
                FocusedObject.SendMessageUpwards("OnHoldCompleted");
            }
        };
        recognizer.HoldCanceledEvent += (source, ray) =>
        {
            if (FocusedObject != null)
            {
                updateFocusedObject = true;
                FocusedObject.SendMessageUpwards("OnHoldCompleted");
            }
        };
        recognizer.StartCapturingGestures();
    }

    void Update()
    {
        if (!updateFocusedObject)
        {
            return;
        }

        GameObject oldFocusObject = FocusedObject;
        
        var headPosition = Camera.main.transform.position;
        var gazeDirection = Camera.main.transform.forward;

        RaycastHit hitInfo;
        if (Physics.Raycast(headPosition, gazeDirection, out hitInfo))
        {
            FocusedObject = hitInfo.collider.gameObject;
        }   
        else
        {
            FocusedObject = null;
        }
        
        if (FocusedObject != oldFocusObject)
        {
            recognizer.CancelGestures();
            recognizer.StartCapturingGestures();
        }
    }
}

TextMessage.cs file

public class TextMessage : MonoBehaviour {

    public static TextMessage Instance { get; private set; }

    public GameObject textMessageCursorOverGmObj;
    public GameObject textMessageGesturesGmObj;

    void Awake()
    {
        Instance = this;
    }

    public void ChangeTextMessage_CursorOver(string textMessage)
    {
        textMessageCursorOverGmObj.GetComponent()<TextMesh>.text = textMessage; 
    }
    public void ChangeTextMessage_Gestures(string textMessage)
    {
        textMessageGesturesGmObj.GetComponent()<TextMesh>.text = textMessage;
    }
}

GlobalCursor.cs file

...
            this.transform.rotation = Quaternion.FromToRotation(Vector3.up, hitInfo.normal);

            TextMessage.Instance.ChangeTextMessage_CursorOver(Constants.CURSOR_OVER);
        }
        else
        {
            meshRenderer.enabled = false;

            TextMessage.Instance.ChangeTextMessage_CursorOver(Constants.CLEAR);
            TextMessage.Instance.ChangeTextMessage_Gestures(Constants.CLEAR);
        }
...

CubeGestures.cs file

public class CubeGestures : MonoBehaviour {

    private bool holding = false;

    void OnSelect()
    {
        TextMessage.Instance.ChangeTextMessage_Gestures(Constants.TAP);
    }

    void OnHoldStart()
    {
        holding = true;
        TextMessage.Instance.ChangeTextMessage_Gestures(Constants.HOLD_START);
    }

    void OnHoldCompleted()
    {
        TextMessage.Instance.ChangeTextMessage_Gestures(Constants.HOLD_COMPLETED);
        holding = false;
    }

}

You can find this sample and download it from github to this link https://github.com/gntakakis/Hololens-Gaze-Gestures-Unity

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s