Leap Motion + Ultrahaptics tech demo – Elements (UE4, Blueprint)

• All gameplay programming and project setup
• Prototyping and implementation of user interaction using Leap Motion
• Setting up pipeline of data consumption and output (data table parsing, visualising it in UE4, making it interactive)
• Shaders and all other visual aspects

Further breakdown

At its core Elements is an interactive data visualisation using Leap Motion and Ultrahaptics. Initial idea come from this https://research. google.com/bigpicture/elements/ The demo showcase data of chemical elements in technology, in form of a periodic table that user can interact with using Leap Motion as well as experiencing mid air haptics in the process.

I worked on all gameplay programming/setup (Leap Motion interaction, cubes reaction to player, data table parsing and setup, etc), as well as all visual elements. Adam Johnston (a software engineer) handle the Ultrahaptics plumbing to Unreal part, as this involved more lower level programming.

Data source

The demo start with data coming from Arm Research, containing year, elements related to it and many other complex number/data. This will be called DT_Semicondutor_Materials_Timeline.

We then bring this to Unreal and created data table and struct, which will be the back bone of this demo, almost all visualisation will be driven by the number and data in it. We use second data table containing information of the chemical elements that we can use to construct the periodic table at runtime inside Unreal.

This is how the periodic table look, containing all information we need including the row and column number that can be used to construct our table, this table is called DT_PeriodicDataTable.

Interaction Prototype

For this demo we speculate that we will need some sort of UI to display something down the line. This UI need to be as less intrusive as possible due to the screen real estate is limited, so this will need to be prototyped. On this attempt I use ‘grab’ gesture to hide and unhide the ‘UI elements’ which are interactable button that player can summon only when they need them.

Box slotting mechanic, we can’t polish this feature to be good enough and decided not to go ahead with this. My idea on this one was to make the demo even more interactive, so based on the ‘year’ selected element boxes will appear which user can grab and throw to the periodic table. Animation done by interpolating vector/float position value using finterpto function in Unreal.

Periodic table construction in Unreal
The table constructed from multiple element boxes (containing information from periodic table data table) that’s spawned at runtime and positioned based on the data table. To create the periodic table we need 2 blueprint classes BP_ElementBox (the box that get spawned) and BP_ElementController is blueprint that manage spawning of these boxes.

This is BP_ElementController, it have the Spawn Periodic Table event that’s triggered and do a for loop of DT_PeriodicDataTable column and create BP_ElementBox in the process as well as passing the information from data table to these boxes.

This is the single BP_ElementBox class composed of meshes, box collision and UMG (Unreal Motion Graphics) widget for displaying text. When created by the ElementController these boxes will also contain various variable (element name, symbol, position in the overall data table) extracted from the data table. This class also contain code that manage behaviour of these boxes when touched, highlighted, etc.

Periodic table behaviour
Most gameplay code of the periodic table is in this blueprint. After the table is constructed by BP_ElementController, it will be taken over by BP_MaterialController. This blueprint do various task

• Handle the element box highlighting based on year
• Passing on information from DT_Semicondutor_Materials_Timeline to various UI elements at the top of the screen.
• It was also handle creation of the throwable element box (not used in final demo)

Hand material, with controllable emissive that then hooked to glow whenever user approach interactable objects.