Project website documentation : https://www.homesmartmesh.com/docs/applications/home3d/

If you would like to get support, give feedback or discuss new ideas related to this project ?

It's possible to directly discuss on the discourse forum

discourse forum: https://homesmartmesh.discourse.group/

Vision

Many open source home automation frameworks start to get popular, but few care about 3d interactions while it has potential to be very user friendly:

Community shared .glTF interactive items can make such framework easily usable for everyone.

IoT device suppliers could provide an mqtt interface API and a glTF model with animations.

hue lights and mqtt devices in sync with the 3d Model

Github repohttps://github.com/HomeSmartMesh/smart_home_3d_webapp

First time usage

  1. host this project on your local raspberry pi or see 'Local host vs Remote host' section on how to use the live demo
  2. create your own glTF model from blender (optional)

activate hue bridge connection

  1. adjust the light names to match your hue lights names
  2. press the Hue Gateway authorisation button (on the real device)
  3. Click on the 3d model of the hue gateway
  4. An alert will apear on the screen to wait for confirmation
  5. The user creation will proceed and the username will be stored as local storage (ctrl+j in chrome to oben the debug window)

activate mqtt connection

  1. adjust the mqtt topic names to match your mqtt devices
  2. install a mosquitto version with websocket support
  3. Click on the 3d model of the mosquitto gateway

Switching on a real light with power logging

We see in this gif demo above a home 3d model augmented with interactive mesh items. Some meshes represent light bulbs and generate events when the user clicks on them. Those events are handled by a hue light client that interacts with the real hue gateway. The hue client publishes as events the actual state of the switched light, and that state is updated by the mesh bulb color and the associated 3d light.

As a demonstration for the reaction time of the real light switching on and off, we can see in the gif animation the power consumption log of the light switched. This log comes from a shelly 2.5 device with power monitoring capabilities. This measure device is itself powering up the hue light. Note that the slow power up and down ramp are due to the hue effect of slow variation when switching on and off.

Hue lights and lightgroups usage from the 3d webapp

see the full details on the project log entryhue-lights-and-lightgroups-usage-from-the-3d-webapp

Concept

Boiler plates examples linked below

Examples and explanation from the github readmegithub.com/web_three_interface/
All boilerplate examples embedded in a single pagegithub.io/web_three_interface/

3d objects properties

Manipulation of properties such as mesh animation and color. This control can operate from an external interface of through internal 3d mouse events.

call to modify a property:

send_custom_event("three_color",{name:"Kitchen", val:0.3});

3d mouse events

Events such as hover, click and long press. These events uses rays to match the 3d objects meshes.

user handling of 3d mouse events:

window.addEventListener( 'mesh_touch_start', onMeshTouch, false );

function onMeshMouseDown(e){    console.log(`Mesh Touch on ${e.detail.name}`);
}

3d slider control

On activation of the control, new objects such as a slider pop up in the scene and takes over the mouse control. While the user is acting on the control, the control is producing events. The whole cycle can be as short as the user touching the 3d element, the slider popup with the bullet already under the user's finger, the user shifts slightly to the new desired position and on touch release the slider goes away.

window.addEventListener( 'mesh_control', onMeshControl, false );

function onMeshControl(e){    console.log(`update of ${e.detail.name} value to ${e.detail.val}`);
}

Note on related work : 3d interactions are not new new this project, there are some examples such as https://doc.babylonjs.com/examples/, but still the interaction is different as in babylon js the sliders for example are still 2D overlays.

multiple parameters - a full light service

Link to live demoLive demo
Link to boiler plate example12 multiple parameters - github directory

In this demo, the user is interacting with the light bulb in two ways, clicks for quick actions full switch on or off for all parameters, or with the hold (>600 ms) to smoothly adjust the value with the 3d slider.

Usage

Mouse event from web window to three.js mesh

The user moves the mouse over the sphere object which changes color on enter and on exit events.

Link to live demoLive demo
Link to boiler plate example01 mouse hover on mesh - github directory

Design

This mouse hover example is easily reusable as it is split in modules. The "three_mouse.js" contains the conversion logic from the web mouse events to the 3D mouse mesh events. It is isolated from the main application logic in "main.js", and both isolated from the "three_app.js" which contains clasical three environment and shapes creation.

Mouse events on glTF models imported from Blender

Link to live demogithub pages web_three_interface 03_mouse_on_glTF_imports
Link to boiler plate example03 mouse on glTF imports - github directory

Workflow

Once familiar with THREE.js concepts, and there is the need to render a more complex scene, it becomes quickly complex to configure every parameter in the code. Looping between editing parameters and display require a reload. That's where professional 3d editing tools coming handy. Thanks to the glTF (OpenGL Transmission Format), exporting a scene from Blender to THREE.js becomes easy. Not only 3d objects with beshes are exported but also cameras and lights. Note that export include options on blender are required.

Interactive Full Home Model

Note
A complex detailed home model is not required, the most simplistic multiple squares model is enough to have 3d interactions. This sections shows a method that can produce a very accurate result. Other options are of course possible, the easiest is to model the home geometry directly in blender.

Link to live demoLive demo
Link to boiler plate example01 mouse hover on mesh - github directory

Parametric modelling

For technical and architectural designs, parametric modelling can come in quite handy. Fusion360 is a reference tool when it comes to that. The usage of geometrical sketches reduces the complexity of rooms modelling by simply entering the dimensions.

Workflow

The workflow is extended as shown below.

File Format

".fbx" is the only common format between both tools that keeps the scale and the hirerchy of objects. The ".stl" and ".obj" pack the whole model in a single mesh.

Mesh to light association and light groups

Link to live demoLive demo
Link to boilerplate example06 light groups - github directory

Mesh to light with Parenting

A new concept different from the previous example 05, is to simplify the mesh used to activate the light, and the light obejct connected without custom properties, rather with the light being a child object of the mesh within the blender model and as imported in the three.js scene.

As the mouse interaction is not specific to lights but could be applied to any sort of mesh, it is now explicitely set as custom property "mouseEvent"

Light grouping

Multiple lights having a similar blender structure as in the previous paragraph, can be gruoped under a common parent (in this figure above 5 of them). The parent should have as custom property "lightgroup" and also "mouseEvent" in order to receive a mouse interaction. The three_app provide functions to manage the light group by forwarding Set functions to the children lights.

Note the logic of group light click is defined to switch off all lights if any of the children is on.

Associating a parameter to an animation

Link to live demoLive demo
Link to boiler plate example08_parameter_to_animation


We see above the user moving the gui.dat parameter which is rotating the object. Although rotating an object in three.js is quite simple, this demo shows how to do this through an animation. That gives the advantage of abstracting the animation design from the parameter that will be controlling the animation.

blender side

Due to a three.js limitation (apparently), it was not possible to identify which object has which animation although it is available in the glTF and known by the internal three.js objects somehow. This was overcome by using a custom parameter on the object that mention the animation name:

animation hint

For controlled animation type, the animation is expected to be linear otherwise any distorsion will impact the parameter mapping as well :

design param to animation

Animating rotation scale and changing views

Link to live demoLive demo
Link to boiler plate example09 Rotation Scale View parameters - github directory

In this demo above we see the user acting on the dat.gui elements to modify the rotation of the first item through the "pull" parameter, the scale of a rectangle within the second item through the "heat parameter" and updating the view of the third item through the drop down "view" menu

Animating colors

We will be talking about "parameters" in stead of animations as it is unfortunately not possible to map all required parameters modifications to animations. One of the main goals of this project is to animate color and that is simply not yet supported by the .glTF standard. Therefore, in a first step, a multiple views concept is introduced where each view can be selected by its name, and each view is a completely different child mesh that can have a different geometry and material. Note that modifying geometry only is not required here as it could be done through animation, but there's no other way to assign more than one material to one mesh.

references for glTF limitations : No support for material animations

blender structure for the state as parameter

Usage

To select one view, it is enough to send a javascript custom event with the name of the object to be updated and as value, the child state to be assigned :

send_custom_event("three_param",{name:"Cameleon", val:"plate1"})

Material color animation

Although not supported in glTF export, a material color animation is developed in this sample to fulfill this function in a very simple way.

Link to live demoLive demo
Link to boiler plate example10 More States and material color animation - github directory

In this demo, the previous example is extended with another state. Any number of states can actually be set. A method for color variation is also acheived. This change only concerns the material RGB color, not the speculiar or others. Here a parameter is interpolating two colors, but it is possible even to have any number of colors, where each will have an interpolation weight.

Blender structure of the color as parameter

Usage

send_custom_event("three_param",{name:"Cameleon", appear1:0.3, appear2:0.7})

in order to have a coherent result, the sum of the given parameters is expected to be equal to one.

3d slider

Link to live demo Live demo
Link to boiler plate example11 Controls 3d slider - github directory

In this demo, we see the user seamlessly using both a classical slider from the dat.gui and the new 3d slider control.