Tips for Reproducing Real Space in Photogrammetry

This is the first series of content creation tips for creators using Spatial Reality Display (referred to as SRD in the following text).
This is an article by Ryu of HoroLab Inc., who creates photogrammetry content.
This article shows you the benefits of watching photogrammetry in SRD and tips for creating content.

What is Photogrammetry?
Photogrammetry is a technique in which an object is photographed from various angles and a 3D model is generated based on the photograph. It is used in various applications such as surveying, architecture, and digital archives.

Benefits of viewing photogrammetry in SRD

■ Provide a representation that takes advantage of both physical models and 3DCG

Provide a representation that takes advantage of both physical models and 3DCG Even now that 3DCG has become popular, the demand for physical models has not disappeared and they are still being used.This is probably due to the large advantage of being able to easily grasp three-dimensional shapes by looking around the object in front of you from a free angle. On the other hand, 3DCG has the great advantage of being able to represent physical objects in a way that is impossible.

SRD can achieve both of these advantages at the same time.

However, it should be noted that if the content is expressed too freely with the benefit of 3DCG in mind, the benefit of showing it in a model way will be damaged. It is also necessary to set limits.

Tips for displaying content attractively

■ Improve a sense of reality and presence.

・Position the model so that it extends to the front of the display

By placing the model's geometry in front of the display surface (shown in blue below), the model feels as if it's in physical space, as if it's sticking out of the display. However, if it's too far out of range, the focus will be hard and crosstalk will be noticeable, so try to keep it within a few centimeters (shown in yellow below).

Here's an example of the hood of a car protruding too far in front of the display:

・Match the ground in 3DCG space with the bottom of the display

Matching the position of the bottom of the SRD physical monitor with the position of the ground in the 3DCG space creates spatial continuity between the real space and the 3DCG space, increasing the sense of reality.

A : 3DCG ground position
B : Housing frame top position

If you are preparing the floor in CG, it is more effective to have an Unlit shader that casts shadows to make the floor feel more installed and real.

■ Include a unique way of showing 3DCG that is impossible with physical models

One of the impossible expressions in a physical model is to show a cross-section. "If the cross-section position is fixed, it can be produced in a model, but if you cut a cross-section at a free position, it can be represented only because it is a 3DCG"
"A better experience can be obtained if the design allows the cross-section position to change according to the operator's operation."

・Section mode

・Normal mode

・Section mode

・Normal mode

・Section mode

・Normal mode

In the above content, we have implemented the switch between "Horizontal section," "Vertical section" and "No section" each time the Enter key is pressed.
In addition, the position of the section can be changed by using the arrow keys, so that you can "see where you want to see."

■ Avoid expressions in which the entire model moves or changes its appearance by itself

Unlike viewing a video work, viewing it from a free position of your own will is a big advantage of making it look like a model, so don't let the model move around on its own and you can't see it as you want.
If you want to move the model or create some effect, you can experience it stress-free by adapting it to the interaction from the controller or keyboard.
"If you want the model to animate automatically, you can keep the movement constant."
"A model that rotates automatically at a constant speed, like an object on a turntable, will be easier to view without degrading the experience."

Tips for displaying more clearly

Photogrammetry is three-dimensional and fun, but "depending on the results of compositing and shooting," the mesh can still run wild and look like noise. The trick to improving the quality of your content is to make it less obvious.

■ Use Unlit type materials

It depends on how clean the model of photogrammetry you use is, but if you want to display it easily and neatly, you can use Unlit type material, which is not affected by light, so that the mesh doesn't show off, and the color tone is just as it was when you took the photo, so you can experience it as if the real thing is there even when you see it on SRD.

※What is an Unlit shader: A lit shader is affected by lighting, but an Unlit shader is not affected by lighting. It has no roughness or metallic values and expresses only the base color's color tone.

・Differences in the appearance of mesh irregularities

・When using Unlit shader

When set to Unlit, unnecessary unevenness becomes less noticeable.

・When using standard shader

The unevenness of the polygons when converted to low poly is noticeable.

・Difference in color

・When using Unlit shader

Unlit allows the color of the photo to be used.

・When using standard shader

In order to achieve a faithful reproduction with a different color from the actual site, it is necessary to work out the lighting in each room, which takes time to adjust, and the reproduction is limited.

・Drawing load can be reduced.

Unlit eliminates the calculation of lighting effects and reduces drawing load.

■ Optimize the number of polygons and textures

Because of the nature of photogrammetry, geometry and textures are generated to faithfully reproduce the colors of all coordinates, so the number of polygons increases when the setting is to reproduce even the smallest shape, and the size and number of textures increase when the setting is to emphasize texture sharpness.
Therefore, focusing on quality tends to increase data capacity and drawing load. However, when displaying like a model, the model itself will be displayed smaller than the actual size in VR, so reducing the number of polygons and textures instead of keeping the original data will not significantly reduce the quality of the experience. In fact, the disadvantage of reducing the quality of the experience by lowering the fps due to focusing on the quality is greater, so be careful not to reduce the number of polygons and textures depending on the size of the model you want to display, and avoid excessive quality.

Tips for making the expression more present -Fusion of real space and virtual space-

■ Scan the exhibition space itself and set it as the background

A special method, but one that provides more continuity with real space, is to photogrammetry the space itself where the SRD is displayed and set the photogrammetric model as the background of the CG space. This is a representation method that can be called a pseudo AR.
You won't be able to see the enclosure that's supposed to be on the back of the display, so it's also a good way to reduce the presence of the enclosure.

HoloLab Inc. accepts requests for the production of photographic content.
From shooting to application production, we are a one-stop shop.

Spatial Information Technology Department SIAR　　https://hololab.co.jp/siar/

If you are interested, please contact us at the following address.

HoloLab Inc.　　　　https://hololab.co.jp/#contact
Creator:⿓ lilea　　　https://twitter.com/lilealab

・Related Article

Design guide：https://www.sony.net/Products/Developer-Spatial-Reality-display/en/develop/Design/Scene.html