Updating SCNMaterial texture in SceneKit

With the recent release of iOS12 and ARKit 2, we at Fueled decided to build a multidevice AR game called Toppler to explore some of its capabilities. While building this AR experience, we needed to give feedback to the user as to which block was going to be selected and grabbed. We ultimately decided that a run-time update to the the material texture was the way to go. However, in doing so we faced an issue that others might also encounter when building an SceneKit game.

Swapping the textures

We had two different images that we wanted to apply as textures, depending on the block state:

I’ll skip past the UV maps, which are responsible for unfolding the texture on the block, but if you want more detail here I’ve previously written about how to export a 3D model from Blender to SceneKit.

I decided to handle this by providing conditional sources depending on the material state. Below is a simplified version of that TopplerBlockMaterial class, a SCNMaterial subclass responsible for handling the visual state of the blocks:

final class TopplerBlockMaterial: SCNMaterial {

    private static let baseImage: UIImage = UIImage(named: "base.png")
    private static let highlightedImage: UIImage = UIImage(named: "highlighted.png")


    override init() {
        super.init()
        diffuse.contents = TopplerBlock.baseImage
    }

    var highlighted: Bool = false {
        didSet {
            // Prevent unneeded updated
            guard highlighted != oldValue else { return }
            diffuse.contents = highlighted ? TopplerBlockMaterial.highlightedImage : TopplerBlockMaterial.baseImage
        }
    }
}

Using this technique we noticed dropped frames every time a block needs to be highlighted and/or unhighlighted, sometimes even dropping well below 30fps.

Using Instruments and the appropriate profiles, we were able to identify the root cause of that poor performance:

Turns out that underlying processes were spending nearly 22% of the allowed time to render the frame and create the image material. CoreGraphics needs to redraw the UIImage in a new buffer before applying it to a material, which proved to be very time consuming.

We definitely had to look somewhere else to update our texture without impacting the game and AR experience, which required a constant frame rate.

Offsetting the textures

After spending a couple hours looking for a solution in the documentation, I finally discovered an interesting instance property on SCNMaterialProperty : contentsTransform.

var contentsTransform: SCNMatrix4 { get set }

SceneKit applies this transformation to the texture coordinates provided by the geometry object the material is attached to, then uses the resulting coordinates to map the material property’s contents across the surface of the material. (This transformation has no effect if the material property’s contents object is a constant color.)

For example, you can use this property to grow, offset, or rotate a texture relative to the surface of a material, as illustrated below.

Turns out that’s exactly what we needed. This would allow us to create a single image file holding the two different states, and then we could offset the contentsTransform to only display the needed portion of the texture.

Here is the new texture image, and the red rectangle illustrates the transform we can use as the contentsTransform:

So we’ll need to use two different transforms combinations, one for the normal, and one for the highlighted state. Here is our updated TopplerBlockMaterialclass that uses this approach:

final class TopplerBlockMaterial: SCNMaterial {

    private static let textureImage: UIImage = UIImage(named: "texture.png")

    override init() {
        super.init()
        diffuse.contents = TopplerBlock.textureImage
        diffuse.contentsTransform = SCNMatrix4MakeScale(0.5, 1.0, 0.0)
    }

    var highlighted: Bool = false {
        didSet {
            // Prevent unneeded updates
            guard highlighted != oldValue else { return }
            diffuse.contents = highlighted ? TopplerBlockMaterial.highlightedImage : TopplerBlockMaterial.baseImage
                let translateOffset: Float = highlighted ? 0.5 : 0.0
            let scale = SCNMatrix4MakeScale(0.5, 1.0, 0.0)
            let translateAndScale = SCNMatrix4Translate(scale, translateOffset, 0.0, 0.0)
            diffuse.contentsTransform = translateAndScale
        }
    }
}

Using our new material switching technique in Toppler, we have rock solid performances and a smooth AR experience.

That kind of blocker that we faced is part of a long list that one usually don’t expect to hit when coming from UIKit. Swapping the UIImage of a UIImageView is a very straightforward process, and should not introduce performance issues. But when working with a 3D environment and SceneKit, we faced some unexpected performance considerations we don’t normally encounter when working with UIKit. Hopefully this article helps others as well.

Written by Benoît Layer Principal Engineer, Mobile