Projected Head | Frankie Pang Studios

Abstract

Expression as light

Animatronic faces are expensive because every expression needs an actuator, a linkage and a skin that survives being stretched thousands of times a day. Projected faces avoid that cost by moving the expression into light, but they bring a new problem: the image only works while it stays registered to a surface that may itself be moving.

This project builds a complete, working pipeline for a projection-mapped head. A scanned head is printed and mounted on a three-axis servo neck. Its digital twin in Blender carries a rigged pair of eyes, a texture-space facial expression system and a library of interchangeable faces. A virtual camera matched to the physical projector renders the twin, and the projector paints that render back onto the print.

The page documents the system, the mathematics behind each stage, the human-factors research that shaped the animation, and an honest account of what failed. It ends with the next mechanical revision: replacing hobby servos with closed-loop steppers, because the backlash in the current neck is now the largest source of error between the projected image and the head it is meant to fit.

The physical head moves slowly and coarsely. The projected face moves quickly and finely. The illusion lives in how well the two agree.

01 · Context and preliminary research

Why project a face?

Projected faces are not new, but projected faces on heads that move freely have only reached parks and trade floors in the last two years. Before building anything, I looked at how the industry and the research community have approached this, from the earliest projected faces to the dynamic systems shown in 2025 and 2026, so this project could be positioned against the state of the art rather than reinvent it.

Fifty years of projected faces

Disney's Haunted Mansion (1969) projected filmed performances onto Madame Leota's head in a crystal ball and onto a row of singing busts. The principle has not changed since: a sculpted form supplies real depth and silhouette, and projection supplies a performance that would be impossible to engineer mechanically. The limitation was also fixed from the start. The heads could not move, because the film was locked to a static surface.

The Seven Dwarfs Mine Train (2014) brought projection onto figures that do move, blending mechanical animatronics with computer animation projected onto the Dwarfs' faces [2]. The same facial approach was carried into Frozen Ever After in 2016 [3]. In both, the head's movement is choreographed in advance, so the projected content can be authored to match a known motion.

Madame Leota or the singing busts, Haunted Mansion — **Plate R1** Haunted Mansion (1969): film projected onto static sculpted heads. Credit: [photographer / source].

A Dwarf's projected face, Seven Dwarfs Mine Train — **Plate R2** Seven Dwarfs Mine Train (2014): computer animation projected onto moving animatronic faces. Credit: [source].

Anna or Elsa's projected face, Frozen Ever After — **Plate R3** Frozen Ever After (2016): the same facial approach on a newer generation of figures. Credit: [source].

Third-party images: use your own photographs, or press images used with permission and credited in the caption.

The research foundation

Disney Research turned these techniques into a method. Bermano et al. (SIGGRAPH Asia 2013) split a facial performance into low-frequency motion that an animatronic head can physically reproduce and high-frequency detail added with projected shading [1]. That decomposition is the theoretical basis for this project: the neck carries the slow motion, and light carries everything faster.

Four years later, Makeup Lamps (Eurographics 2017) projected onto live human faces with no markers at all [8]. A high-speed camera under infrared illumination, running at 1,300 Hz, tracks head pose and expression; a customised Christie DLP projector runs at 480 Hz; adaptive Kalman filtering predicts where the face will be when the light arrives. The measured end-to-end latency was 9.8 ms. The lesson for any moving target is that latency, not resolution, decides whether projection stays on the face.

State of the art: two systems shown in 2025 and 2026

Walt Disney Imagineering · Pirates of the Caribbean, Disneyland

In June 2026, Disney debuted its first projection-mapped face on an Audio-Animatronic: the skeleton pirate on the treasure in scene 11, which now transforms between pirate and skeleton in a looping story about a cursed coin [9, 11]. The face is a 3D-printed shell with almost no mechanical articulation. Imagineers prototyped and kept only what they needed: the jaw stayed and the nose was cut [10].

Pose source. The figure's animation data is fed in real time into Unreal Engine, which renders the face [9, 10].
Tracking and calibration. Cameras keep the face aligned as the figure moves. Calibration markers hidden in the bandanna glow under UV light at night, when the system calibrates, and are invisible to daytime guests. A blue and white grid projected over the figure maps the projector to every contour [9, 10, 11].
Robustness. Two projectors, one running and one on standby. Disney states the face stays aligned even if a motor fails and the figure moves incorrectly [9, 11].
Intellectual property. A Disney patent application (US 18/592,863, filed March 2024, published September 2025) describes sensing the figure's orientation, rendering the next frames from live sensor input, and sending motion commands to keep body and image in sync [12].

P&P Projects · ThemedMotion × 7thSense

P&P Projects, the Dutch theming company, launched ThemedMotion as its animatronics division at IAAPA Expo Europe 2024 [16]. With 7thSense, its sister company CurtainUp and Scalable Displays, it now offers dynamic projection mapping onto moving animatronic figures as a turnkey product [13, 17]. I saw the demonstration, a projected face on a freely moving animatronic head, at IAAPA Expo Europe 2026 in London.

Pose source. Measured, not commanded. 7thSense's reactive projection mapping uses high-speed tracking cameras and motion-capture tracking markers to update a digital twin of the object up to 240 times a second [14, 15].
Rendering. 7thSense's Actor render engine, driven by its Compere workflow, draws each projector's pixels at the object's current position, independent of the media's own frame rate [14].
Calibration. Scalable Displays' software solves the relative positions of projectors and objects [15].
Markers. On the demo head, the markers were not visible to guests, which is consistent with infrared motion capture. 7thSense describes the tracking as camera-based with tracking markers but does not publish the marker type.

Makeup Lamps: projected makeup tracking a live face — **Plate R4** Makeup Lamps (Bermano et al., 2017): markerless infrared tracking at 1,300 Hz and projection at 480 Hz. Credit: Disney Research.

Disney's projection-mapped skeleton pirate, Pirates of the Caribbean — **Plate R5** Disneyland, June 2026: the first projection-mapped face on an Audio-Animatronic. Credit: [source].

Your photo of the P&P × ThemedMotion × 7thSense head at IAAPA Expo Europe 2026 — **Plate R6** Dynamic projection mapping on a moving animatronic head, IAAPA Expo Europe 2026, London. Photo: Frankie Pang.

Add videoOptional: your video of the IAAPA demo in motionassets/research-iaapa-themedmotion.mp4

Plate R7 The demo head moving freely while the projected face stays locked to it. Video: Frankie Pang.

Aspect	Disney · Pirates	P&P × ThemedMotion × 7thSense	This project (rev. 1)
Where the head pose comes from	Animation data in real time, corrected by camera tracking	Measured by motion-capture cameras, up to 240 Hz	Commanded pose from Bottango over USB serial
Markers	UV-fluorescent, hidden in costume; used for calibration at night	Tracking markers not visible to guests (IR inferred)	None; calibrated by eye with projected landmarks
Renderer	Unreal Engine	7thSense Actor media server	Blender Eevee
Projection	Two projectors, hot standby	Enterprise projectors, high frame rate	One consumer 1280 × 720 LCD projector
Calibration	Projected grid; daily recalibration with sensors	Scalable Displays auto-calibration	Landmarks projected from Blender and aligned by hand to the print
Face surface	3D-printed shell, articulated jaw	Sculpted animatronic head	3D-printed scan, no facial articulation
Cost class	In-park flagship	Turnkey enterprise product	Student proof of concept

What the research changed in my design

Use both commanded and measured pose. Disney combines animation data, which arrives instantly, with camera tracking, which catches what actually happened. My first revision only has the commanded pose. The encoders planned for revision 2 (Section 10) add the measured half at a fraction of the cost of a motion-capture rig.
Latency is the budget that matters. Makeup Lamps needed 9.8 ms and motion prediction; 7thSense updates at up to 240 Hz. For a head turning at angular speed ω with a total delay τ between pose and photons, the image trails the surface by:

(C1)

e_{lag} = r ω τ e.g. 0.15 m \times 0.52 rad/s \times 0.05 s \approx 3.9 mm \approx 10 px

A 30° per second head turn with 50 ms of combined serial, render and projector delay, at 0.15 m from the pivot and 0.39 mm per projector pixel. This is why the neck motion in the demo is slow and smooth, and why prediction from the known animation curve is on the roadmap: Bottango knows where the head will be next, so the twin can render slightly ahead.

Hide the machinery. Both professional systems keep their markers out of the guest's view, using UV markers that only appear at night or infrared markers that never appear. My current revision has no markers at all, because it is calibrated by eye. If a future revision adds camera tracking with fiducial markers, those markers will need to be hidden in the same way.
Calibrate every day and design for failure. Disney recalibrates daily and runs a standby projector. Operations are part of the design, not an afterthought.
Keep only the mechanism you need. Disney kept the jaw and cut the nose. My head has no jaw, and the static lips are the weakest part of the illusion (Section 06). A single jaw actuator is the most valuable mechanical addition after the neck.

The professional systems differ in where they get the pose. Disney trusts the animation and checks it with cameras; 7thSense measures everything. This project shows how far commanded pose alone can go, and where it stops.

Why it matters for attractions

Cost and reliability. A face with no moving skin has no skin to tear and no facial linkages to wear out.
Content agility. A new character, costume or makeup is a file, not a new sculpt, as Disney's transforming pirate shows.
Maintenance. The parts that can fail (projector, tracking and neck) are serviceable from behind the figure.

What it has to overcome

Registration and latency. Any movement the twin does not know about, or learns about too late, shows up as the face sliding off the head.
Brightness. Projection competes with ambient light, so the scene must be designed around it.
Static relief. Sculpted lips stay closed while projected lips open. The animation must respect what the form can sell.

02 · System

One head, two twins

There are two copies of the head: the print on the neck and the model in Blender. The system's whole job is to keep them identical in pose and to light the print with exactly what the model looks like at that instant. Motion is authored in Bottango, which drives the servos through an Arduino and streams the same pose to Blender over USB serial through a companion add-on, so the twin follows the physical head rather than the other way round.

Three Blender add-ons were written for the face. They are deliberately independent so each can be used, tested and removed on its own, but they share one head material and cooperate in a fixed order: overlays first, then the expression warp and mouth interior, with the eye rig on its own eyeball materials. Every control is an animatable property, so the whole performance is one Blender timeline that loops seamlessly.

Unwrapped 8K skin texture of the scanned head — **Fig. 3** The head's UV texture. Every effect on this page happens in this 2D space, so it follows the head wherever the twin is posed. Head scan and texture: 3D Scan Store.

Close-up of the texture showing blank eye sockets — **Fig. 4** The scan's eye sockets are blank in the texture because the eyes are separate geometry. The first experiment painted eyes into these discs; the final system rigs the real eyeballs.

03 · Building the head

From scan to projection surface

The digital head and the physical head start from the same file. The scan is printed, finished as a projection surface, mounted on the servo neck, and set up in front of the projector. Every later step depends on the print matching the model, so the build is documented step by step.

The finished head with the face projected onto it — **Plate B1** The physical head with the projected face running. Photo: Frankie Pang.

Step 1: printing the scanned head — **Plate B2** Step 1. Printing the head from the same scan used in Blender.

Step 2: finishing the print as a projection surface — **Plate B3** Step 2. Finishing the surface for projection.

Step 3: mounting the head on the servo neck — **Plate B4** Step 3. Mounting the head on the servo neck.

Step 4: Arduino, servo wiring and Bottango — **Plate B5** Step 4. Electronics: Arduino, servo wiring and Bottango.

Step 5: projector position relative to the head — **Plate B6** Step 5. Setting up the projector in front of the head.

Calibrating by landmarks

The projector is aligned to the head from inside Blender. Landmarks are placed on distinct features of the digital head, projected through the virtual projector camera onto the print, and the camera is adjusted until every landmark lands on its physical feature. The sequence below shows the process from first projection to final alignment.

Screenshot: landmarks placed on the digital head in Blender — **Plate C1** 1. Landmarks placed on distinct features of the digital head.

Photo: landmarks projected onto the print before alignment — **Plate C2** 2. First projection: the landmarks miss their features on the print.

Photo or screenshot: adjusting the virtual projector camera — **Plate C3** 3. Adjusting the virtual camera's position, rotation, focal length and lens shift.

Photo: every landmark on its feature — **Plate C4** 4. Aligned: each projected landmark sits on its physical counterpart.

Add videoVideo: the head moving through the stages of the showassets/build-stages.mp4

Plate B7 The physical head running the show: gaze, blinks, expressions and face changes while the neck moves. Video: Frankie Pang.

04 · Projection maths

Treating the projector as a camera

A projector is optically a camera run backwards: each pixel is a ray leaving a single centre of projection. If Blender's camera uses the same ray geometry as the real projector, every rendered pixel lands on the physical surface point that the virtual camera saw. That is the whole trick, and it reduces to the standard pinhole model.

(1)

s [\begin{matrix} u \\ v \\ 1 \end{matrix}] = K [R | t] [\begin{matrix} X \\ 1 \end{matrix}], K = [\begin{matrix} f_{x} & 0 & c_{x} \\ 0 & f_{y} & c_{y} \\ 0 & 0 & 1 \end{matrix}]

X is a point on the head in world space, (u, v) the projector pixel that lights it, K the projector's intrinsics and [R | t] its pose. The focal length in pixels follows from the throw: f = W · D / w, where W is the panel width in pixels, D the throw distance and w the width of the projected image at that distance.

Matching Blender's camera

Blender describes the same model in millimetres and normalised lens shift. Compact projectors throw their image upwards from the lens axis, so the principal point is not at the image centre, and that offset becomes a vertical shift on the virtual camera.

(2)

f_{mm} = \frac{f_{x} \cdot S_{mm}}{W}, {shift}_{x} = \frac{c_{x} - W / 2}{W}, {shift}_{y} = \frac{c_{y} - H / 2}{W}

S is the virtual sensor width with horizontal sensor fit. Blender normalises both shifts by the larger image dimension, which is W for a 16:9 projector. The sign of shift_y depends on whether pixel rows are counted from the top.

Finding the pose

The projector has no camera of its own, so the current revision is calibrated by placing landmarks from Blender. Distinct points on the digital head, such as the nose tip, the eye corners and the chin, are projected through the virtual projector camera onto the real print. The virtual camera's position, rotation, focal length and lens shift are then adjusted until every projected landmark sits on its physical counterpart. The human eye is the sensor: when the landmarks line up from the guest's viewpoint, the face does too.

Aligning by eye is a manual way of solving the same problem a computer-vision system would solve numerically. Each landmark is a 3D point X_i that should land on a known spot p_i, and good alignment means the pose that minimises the reprojection error below. A future revision could automate this with a camera and fiducial markers, such as ArUco tags, and OpenCV's solvePnP, which finds that pose directly and can repeat the calibration every morning.

(3)

R^{*}, t^{*} = \underset{R, t}{\arg \min} \sum_{i} {‖ p_{i} - π (K, R, t, X_{i}) ‖}^{2}

What an error costs

The same model shows how unforgiving registration is. A projected pixel's footprint on the head is roughly w / W. With the image about 0.5 m wide across the head, a 1280-pixel panel puts one pixel at roughly 0.39 mm. A human iris is about 11.7 mm across, so it is only about 30 projector pixels wide. A misalignment of a few millimetres is therefore a large fraction of an eye, and the guest reads it immediately as a face that has slipped. This sensitivity is what drives the next mechanical revision.

05 · Eye Rig add-on

Eyes that look, converge and blink

The scan's eyeballs are separate spheres, so the eyes are animated as real rotations rather than texture tricks. The add-on finds the face's own axes from the geometry instead of trusting the scene's world axes: the side axis runs between the eye centres, the forward axis comes from where the iris is painted on the eye's UVs, and the up axis is the perpendicular that points away from the neck.

(4)

\hat{s} = \frac{c_{R} - c_{L}}{‖ c_{R} - c_{L} ‖}, \hat{f} \propto d - (d \cdot \hat{s}) \hat{s}, \hat{u} = \pm \hat{f} \times \hat{s} with \hat{u} \cdot (m - h) > 0

d is the mean iris direction, m the midpoint between the eyes and h the centre of the head-and-neck mesh. An early version assumed world Z was up. When the head was modelled facing along Z, the painted lid closed outward from the pupil (Fig. 24). Deriving the frame from the anatomy removed the assumption.

Gaze and vergence

Each eye sits on a pivot whose rest orientation is the modelled pose, so zero on every slider means the eyes are exactly as sculpted. Pitch is applied before yaw, as in a gimbal, and convergence turns the two eyes towards the nose by equal and opposite amounts.

(5)

R_{eye} = R_{z} (ψ) R_{x} (θ), ψ_{L, R} = Θ_{H} g_{x} \mp c + τ_{L, R}, θ = Θ_{V} g_{y} + τ_{L, R}^{'}, c = \arctan \frac{IPD / 2}{D}

g_x, g_y ∈ [−1, 1] are the gaze controls, Θ_H and Θ_V the maximum turn and tilt, τ per-eye calibration trims, D the fixation distance. With a typical interpupillary distance of 63 mm, fixating 0.7 m away needs about 2.6° per eye; the first rig aimed both eyes at a target only 0.36 m away, which produced 5° of convergence and the cross-eyed look that prompted the rewrite.

Painted lids

The sculpted lids cannot close, so the blink is painted onto the eyeball in a lid frame that does not rotate with the eye. Coordinates are scaled by the eye radius, so the same expressions work on any eye size. Each lid edge is a parabola; a point is covered when it lies beyond either edge.

(6)

a = z - U + 0.35 x^{2}, b = - z - L + 0.25 x^{2}, lid = \max (S (a), S (b))

(7)

U = u_{0} + 0.25 λ g_{y} - (u_{0} + 0.25 λ g_{y} + L + 0.03) β, L = l_{0} (1 - 0.2 β), β = clamp (blink + wink, 0, 1)

S is a smoothstep over ±0.012 radii for an anti-aliased edge. u₀ and l₀ are the resting lid heights, λ is how strongly the upper lid follows vertical gaze, and the 0.03 overlap guarantees a full closure. Lash lines are a narrow band around each edge, and an optional painted contact shadow darkens the eyeball just under the upper lid.

Lighting: realistic or self-lit

Each eye material carries two shaders and blends them live. The lit shader receives real shadows from the brow and lids; the self-lit shader emits the texture's exact colour, which can read better in a dark show space. The blend is animatable, so a character can be lit naturally and then glow for a story beat.

(8)

surface = (1 - σ) {BSDF}_{lit} (c) + σ Emission (c, E)

Grid of nine renders testing rest, look right, look up-left, half blink, left wink, convergence, trim, target follow and sleepy lids — **Fig. 5** Control verification on the twin: rest, look right, look up-left, half blink, left wink, convergence 10°, right-eye trim 12°, follow-target and a lowered upper-lid rest. The rest pose matches the modelled pose to within 10⁻⁹ in the transform matrices.

Three renders: realistic lit eyes, self-lit eyes, self-lit with painted lid shadow — **Fig. 6** Realistic lit (top), self-lit (middle), and self-lit with the painted lid shadow (bottom), lit by a key light above the brow.

Lighting modes compared in Cycles and Eevee — **Fig. 7** The same four lighting states rendered in Cycles (left) and Eevee (right). Eevee is the engine that drives the projector, so parity matters.

06 · Face Expressions add-on

Moving a face that cannot move

The mouth is part of the print, so expressions are made by warping the texture in UV space. Landmarks were measured directly on the 8K texture (Figs. 8, 9): the lip line sits at v = 0.4724, the mouth centre at u = 0.5004, and each corner 0.0496 UV (about 397 texture pixels) from the centre. Brows sit at v = 0.6768, 0.0663 either side of the facial midline.

Mouth region of the texture with a measurement grid — **Fig. 8** Measuring the lip line and mouth corners on a 100-pixel grid over the 8K texture.

Brow region of the texture with a measurement grid — **Fig. 9** Brow centres and inner ends measured the same way. These become the add-on's calibration defaults and can be re-aligned with on-face guides for another head.

Inverse warping

A shader cannot push pixels; it can only choose where to read from. So each expression is expressed as a displacement field d, and the texture is sampled at q = p − d(p). The fields are sums of Gaussian radial basis functions centred on the anatomy: mouth corners, cheeks, brows and inner brows. Coordinates are normalised to the mouth's half-width h so the shapes are scale-free.

(9)

X = \frac{u - u_{m}}{h}, Y = \frac{v - v_{m}}{h}, d (p) = \sum_{k} a_{k} v_{k} e^{- {‖ p - c_{k} ‖}^{2} / r_{k}^{2}}, q = p - d (p)

For a smile, each corner Gaussian (r = 0.55) moves up by 0.27 and out by 0.18 mouth units, a cheek Gaussian rises by 0.10, and a fold-shading term deepens the nasolabial line along a line segment from nose wing to corner. The approximation q = p − d(p) is only valid while the map stays fold-free, which means the Jacobian of p − d(p) must keep a positive determinant. The global Intensity control exists to keep every expression inside that limit on a given head.

Opening a closed mouth

Jaw opening cannot be a smooth field, because the lips must separate. The texture is split along the lip line, the upper lip is lifted and the lower lip and chin are dropped, and the gap that opens between them is filled with a rendered mouth interior. A taper keeps the corners joined.

(10)

T (X) = \max {(0, 1 - X^{2})}^{0.7}, A_{u} = (0.08 J + 0.16 R) T, A_{l} = 0.55 J T

(11)

Y^{'} = {\begin{cases} Y - A_{u} e^{- {(Y - A_{u})}^{2} / {0.45}^{2}} & Y > 0 \\ Y + A_{l} e^{- {(Y + A_{l})}^{2} / {0.45}^{2}} + A_{j} & Y < 0 \end{cases}

(12)

G = S (Y + A_{l}) [1 - S (Y - A_{u})] S (A_{u} + A_{l}), {spec}^{'} = spec \cdot (1 - 0.95 G)

J is jaw opening and R upper-lip raise. Each lip edge is anchored so the sampled point is exactly the lip line at the edge. The jaw shift A_j carries the chin down but is zero at the lower lip and decays beyond the chin. G is the gap mask; its last factor removes a hairline seam when the mouth is closed. Teeth occupy a band just under the upper edge, and the specular term is suppressed inside the gap, because skin gloss over the mouth interior read as grey in the first pass (Fig. 12).

Eleven renders of the head with smile, laugh, surprise, sad, angry, oo, disgust, smirk, talking, clown overlay laughing and eye-shadow overlay surprised — **Fig. 10** Expression presets on the twin: smile, laugh, surprise, sad, angry, oo, disgust, smirk and talking, plus two expressions on top of face overlays. The clown paint stretches with the smile because overlays are sampled through the same warp.

First pass of expressions with calibration guides — **Fig. 11** First pass, with calibration guides visible on the neutral face (green rings at the mouth corners, lip centre and brows).

Close-up of open mouth with a grey interior — **Fig. 12** IssueThe open mouth read as grey: skin specular was reflecting the scene's lights over the interior.

Close-up of open mouth with a dark interior and teeth — **Fig. 13** FixedSpecular masked by the gap, a darker interior gradient and toned teeth with a soft lower edge.

Six frames from the generated expression loop — **Fig. 14** Frames from the generated loop in Fig. 1. The loop's first and last frames are identical in every control, measured at zero difference.

07 · Face Overlays add-on

One head, many faces

3D Scan Store publishes free overlay textures that share its heads' UV layout: makeup, tribal paint, clown faces, mud and grime. The add-on loads a folder of them into a playlist and crossfades between faces on a looping timeline, so the demo cycles through characters without reprinting anything.

(13)

C_{k} = B (1 - α_{k} s_{k}) + O_{k} α_{k} s_{k}, C = (1 - f) C_{A} + f C_{B}, t = \frac{n - n_{0}}{fps} \mod \sum_{i} (h_{i} + T_{fade})

B is the base skin, O_k and α_k the overlay's colour and alpha, s_k its strength, and f a smoothstep through each crossfade. Only two texture slots exist in the shader; a frame handler loads the outgoing and incoming faces, which keeps GPU memory flat however long the playlist grows.

The PSD problem

The overlays ship as layered PSD files containing albedo, roughness, gloss, specular, normal and height maps. Blender reads only a PSD's merged image, which has no transparency. The first build therefore showed each face correctly while it faded in, then turned the whole head one solid colour as the opaque image reached full strength (Fig. 17). The fix was a PSD and PSB reader written for the add-on in Python and NumPy. It parses the layer records, follows layer groups, applies layer masks, decodes raw, RLE and ZIP-with-prediction channels at 8, 16 or 32 bits, and extracts only the albedo layer. Downsampling uses alpha-weighted averaging so transparent edges do not bleed dark halos.

(14)

\hat{c} = \frac{\sum_{i \in Ω} α_{i} c_{i}}{\sum_{i \in Ω} α_{i}}, \hat{α} = \frac{1}{| Ω |} \sum_{i \in Ω} α_{i}

Ω is each k × k block of source pixels. The reader was cross-checked against the independent psd-tools library on 8-bit RLE, 16-bit ZIP-with-prediction and PSB test files, with zero difference in alpha and premultiplied colour. A 4K file decodes in about a second.

Eyelids that match the makeup

When a face changes, the painted blink lids on the eyes must change too, or eye shadow vanishes every time the character blinks. The add-on samples each face's upper-eyelid region in UV space, alpha-weighted and composited over the base skin, and drives the Eye Rig's lid colour through every crossfade.

Six renders: bare skin, clown, crossfade, eye shadow, tribal paint and tribal while blinking — **Fig. 15** Bare skin, clown, a crossfade, eye shadow, tribal paint, and tribal paint mid-blink with the lid colour matched to the face. These overlays are synthetic test images built to the 3D Scan Store layout.

Four Eevee frames across a crossfade — **Fig. 16** The same timeline in Eevee: fade in, hold, fade between faces, hold. Used to rule out the real-time renderer as the cause of the solid-colour bug.

Five renders: bare, clown from PSD, eye shadow from 16-bit PSD, solid colour failure, tribal from PSB — **Fig. 17** Faces loaded through the PSD reader. The fourth frame reproduces the original failure on purpose: an opaque image covering the whole head, now flagged with a warning in the playlist.

08 · Designing for life

What makes a face feel alive

Guests do not inspect a character's face; they read it, in a fraction of a second and mostly through the eyes. Realism in a texture matters far less than whether the face behaves the way faces behave. The animation in this project is built on a small number of observations about human faces, each turned into a rule the generators follow.

Observation	Rule in the system	Why a guest notices
Eyes move in fast jumps (saccades) between still fixations, not in smooth pans.	Gaze changes in 0.12 s and then holds; holds are random with a mean set by the designer.	Smoothly drifting eyes read as mechanical or drugged.
People blink around 15 times a minute at rest, fewer when concentrating [4].	Blinks every 3.5 s on average with variation; occasional double blinks; fast close, brief hold, slower open.	Regular, metronomic blinking is one of the quickest tells of a machine.
The upper lid follows vertical gaze.	Lid follow control couples lid height to look up and down.	Eyes looking down under a fixed lid look startled.
Two eyes converge on what they look at.	Vergence from fixation distance; per-eye trims for calibration.	Parallel or over-converged eyes look unfocused or cross-eyed (Fig. 23).
A felt smile also narrows the eyes, the Duchenne marker [5].	Smile and laugh presets raise the Eye Rig's lower lid.	A mouth-only smile reads as polite or false.
Faces are asymmetric and never quite still.	Independent left and right corners and brows; small random drift during holds; smirk and thinking presets.	Perfect symmetry and stillness push a face towards the uncanny [6].
Speech has a syllable rhythm of a few per second.	Talking passages flap the jaw on 0.12 to 0.24 s syllables with varied mouth width and occasional pauses.	Constant-rate flapping reads as a puppet.
Eyes reflect the light around them.	A catch-light in the iris; real reflections in lit mode.	Matte eyes look dead.

Interaction: from playback to presence

The current demo is a loop. The architecture was built so that it does not have to stay one. Because every behaviour is a property on a rig object, any input that can set a number can perform the face. The next stage is to give the head something to respond to.

Mutual gaze

A camera near the figure tracks guests' faces, and the nearest or most recently arrived guest becomes the eyes' target. Being looked at is the strongest social cue a character can give, and it costs nothing mechanically because the eyes are light.

Proximity and attention

Expressions respond to how close a guest is and how long they stay: a greeting on approach, attention while they remain, a glance away as they leave. In a queue, the figure can divide its attention across a group rather than staring at one person.

Contingency over fidelity

A face that responds at the right moment can be more convincing than a more detailed face that ignores you. Timing variability, reaction latency and pauses are therefore treated as design parameters, not noise.

Inclusive by default

My time supporting customers with accessibility needs at Apple shaped the requirements. Performances should not depend on audio alone, captions must be possible, and lighting changes should avoid rapid flashing. Faces carry grammar in sign languages such as BSL, which raises a research question: could a projected face give a signing character the facial expression that a mechanical one cannot?

Operations and safety

Calibration as a daily routine. Today the projector is aligned by hand with projected landmarks. A guest-facing version needs a repeatable routine, ideally automated with a camera and fiducial markers, so alignment can be re-checked at opening instead of relying on a one-time setup that drifts.
Content without hardware. New faces, makeups and performances ship as files. The physical figure stays the same.
Projector eye safety. The lens should sit above and away from guest sightlines so nobody can look straight into the beam at close range.
Graceful failure. If projection fails, the print is still a sculpted head rather than a broken mechanism. The live show-control version should be able to detect a lost video feed and dim or park the neck.

09 · Iterations

What failed, and what it taught

Every figure below was rendered during development on the digital twin. The failures are kept on purpose: each one changed the architecture.

Early painted-eye frames with dark rectangles below the eyes — **Fig. 18** IssueIteration 1 painted eyes into the texture. A shading term leaked outside the lid area and left dark rectangles under each eye.

Painted texture eyes looking around and blinking — **Fig. 19** FixedPainted eyes looking around and mid-blink. Convincing in 2D, but the head turned out to have separate eyeball geometry, which made this approach redundant.

Painted eyes fully closed — **Fig. 20** Full closure after correcting the lid maths so the upper edge meets the lower lid.

Fig. 21 Iteration 1 as a 10-second looping texture video.

First rigged eyeball renders, overexposed with square highlights — **Fig. 22** First rigged eyeballs on stand-in spheres. Rotation and shader blinks worked; the test lights were overexposed and reflected as squares.

Fig. 23 The first rig's animation. Both eyes aimed at a target 0.36 m away, which made them converge noticeably.

Eye rig working on a head that faces along the Z axis — **Fig. 24** FixedWith the face pointing along Z, the original lids grew outward from the pupil. Deriving axes from the anatomy fixed blinks, gaze and winks in any orientation.

Eyes showing correct iris colour in lit and self-lit modes on two mock materials — **Fig. 25** FixedA plain white eye appeared when the rig read the wrong texture from a six-map eye material. The rig now traces the colour map from the shader graph and scores images by content.

Six frames from the eye idle animation — **Fig. 26** Frames from the generated eye idle loop shown in Fig. 2.

10 · Next revision

A neck that holds still

With the face software working, the dominant error in the system is now mechanical. The neck uses hobby servos. They are cheap and easy to drive, but their gear trains have backlash: free play that lets the output move without the motor turning. When the head reverses direction, or when its weight shifts across an axis, it rocks through that play and keeps wobbling for a moment after it stops. The twin in Blender does not know this has happened, so the projected face slides across the print.

How the neck has already evolved

The wobble was visible from the first physical tests, and the neck has already been through one round of fixes.

Revision 1a: head on the servo. The head was connected directly to a servo, so the servo's output shaft carried the whole weight of the head as well as turning it. Any side load went straight into the servo's small internal bearing and gear train, and the head rocked on the shaft with visible play.

Revision 1b: adding a thrust bearing. A thrust bearing was added so the bearing carries the head's weight and the servo only has to supply rotation. Rotation on that axis became noticeably steadier.

What remained. The servo driving the other axis still had backlash in its gears, so the head still rocked whenever that axis reversed or the head's weight shifted, and the projected face still slipped. Supporting one axis better fixed a symptom, not the cause. The neck needs a new motion system in which every axis is both supported and driven without play.

Before: head connected directly to the servo — **Plate N1** Revision 1a. The head connected directly to the servo, which carried both load and rotation.

After: the thrust bearing added under the head — **Plate N2** Revision 1b. A thrust bearing takes the head's weight, so the servo only turns it.

Close-up of the thrust bearing assembly — **Plate N3** Detail of the thrust bearing assembly. Photo: Frankie Pang.

Add videoVideo: the remaining backlash on the other axisassets/neck-backlash.mp4

Plate N4 The remaining backlash: the second axis rocks on reversal and the projection slips. Video: Frankie Pang.

Error budget

A rotation error Δθ about an axis moves a point at radius r by an arc of r · Δθ. Dividing by the pixel footprint gives the misregistration in projector pixels.

(15)

e_{px} = \frac{r Δ θ}{w / W} e.g. \frac{0.15 m \times 0.01745 rad}{0.39 mm} \approx 6.7 px

Worked for one degree of play on the tilt axis, with the eyes about 0.15 m above the pivot and a 0.5 m wide image on a 1280-pixel panel. The figures are illustrative; the next step is to measure the real neck, for example with a camera tracking fiducial markers on the head.

Proposal: closed-loop steppers

Revision 2 replaces each servo with a NEMA 17 stepper motor fitted with an integrated encoder and closed-loop driver, coupled to the axis through a low-backlash reduction. Two transmissions are under consideration: a tensioned GT2 timing belt, which is simple, quiet and printable, and a capstan cable drive, which has effectively zero backlash and is well established in haptics and robotics.

(16)

Δ θ_{step} = \frac{1.8 °}{16 \times 5} = 0.0225 °, Δ θ_{enc} = \frac{360 °}{2^{14} \times 5} \approx 0.0044 °

1.8° full steps, 1/16 microstepping and a 5:1 reduction; a 14-bit magnetic encoder on the motor shaft. Commanded resolution falls to a fraction of a projector pixel, so the remaining error comes from structural stiffness, which is a design problem that can be solved.

Closing the loop on the image as well

Encoders change more than the motor control. Today the twin follows the commanded pose and assumes the head obeyed. With encoders, the controller can report the measured angle of every axis, and Blender can render from where the head actually is. Any error that does remain is corrected in the image instead of being shown to the guest.

Criterion	Revision 1: hobby servos	Revision 2: closed-loop steppers
Position feedback	Internal potentiometer, not readable by the show system	Encoder per axis, reported back to Blender
Backlash	Gear-train play, visible as wobble on reversals	Tensioned belt or capstan, near zero
Resolution at the output	Limited by pulse deadband and gear play	0.0225° step, 0.0044° encoder
Holding under load	Hunts around the setpoint; can buzz	Constant holding torque; corrects missed steps
Motion profile	Servo's own speed curve	S-curve acceleration, tuned to avoid exciting the structure
Homing	Absolute by design	Hall sensor or end-stop per axis at power-up
Maintenance	Gear wear increases play over time	No brushes or gear teeth to wear; belt tension and bearings are the service items
Cost and complexity	Lowest	Higher: drivers, power supply, homing

Mechanical design goals for revision 2

Intersecting axes. Pan, tilt and roll axes meet at one point inside the head, so a rotation never adds a translation the twin would also need to model.
Counterbalance. Balance the head about the tilt axis so the motors hold position rather than fighting gravity, which also removes the preload that hides backlash until a reversal.
Stiffness before speed. A stiff frame with a high natural frequency, and motion profiles that stay below it.
Measure, then trust. Add a camera and fiducial markers to record actual head pose against commanded pose for both revisions, so the improvement is a measured number rather than a claim.
Designed for the people who maintain it. Driver boards and belts reachable from the back of the figure, keyed connectors, and a documented homing and calibration routine for the start of each day.

11 · Reflection

Making the invisible visible

The project began as a texture on a print and became a lesson in where illusions actually break. None of the hard problems were where I expected. The maths of projection was straightforward; the failures came from assumptions about axes, about file formats, about which texture a material really uses, and finally about a few degrees of gear play. Each one was invisible until the system was built end to end and looked at the way a guest would.

That is the work I want to do in themed entertainment: connecting story, mechanism, software and operations, and carrying an idea through to a figure that is reliable, maintainable and convincing every day, not only on opening night.

Roadmap

Measure the residual error of the landmark calibration on the print, then trial automatic calibration with a camera and fiducial markers.
Build revision 2 of the neck and publish servo-versus-stepper measurements.
Camera tracking with hidden markers (infrared or UV), and render-ahead prediction from the known animation curve to cancel latency.
A jaw actuator, the one facial mechanism that both Disney and the static-lip results suggest is worth keeping.
Camera-based guest tracking driving the eyes' target.
Audio-driven lip sync, mapping speech to jaw and mouth shapes.
Two-projector coverage for the sides of the head, with edge blending.

References

Sources

A. Bermano, P. Brüschweiler, A. Grundhöfer, D. Iwai, B. Bickel and M. Gross, "Augmenting Physical Avatars Using Projector-Based Illumination," ACM Transactions on Graphics (SIGGRAPH Asia), 2013. Disney Research
"Disney takes a new approach with Audio Animatronics on its Seven Dwarfs Mine Train," Theme Park Insider, April 2014. themeparkinsider.com
"Frozen Ever After," Wikipedia, accessed October 2026. en.wikipedia.org
"The timing of spontaneous eye blinks in text reading suggests cognitive role," Scientific Reports, 2025. nature.com
P. Ekman, R. J. Davidson and W. V. Friesen, "The Duchenne smile: Emotional expression and brain physiology II," Journal of Personality and Social Psychology, 58(2), 1990.
M. Mori, "The Uncanny Valley," Energy, 7(4), 1970; English translation by K. F. MacDorman and N. Kageki, IEEE Robotics & Automation Magazine, 2012.
3D Scan Store, "Overlay Maps." 3dscanstore.com
A. H. Bermano, M. Billeter, D. Iwai and A. Grundhöfer, "Makeup Lamps: Live Augmentation of Human Faces via Projection," Computer Graphics Forum (Eurographics), 2017. Disney Research (PDF)
"Disney Debuts First Projection-Mapped Face On An Audio-Animatronic," BlogMickey, June 2026. blogmickey.com
"I watched Disney's next-gen audio-animatronic transform from a pirate to a skeleton," TechRadar, 2026. techradar.com
"Disney Debuts Its First Projection-Mapped Animatronic Face," DVC Shop, 2026. dvcshop.com
"Disney Files NEW Patent That Could Be a Game-Changer for Theme Parks" (US application 18/592,863), AllEars.Net, September 2025. allears.net
P&P Projects, "Dynamic Projection Mapping," July 2026. ppprojects.com
"Reactive Projection Mapping on Moving 3D Objects" (7thSense), Blooloop Innovation Awards, 2022. blooloop.com
"Scalable and 7thSense Unveil Multi-Object Reactive Projection Mapping at ISE 2025," rAVe, 2025. ravepubs.com
"P&P Projects launches ThemedMotion, specialising in animatronics & animated figures," Blooloop, September 2024. blooloop.com
"7thSense launches sister company, CurtainUp," Blooloop, September 2025. blooloop.com