Casper DPM: Cascaded Perceptual Dynamic Projection Mapping onto Hands

SIGGRAPH Asia 2024 Conference Track & Emerging Technologies Demo

Yotam Erel1, Or Kozlovsky-Mordenfeld1, Daisuke Iwai2, Kosuke Sato2, Amit H. Bermano1
1Tel Aviv University 2Osaka University
Paper Supplementary Code arXiv

Short video showing our work.

Abstract

We present a technique for dynamically projecting 3D content onto human hands with short perceived motion-to-photon latency. Computing the pose and shape of human hands accurately and quickly is a challenging task due to their articulated and deformable nature. We combine a slower 3D coarse estimation of the hand pose with high speed 2D correction steps which improve the alignment of the projection to the hands, increase the projected surface area, and reduce perceived latency. Since our approach leverages a full 3D reconstruction of the hands, any arbitrary texture or reasonably performant effect can be applied, which was not possible before. We conducted two user studies to assess the benefits of using our method. The results show subjects are less sensitive to latency artifacts and perform faster and with more ease a given associated task over the naïve approach of directly projecting rendered frames from the 3D pose estimation. We demonstrate several novel use cases and applications.

Projegraphy

Inspired by Shadowgraphy, the art of performing a story or show using images made by hand shadows, we developed a new concept art form for projecting live content onto hands depending on their shape and form. We name this Projegraphy. Here, the hand or hands form some intricate shape, and the system generates suitable adaptive content for that shape. For example, the hand imitates some features of an animal, and the system projects relevant content without any additional user input. This can be used both for performances, and for social gaming ("guessing the animal").


projegraphy

Gaming

We modified an open sourced space shooter game for the purpose of showing a simple gaming application with the system. Players can move the space ship to the left or right by tilting their palm around the elbow-to-wrist axis, and shoot alien ships in front of the space ship. The score is displayed on the thumb, while the HP is visualized as bars displayed on the fingers, which decrease every time the spaceship is hit.


spaceshooter

Dynamic Effects

We can augment the hand with any arbitrary 3D effect as the full hand is reconstructed.


BibTeX

@article{erel2024casperdpmcascadedperceptual,
        title={Casper DPM: Cascaded Perceptual Dynamic Projection Mapping onto Hands}, 
        author={Yotam Erel and Or Kozlovsky-Mordenfeld and Daisuke Iwai and Kosuke Sato and Amit H. Bermano},
        year={2024},
        eprint={2409.04397},
        archivePrefix={arXiv},
        primaryClass={cs.GR},
        url={https://arxiv.org/abs/2409.04397}, 
  }