Contact4D: A Video Dataset for Whole-Body Human Motion and Finger Contact in Dexterous Operations

Building human-centric AI that understands how people use their hands requires data on how humans physically interact with objects, yet large-scale datasets for this remain scarce. Most existing ones are recorded entirely in controlled lab settings, and their contact labels are typically estimated by thresholding hand–object distances rather than measured directly. To address this, we present Contact4D, a multi-view video dataset of people assembling furniture, captured with a flexible multi-camera system and annotated with human detections, tracks, 2D/3D body and hand poses, and ground-truth finger contact. Because hands are often heavily occluded during assembly, we also introduce a processing pipeline that recovers accurate hand poses under severe occlusion. The dataset contains 2M images from 19 synchronized cameras over 350 video sequences, covering a variety of environments, furniture types, and subjects. Benchmarking current human pose estimation and contact estimation methods shows that they generalize poorly to these scenarios. Finally, fine-tuning a pretrained MultiHMR model on Contact4D improves body MPJPE by 56.6% and hand MPJPE by 26.4% under severe self- and object occlusion.

The Contact4D dataset is hosted on Hugging Face. Data is organized by capture session (session_01, session_02, …), and each session contains its recorded sequences (001_session_01, …). Inside a sequence, exo/ holds the exocentric camera images (cam01–cam18), ego/ holds the Aria RGB and SLAM streams, and processed_data/ holds all annotations:

• body_pose3d/, hand_pose3d/ — 3D body (17 joints) and hand (21 joints per hand) keypoints in world coordinates, in meters.
• body_poses2d/, hand_poses2d_corrective/ — per-camera 2D keypoints.
• smpl/, smplx/, mano/ — fitted body and hand model parameters and meshes.
• finger_contact/annotations.json — per-frame contact state for all ten fingertips.
• camera_params/ — intrinsics and extrinsics for every exo camera and Aria stream.

The Contact4D Toolkit is a lightweight Python package (numpy, OpenCV, PyTorch, smplx) for working with the release. It projects 3D keypoints into any view, converts SMPL/SMPL-X/MANO to camera space, renders meshes on images, undistorts exo and Aria images, and visualizes fingertip contact. For example:

git clone https://github.com/jyuntins/Contact4D_Toolkit
cd Contact4D_Toolkit && pip install -r requirements.txt

python scripts/visualize_finger_contact.py --sequence-path /path/to/001_session_01 \
    --camera cam05 --frame 1 --output-dir out/finger_contact

Full annotation schemas, camera models, and body model setup are documented in the toolkit's docs.