DeepHandMesh: A Weakly-Supervised Deep Encoder-Decoder Framework for High-Fidelity Hand Mesh Modeling

Introduction

This repo is official PyTorch implementation of DeepHandMesh: A Weakly-Supervised Deep Encoder-Decoder Framework for High-Fidelity Hand Mesh Modeling (ECCV 2020. Oral.).

Dataset download (only provide data of subject 4)

Images

• Images.
• Run python download_images.py.
• All image files take 9.2 GB.

Annotations

• Depthmaps.
• 3D scans.
• Run python download_depthmaps.py and python download_3d_scans_decimated.py.

Hand model, 3D keypoints, and camera parameters

• Download.
• Run python download_others.py.

* 3D_scans_decimated: decimated 3D scans (uncompressed files take 114 GB)
|-- [frame_index].ply

* depthmaps: multi-view depth maps (uncompressed files take 421 GB)
|-- [frame_index]
|   |-- depth[camera_index].pkl

* keypoints.zip: 3D hand joint coordinates (in world coordinate system. milimeter scale.)
|-- keypoints[frame_index].pts: (joint index, x, y, z, summation of scores of the annotator, the number of views used for the triangulation)
The joint index follows this order: 'b_r_thumb_null', 'b_r_thumb3', 'b_r_thumb2', 'b_r_thumb1', 'b_r_index_null', 'b_r_index3', 'b_r_index2', 'b_r_index1', 'b_r_middle_null', 'b_r_middle3', 'b_r_middle2', 'b_r_middle1', 'b_r_ring_null', 'b_r_ring3', 'b_r_ring2', 'b_r_ring1', 'b_r_pinky_null', 'b_r_pinky3', 'b_r_pinky2', 'b_r_pinky1', 'b_r_wrist'

* KRT_512: camera parameters
|-- [camera index, intrinsic matrix, dist, extrinsic matrix]

* hand_model.zip: contains an initial 3D hand model

Codes and pre-trained models

Go to github.

Results

Effect of Identity- and Pose-Dependent Correctives

Comparison with MANO

Reference

@InProceedings{Moon_2020_ECCV_DeepHandMesh,  
author = {Moon, Gyeongsik and Shiratori, Takaaki and Lee, Kyoung Mu},  
title = {DeepHandMesh: A Weakly-supervised Deep Encoder-Decoder Framework for High-fidelity Hand Mesh Modeling},  
booktitle = {European Conference on Computer Vision (ECCV)},  
year = {2020}  
}