DeepHandMesh: A Weakly-Supervised Deep Encoder-Decoder Framework for High-Fidelity Hand Mesh Modeling

Introduction

This repo is official PyTorch implementation of DeepHandMesh: A Weakly-Supervised Deep Encoder-Decoder Framework for High-Fidelity Hand Mesh Modeling (ECCV 2020. Oral.).

Dataset download (only provide data of subject 4)

Images

• Images (v1.0).
• To verify downloaded files using md5sum, run python verify_download.py.
• To unzip images, run sh unzip.sh.
• All image files take 9.2 GB.

Annotations

• Annotation (v1.0).
• To verify downloaded 3D scan files using md5sum, run python verify_download_3d_scan.py.
• To verify downloaded depthmap files using md5sum, run python verify_download_depthmap.py.
• To unzip 3D scan files, run sh unzip_3d_scan.sh.
• To unzip depthmap files, run sh unzip_depthmap.sh.

* 3D_scans_decimated: decimated 3D scans (uncompressed files take 114 GB)
|-- [frame_index].ply

* depthmaps: multi-view depth maps (uncompressed files take 421 GB)
|-- [frame_index]
|   |-- depth[camera_index].pkl

* keypoints.zip: 3D hand joint coordinates (in world coordinate system. milimeter scale.)
|-- keypoints[frame_index].pts: (joint index, x, y, z, summation of scores of the annotator, the number of views used for the triangulation)
The joint index follows this order: 'b_r_thumb_null', 'b_r_thumb3', 'b_r_thumb2', 'b_r_thumb1', 'b_r_index_null', 'b_r_index3', 'b_r_index2', 'b_r_index1', 'b_r_middle_null', 'b_r_middle3', 'b_r_middle2', 'b_r_middle1', 'b_r_ring_null', 'b_r_ring3', 'b_r_ring2', 'b_r_ring1', 'b_r_pinky_null', 'b_r_pinky3', 'b_r_pinky2', 'b_r_pinky1', 'b_r_wrist'

* KRT_512: camera parameters
|-- [camera index, intrinsic matrix, dist, extrinsic matrix]

* hand_model.zip: contains an initial 3D hand model

Tips for a Google Drive download

To download multiple files from Google drive without compressing them, try this. If you have a problem with ‘Download limit’ problem when tried to download dataset from google drive link, please try this trick.

* Go the shared folder, which contains files you want to copy to your drive  
* Select all the files you want to copy  
* In the upper right corner click on three vertical dots and select “make a copy”  
* Then, the file is copied to your personal google drive account. You can download it from your personal account.  

Codes and pre-trained models

Go to github.

Results

Effect of Identity- and Pose-Dependent Correctives

Comparison with MANO

Reference

@InProceedings{Moon_2020_ECCV_DeepHandMesh,  
author = {Moon, Gyeongsik and Shiratori, Takaaki and Lee, Kyoung Mu},  
title = {DeepHandMesh: A Weakly-supervised Deep Encoder-Decoder Framework for High-fidelity Hand Mesh Modeling},  
booktitle = {European Conference on Computer Vision (ECCV)},  
year = {2020}  
}