SMPLer-X: Scaling Up Expressive Human Pose and Shape Estimation

Useful links

[Homepage]      [HuggingFace]      [arXiv]      [Video]      [MMHuman3D]

News

• [2024-02-29] HuggingFace demo is online!
• [2023-10-23] Support visualization through SMPL-X mesh overlay and add inference docker.
• [2023-10-02] arXiv preprint is online!
• [2023-09-28] Homepage and Video are online!
• [2023-07-19] Pretrained models are released.
• [2023-06-15] Training and testing code is released.

Gallery

Install

conda create -n smplerx python=3.10 -y
conda activate smplerx
conda install cudatoolkit=11.7 -c nvidia -y
pip install -r pre-requirements.txt
pip install mmcv==2.1.0 -f https://download.openmmlab.com/mmcv/dist/cu117/torch2.0.0/index.html
pip install -r requirements.txt

# install mmpose
cd main/transformer_utils
pip install -v -e .
cd ../..

Pretrained Models

Model	Backbone	#Datasets	#Inst.	#Params	MPE	Download	FPS
SMPLer-X-S32	ViT-S	32	4.5M	32M	82.6	model	36.17
SMPLer-X-B32	ViT-B	32	4.5M	103M	74.3	model	33.09
SMPLer-X-L32	ViT-L	32	4.5M	327M	66.2	model	24.44
SMPLer-X-H32	ViT-H	32	4.5M	662M	63.0	model	17.47

• MPE (Mean Primary Error): the average of the primary errors on five benchmarks (AGORA, EgoBody, UBody, 3DPW, and EHF)
• FPS (Frames Per Second): the average inference speed on a single Tesla V100 GPU, batch size = 1

Preparation

• download SMPL-X and SMPL body models.
• download mmdet pretrained model for inference.

The file structure should be like:

SMPLer-X/
├── common/
│   └── utils/
│       └── human_model_files/  # body model
│           ├── smpl/
│           │   ├──SMPL_NEUTRAL.pkl
│           │   ├──SMPL_MALE.pkl
│           │   └──SMPL_FEMALE.pkl
│           └── smplx/
│               ├──MANO_SMPLX_vertex_ids.pkl
│               ├──SMPL-X__FLAME_vertex_ids.npy
│               ├──SMPLX_NEUTRAL.pkl
│               ├──SMPLX_to_J14.pkl
│               ├──SMPLX_NEUTRAL.npz
│               ├──SMPLX_MALE.npz
│               └──SMPLX_FEMALE.npz
├── main/
└── pretrained_models/  # pretrained ViT-Pose, SMPLer_X and mmdet models
    ├── mmdet/
    │   ├──faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth
    │   └──mmdet_faster_rcnn_r50_fpn_coco.py
    ├── smpler_x_s32.pth.tar
    ├── smpler_x_b32.pth.tar
    ├── smpler_x_l32.pth.tar
    ├── smpler_x_h32.pth.tar
    ├── vitpose_small.pth
    ├── vitpose_base.pth
    ├── vitpose_large.pth
    └── vitpose_huge.pth

Inference

python demo.py --input_video {VIDEO_FILE} --pretrained_model {PRETRAINED_CKPT} --show_verts

# For inferencing test_video.mp4 (24FPS) with smpler_x_h32
python demo.py --input_video test_video.mp4 --pretrained_model smpler_x_h32 --show_verts

Huggingface

• Replace README.md with README_huggingface.md
• add mmcv into requirements.txt
  • eg: if using zero-gpu, add 'https://download.openmmlab.com/mmcv/dist/cu117/torch2.0.0/mmcv-2.1.0-cp310-cp310-manylinux1_x86_64.whl'

FAQ

• RuntimeError: Subtraction, the '-' operator, with a bool tensor is not supported. If you are trying to invert a mask, use the '~' or 'logical_not()' operator instead.

  Follow this post and modify torchgeometry

• KeyError: 'SinePositionalEncoding is already registered in position encoding' or any other similar KeyErrors due to duplicate module registration.

  Manually add force=True to respective module registration under main/transformer_utils/mmpose/models/utils, e.g. @POSITIONAL_ENCODING.register_module(force=True) in this file

• How do I animate my virtual characters with SMPLer-X output (like that in the demo video)?

  • We are working on that, please stay tuned! Currently, this repo supports SMPL-X estimation and a simple visualization (overlay of SMPL-X vertices).

References

• Hand4Whole
• OSX
• MMHuman3D