SMPLer-X

Install

conda create -n smplerx python=3.8 -y
conda activate smplerx
conda install pytorch==1.12.0 torchvision==0.13.0 torchaudio==0.12.0 cudatoolkit=11.3 -c pytorch -y
wget http://download.openmmlab.sensetime.com/mmcv/dist/cu113/torch1.12.0/mmcv_full-1.7.1-cp38-cp38-manylinux1_x86_64.whl
pip install mmcv_full-1.7.1-cp38-cp38-manylinux1_x86_64.whl
rm mmcv_full-1.7.1-cp38-cp38-manylinux1_x86_64.whl
pip install -r requirements.txt

# install mmpose
cd main/transformer_utils
pip install -v -e .
cd ../..

Preparation

• download all datasets
  • 3DPW
  • AGORA
  • ARCTIC
  • BEDLAM
  • BEHAVE
  • CHI3D
  • CrowdPose
  • EgoBody
  • EHF
  • FIT3D
  • GTA-Human
  • Human3.6M
  • HumanSC3D
  • InstaVariety
  • LSPET
  • MPII
  • MPI-INF-3DHP
  • MSCOCO
  • MTP
  • MuCo-3DHP
  • OCHuman
  • PoseTrack
  • PROX
  • RenBody
  • RICH
  • SPEC
  • SSP3D
  • SynBody
  • Talkshow
  • UBody
  • UP3D
• process all datasets into HumanData format, except the following:
  • AGORA, MSCOCO, MPII, Human3.6M, UBody.
  • Follow OSX in preparing these 5 datasets.
• follow OSX in preparing pretrained ViTPose models. Download the ViTPose pretrained weights for ViT-small and ViT-huge from here.
• download SMPL-X and SMPL body models.

The file structure should be like:

SMPLer-X/
├── common/
│   └── utils/
│       └── human_model_files/  # body model
│           ├── smpl/
│           │   ├──SMPL_NEUTRAL.pkl
│           │   ├──SMPL_MALE.pkl
│           │   └──SMPL_FEMALE.pkl
│           └── smplx/
│               ├──MANO_SMPLX_vertex_ids.pkl
│               ├──SMPL-X__FLAME_vertex_ids.npy
│               ├──SMPLX_NEUTRAL.pkl
│               ├──SMPLX_to_J14.pkl
│               ├──SMPLX_NEUTRAL.npz
│               ├──SMPLX_MALE.npz
│               └──SMPLX_FEMALE.npz
├── data/
├── main/
├── pretrained_models/  # pretrained ViT-Pose models
│   ├── vitpose_small.pth
│   ├── vitpose_base.pth
│   ├── vitpose_large.pth
│   └── vitpose_huge.pth
└── dataset/  
    ├── AGORA/       
    ├── ARCTIC/      
    ├── BEDLAM/      
    ├── Behave/      
    ├── CHI3D/       
    ├── CrowdPose/   
    ├── EgoBody/     
    ├── EHF/         
    ├── FIT3D/                
    ├── GTA_Human2/           
    ├── Human36M/             
    ├── HumanSC3D/            
    ├── InstaVariety/         
    ├── LSPET/                
    ├── MPII/                 
    ├── MPI_INF_3DHP/         
    ├── MSCOCO/               
    ├── MTP/                    
    ├── MuCo/                   
    ├── OCHuman/                
    ├── PoseTrack/                
    ├── PROX/                   
    ├── PW3D/                   
    ├── RenBody/
    ├── RICH/
    ├── SPEC/
    ├── SSP3D/
    ├── SynBody/
    ├── Talkshow/
    ├── UBody/
    ├── UP3D/
    └── preprocessed_datasets/  # HumanData files

Training

cd main
sh slurm_train.sh {JOB_NAME} {NUM_GPU} {CONFIG_FILE}

# For training SMPLer-X-H32
sh slurm_train.sh smpler-x-h32 16 config_smpler_x_h32.py

• CONFIG_FILE is the file name under ./config, e.g. ./config/config_base.py, more configs can be found under ./config
• Logs and checkpoints will be saved to ../output/train_{JOB_NAME}_{DATE_TIME}

Testing

# To eval the model ../output/{TRAIN_OUTPUT_DIR}/model_dump/snapshot_{CKPT_ID}.pth.tar 
# with confing ../output/{TRAIN_OUTPUT_DIR}/code/config_base.py
cd main
sh slurm_test.sh {JOB_NAME} {NUM_GPU} {TRAIN_OUTPUT_DIR} {CKPT_ID}

• NUM_GPU = 1 is recommended for testing
• Logs and results will be saved to ../output/test_{JOB_NAME}_ep{CKPT_ID}_{TEST_DATSET}

References

• Hand4Whole
• OSX
• MMHuman3D