electron-rare/SMPLer-X
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

merge into: electron-rare/SMPLer-X:main
Branches Tags
electron-rare/SMPLer-X:main electron-rare/SMPLer-X:inf_ref electron-rare/SMPLer-X:huggingface
...
pull from: electron-rare/SMPLer-X:huggingface
Branches Tags
electron-rare/SMPLer-X:main electron-rare/SMPLer-X:inf_ref electron-rare/SMPLer-X:huggingface

1 Commits
main ... huggingface

Author SHA1 Message Date
onescotch 55a7d0c0c3 add huggingface implementation 2024-03-27 19:44:08 +08:00
178 changed files with 1098 additions and 27229 deletions
Expand all files Collapse all files
README.md
+26 -148
View File
@@ -29,12 +29,14 @@
## Install
```bash
conda create -n smplerx python=3.8 -y
conda create -n smplerx python=3.10 -y
conda activate smplerx
conda install pytorch==1.12.0 torchvision==0.13.0 torchaudio==0.12.0 cudatoolkit=11.3 -c pytorch -y
pip install mmcv-full==1.7.1 -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.12.0/index.html
conda install cudatoolkit=11.7 -c nvidia -y
pip install -r pre-requirements.txt
pip install mmcv==2.1.0 -f https://download.openmmlab.com/mmcv/dist/cu117/torch2.0.0/index.html
pip install -r requirements.txt
# install mmpose
@@ -43,18 +45,6 @@ pip install -v -e .
cd ../..
```
## Docker Support (Early Stage)
```
docker pull wcwcw/smplerx_inference:v0.2
docker run --gpus all -v <vid_input_folder>:/smplerx_inference/vid_input \
-v <vid_output_folder>:/smplerx_inference/vid_output \
wcwcw/smplerx_inference:v0.2 --vid <video_name>.mp4
# Currently any customization need to be applied to /smplerx_inference/smplerx/inference_docker.py
```
- We recently developed a docker for inference at docker hub.
- This docker image uses SMPLer-X-H32 as inference baseline and was tested at RTX3090 & WSL2 (Ubuntu 20.04).
## Pretrained Models
| Model | Backbone | #Datasets | #Inst. | #Params | MPE | Download | FPS |
|:------------:|:--------:|:---------:|:------:|:-------:|:----:|:--------:|:-----:|
@@ -66,44 +56,9 @@ docker run --gpus all -v <vid_input_folder>:/smplerx_inference/vid_input \
* FPS (Frames Per Second): the average inference speed on a single Tesla V100 GPU, batch size = 1
## Preparation
- download all datasets
- [3DPW](https://virtualhumans.mpi-inf.mpg.de/3DPW/)
- [AGORA](https://agora.is.tue.mpg.de/index.html)
- [ARCTIC](https://arctic.is.tue.mpg.de/)
- [BEDLAM](https://bedlam.is.tue.mpg.de/index.html)
- [BEHAVE](https://github.com/xiexh20/behave-dataset)
- [CHI3D](https://ci3d.imar.ro/)
- [CrowdPose](https://github.com/Jeff-sjtu/CrowdPose)
- [EgoBody](https://sanweiliti.github.io/egobody/egobody.html)
- [EHF](https://smpl-x.is.tue.mpg.de/index.html)
- [FIT3D](https://fit3d.imar.ro/)
- [GTA-Human](https://caizhongang.github.io/projects/GTA-Human/)
- [Human3.6M](http://vision.imar.ro/human3.6m/description.php)
- [HumanSC3D](https://sc3d.imar.ro/)
- [InstaVariety](https://github.com/akanazawa/human_dynamics/blob/master/doc/insta_variety.md)
- [LSPET](http://sam.johnson.io/research/lspet.html)
- [MPII](http://human-pose.mpi-inf.mpg.de/)
- [MPI-INF-3DHP](https://vcai.mpi-inf.mpg.de/3dhp-dataset/)
- [MSCOCO](https://cocodataset.org/#home)
- [MTP](https://tuch.is.tue.mpg.de/)
- [MuCo-3DHP](https://vcai.mpi-inf.mpg.de/projects/SingleShotMultiPerson/)
- [OCHuman](https://github.com/liruilong940607/OCHumanApi)
- [PoseTrack](https://posetrack.net/)
- [PROX](https://prox.is.tue.mpg.de/)
- [RenBody](https://magichub.com/datasets/openxd-renbody/)
- [RICH](https://rich.is.tue.mpg.de/index.html)
- [SPEC](https://spec.is.tue.mpg.de/index.html)
- [SSP3D](https://github.com/akashsengupta1997/SSP-3D)
- [SynBody](https://maoxie.github.io/SynBody/)
- [Talkshow](https://talkshow.is.tue.mpg.de/)
- [UBody](https://github.com/IDEA-Research/OSX)
- [UP3D](https://files.is.tuebingen.mpg.de/classner/up/)
- process all datasets into [HumanData](https://github.com/open-mmlab/mmhuman3d/blob/main/docs/human_data.md) format, except the following:
- AGORA, MSCOCO, MPII, Human3.6M, UBody.
- follow [OSX](https://github.com/IDEA-Research/OSX) in preparing these 5 datasets.
- follow [OSX](https://github.com/IDEA-Research/OSX) in preparing pretrained ViTPose models. Download the ViTPose pretrained weights for ViT-small and ViT-huge from [here](https://github.com/ViTAE-Transformer/ViTPose).
- download [SMPL-X](https://smpl-x.is.tue.mpg.de/) and [SMPL](https://smpl.is.tue.mpg.de/) body models.
- download mmdet pretrained [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth) and [config](https://github.com/openxrlab/xrmocap/blob/main/configs/modules/human_perception/mmdet_faster_rcnn_r50_fpn_coco.py) for inference.
- download mmdet pretrained [model](https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth) for inference.
The file structure should be like:
```
@@ -123,111 +78,34 @@ SMPLer-X/
│ ├──SMPLX_NEUTRAL.npz
│ ├──SMPLX_MALE.npz
│ └──SMPLX_FEMALE.npz
├── data/
├── main/
├── demo/
│ ├── videos/
│ ├── images/
│ └── results/
├── pretrained_models/ # pretrained ViT-Pose, SMPLer_X and mmdet models
│ ├── mmdet/
│ │ ├──faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth
│ │ └──mmdet_faster_rcnn_r50_fpn_coco.py
│ ├── smpler_x_s32.pth.tar
│ ├── smpler_x_b32.pth.tar
│ ├── smpler_x_l32.pth.tar
│ ├── smpler_x_h32.pth.tar
│ ├── vitpose_small.pth
│ ├── vitpose_base.pth
│ ├── vitpose_large.pth
│ └── vitpose_huge.pth
└── dataset/
├── AGORA/
├── ARCTIC/
├── BEDLAM/
├── Behave/
├── CHI3D/
├── CrowdPose/
├── EgoBody/
├── EHF/
├── FIT3D/
├── GTA_Human2/
├── Human36M/
├── HumanSC3D/
├── InstaVariety/
├── LSPET/
├── MPII/
├── MPI_INF_3DHP/
├── MSCOCO/
├── MTP/
├── MuCo/
├── OCHuman/
├── PoseTrack/
├── PROX/
├── PW3D/
├── RenBody/
├── RICH/
├── SPEC/
├── SSP3D/
├── SynBody/
├── Talkshow/
├── UBody/
├── UP3D/
└── preprocessed_datasets/ # HumanData files
└── pretrained_models/ # pretrained ViT-Pose, SMPLer_X and mmdet models
├── mmdet/
│ ├──faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth
│ └──mmdet_faster_rcnn_r50_fpn_coco.py
├── smpler_x_s32.pth.tar
├── smpler_x_b32.pth.tar
├── smpler_x_l32.pth.tar
├── smpler_x_h32.pth.tar
├── vitpose_small.pth
├── vitpose_base.pth
├── vitpose_large.pth
└── vitpose_huge.pth
```
## Inference
- Place the video for inference under `SMPLer-X/demo/videos`
- Prepare the pretrained models to be used for inference under `SMPLer-X/pretrained_models`
- Prepare the mmdet pretrained model and config under `SMPLer-X/pretrained_models`
- Inference output will be saved in `SMPLer-X/demo/results`
```bash
cd main
sh slurm_inference.sh {VIDEO_FILE} {FORMAT} {FPS} {PRETRAINED_CKPT}
```
python demo.py --input_video {VIDEO_FILE} --pretrained_model {PRETRAINED_CKPT} --show_verts
# For inferencing test_video.mp4 (24FPS) with smpler_x_h32
sh slurm_inference.sh test_video mp4 24 smpler_x_h32
python demo.py --input_video test_video.mp4 --pretrained_model smpler_x_h32 --show_verts
```
## 2D Smplx Overlay
We provide a lightweight visualization script for mesh overlay based on pyrender.
- Use ffmpeg to split video into images
- The visualization script takes inference results (see above) as the input.
```bash
ffmpeg -i {VIDEO_FILE} -f image2 -vf fps=30 \
{SMPLERX INFERENCE DIR}/{VIDEO NAME (no extension)}/orig_img/%06d.jpg \
-hide_banner -loglevel error
cd main && python render.py \
--data_path {SMPLERX INFERENCE DIR} --seq {VIDEO NAME} \
--image_path {SMPLERX INFERENCE DIR}/{VIDEO NAME} \
--render_biggest_person False
```
## Training
```bash
cd main
sh slurm_train.sh {JOB_NAME} {NUM_GPU} {CONFIG_FILE}
# For training SMPLer-X-H32 with 16 GPUS
sh slurm_train.sh smpler_x_h32 16 config_smpler_x_h32.py
```
- CONFIG_FILE is the file name under `SMPLer-X/main/config`
- Logs and checkpoints will be saved to `SMPLer-X/output/train_{JOB_NAME}_{DATE_TIME}`
## Testing
```bash
# To eval the model ../output/{TRAIN_OUTPUT_DIR}/model_dump/snapshot_{CKPT_ID}.pth.tar
# with confing ../output/{TRAIN_OUTPUT_DIR}/code/config_base.py
cd main
sh slurm_test.sh {JOB_NAME} {NUM_GPU} {TRAIN_OUTPUT_DIR} {CKPT_ID}
```
- NUM_GPU = 1 is recommended for testing
- Logs and results will be saved to `SMPLer-X/output/test_{JOB_NAME}_ep{CKPT_ID}_{TEST_DATSET}`
## Huggingface
- Replace README.md with README_huggingface.md
- add mmcv into requirements.txt
- eg: if using zero-gpu, add 'https://download.openmmlab.com/mmcv/dist/cu117/torch2.0.0/mmcv-2.1.0-cp310-cp310-manylinux1_x86_64.whl'
## FAQ
- `RuntimeError: Subtraction, the '-' operator, with a bool tensor is not supported. If you are trying to invert a mask, use the '~' or 'logical_not()' operator instead.`
README_huggingface.md
+13
View File
@@ -0,0 +1,13 @@
---
title: SMPLer X
emoji: ⚡
colorFrom: blue
colorTo: indigo
sdk: gradio
python_version: 3.9
sdk_version: 4.16.0
app_file: app.py
pinned: false
---
Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
app.py
+136
View File
@@ -0,0 +1,136 @@
import os
import sys
import os.path as osp
from pathlib import Path
import cv2
import gradio as gr
import torch
import math
import spaces
from huggingface_hub import hf_hub_download
try:
import mmpose
except:
os.system('pip install /home/user/app/main/transformer_utils')
hf_hub_download(repo_id="caizhongang/SMPLer-X", filename="smpler_x_h32.pth.tar", local_dir="/home/user/app/pretrained_models")
os.system('cp -rf /home/user/app/assets/conversions.py /usr/local/lib/python3.10/site-packages/torchgeometry/core/conversions.py')
DEFAULT_MODEL='smpler_x_h32'
OUT_FOLDER = '/home/user/app/demo_out'
os.makedirs(OUT_FOLDER, exist_ok=True)
num_gpus = 1 if torch.cuda.is_available() else -1
print("!!!", torch.cuda.is_available())
print(torch.cuda.device_count())
print(torch.version.cuda)
index = torch.cuda.current_device()
print(index)
print(torch.cuda.get_device_name(index))
# from main.inference import Inferer
# inferer = Inferer(DEFAULT_MODEL, num_gpus, OUT_FOLDER)
@spaces.GPU(enable_queue=True, duration=300)
def infer(video_input, in_threshold=0.5, num_people="Single person", render_mesh=False):
from main.inference import Inferer
inferer = Inferer(DEFAULT_MODEL, num_gpus, OUT_FOLDER)
os.system(f'rm -rf {OUT_FOLDER}/*')
multi_person = False if (num_people == "Single person") else True
cap = cv2.VideoCapture(video_input)
fps = math.ceil(cap.get(5))
width = int(cap.get(3))
height = int(cap.get(4))
fourcc = cv2.VideoWriter_fourcc(*'mp4v')
video_path = osp.join(OUT_FOLDER, f'out.m4v')
final_video_path = osp.join(OUT_FOLDER, f'out.mp4')
video_output = cv2.VideoWriter(video_path, fourcc, fps, (width, height))
success = 1
frame = 0
while success:
success, original_img = cap.read()
if not success:
break
frame += 1
img, mesh_paths, smplx_paths = inferer.infer(original_img, in_threshold, frame, multi_person, not(render_mesh))
video_output.write(img)
yield img, None, None, None
cap.release()
video_output.release()
cv2.destroyAllWindows()
os.system(f'ffmpeg -i {video_path} -c copy {final_video_path}')
#Compress mesh and smplx files
save_path_mesh = os.path.join(OUT_FOLDER, 'mesh')
save_mesh_file = os.path.join(OUT_FOLDER, 'mesh.zip')
os.makedirs(save_path_mesh, exist_ok= True)
save_path_smplx = os.path.join(OUT_FOLDER, 'smplx')
save_smplx_file = os.path.join(OUT_FOLDER, 'smplx.zip')
os.makedirs(save_path_smplx, exist_ok= True)
os.system(f'zip -r {save_mesh_file} {save_path_mesh}')
os.system(f'zip -r {save_smplx_file} {save_path_smplx}')
yield img, video_path, save_mesh_file, save_smplx_file
TITLE = '''<h1 align="center">SMPLer-X: Scaling Up Expressive Human Pose and Shape Estimation</h1>'''
VIDEO = '''
<center><iframe width="960" height="540"
src="https://www.youtube.com/embed/DepTqbPpVzY?si=qSeQuX-bgm_rON7E"title="SMPLer-X: Scaling Up Expressive Human Pose and Shape Estimation" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen>
</iframe>
</center><br>'''
DESCRIPTION = '''
<b>Official Gradio demo</b> for <a href="https://caizhongang.com/projects/SMPLer-X/"><b>SMPLer-X: Scaling Up Expressive Human Pose and Shape Estimation</b></a>.<br>
<p>
Note: You can drop a video at the panel (or select one of the examples)
to obtain the 3D parametric reconstructions of the detected humans.
</p>
'''
with gr.Blocks(title="SMPLer-X", css=".gradio-container") as demo:
gr.Markdown(TITLE)
gr.HTML(VIDEO)
gr.Markdown(DESCRIPTION)
with gr.Row():
with gr.Column():
video_input = gr.Video(label="Input video", elem_classes="video")
threshold = gr.Slider(0, 1.0, value=0.5, label='BBox detection threshold')
with gr.Column(scale=2):
num_people = gr.Radio(
choices=["Single person", "Multiple people"],
value="Single person",
label="Number of people",
info="Choose how many people are there in the video. Choose 'single person' for faster inference.",
interactive=True,
scale=1,)
gr.HTML("""<br/>""")
mesh_as_vertices = gr.Checkbox(
label="Render as mesh",
info="By default, the estimated SMPL-X parameters are rendered as vertices for faster visualization. Check this option if you want to visualize meshes instead.",
interactive=True,
scale=1,)
send_button = gr.Button("Infer")
gr.HTML("""<br/>""")
with gr.Row():
with gr.Column():
processed_frames = gr.Image(label="Last processed frame")
video_output = gr.Video(elem_classes="video")
with gr.Column():
meshes_output = gr.File(label="3D meshes")
smplx_output = gr.File(label= "SMPL-X models")
# example_images = gr.Examples([])
send_button.click(fn=infer, inputs=[video_input, threshold, num_people, mesh_as_vertices], outputs=[processed_frames, video_output, meshes_output, smplx_output])
# with gr.Row():
example_videos = gr.Examples([
['/home/user/app/assets/01.mp4'],
['/home/user/app/assets/02.mp4'],
['/home/user/app/assets/03.mp4'],
['/home/user/app/assets/04.mp4'],
['/home/user/app/assets/05.mp4'],
['/home/user/app/assets/06.mp4'],
['/home/user/app/assets/07.mp4'],
['/home/user/app/assets/08.mp4'],
['/home/user/app/assets/09.mp4'],
],
inputs=[video_input, 0.5])
#demo.queue()
demo.queue().launch(debug=True)
common/base.py
+3 -273
View File
@@ -9,7 +9,7 @@ from logger import colorlogger
from torch.nn.parallel.data_parallel import DataParallel
from config import cfg
from SMPLer_X import get_model
from dataset import MultipleDatasets
# ddp
import torch.distributed as dist
from torch.utils.data import DistributedSampler
@@ -17,16 +17,6 @@ import torch.utils.data.distributed
from utils.distribute_utils import (
get_rank, is_main_process, time_synchronized, get_group_idx, get_process_groups
)
from mmcv.runner import get_dist_info
# dynamic dataset import
for i in range(len(cfg.trainset_3d)):
exec('from ' + cfg.trainset_3d[i] + ' import ' + cfg.trainset_3d[i])
for i in range(len(cfg.trainset_2d)):
exec('from ' + cfg.trainset_2d[i] + ' import ' + cfg.trainset_2d[i])
for i in range(len(cfg.trainset_humandata)):
exec('from ' + cfg.trainset_humandata[i] + ' import ' + cfg.trainset_humandata[i])
exec('from ' + cfg.testset + ' import ' + cfg.testset)
class Base(object):
@@ -51,266 +41,6 @@ class Base(object):
def _make_model(self):
return
class Trainer(Base):
def __init__(self, distributed=False, gpu_idx=None):
super(Trainer, self).__init__(log_name='train_logs.txt')
self.distributed = distributed
self.gpu_idx = gpu_idx
def get_optimizer(self, model):
normal_param = []
special_param = []
for module in model.module.special_trainable_modules:
special_param += list(module.parameters())
# print(module)
for module in model.module.trainable_modules:
normal_param += list(module.parameters())
# self.logger.info(f"N-{self.gpu_idx}, {normal_param}")
# self.logger.info("S", special_param)
optim_params = [
{ # add normal params first
'params': normal_param,
'lr': cfg.lr
},
{
'params': special_param,
'lr': cfg.lr * cfg.lr_mult
},
]
optimizer = torch.optim.Adam(optim_params, lr=cfg.lr)
return optimizer
def save_model(self, state, epoch):
file_path = osp.join(cfg.model_dir, 'snapshot_{}.pth.tar'.format(str(epoch)))
# do not save smplx layer weights
dump_key = []
for k in state['network'].keys():
if 'smplx_layer' in k:
dump_key.append(k)
for k in dump_key:
state['network'].pop(k, None)
torch.save(state, file_path)
self.logger.info("Write snapshot into {}".format(file_path))
def load_model(self, model, optimizer):
if cfg.pretrained_model_path is not None:
ckpt_path = cfg.pretrained_model_path
ckpt = torch.load(ckpt_path, map_location=torch.device('cpu')) # solve CUDA OOM error in DDP
model.load_state_dict(ckpt['network'], strict=False)
self.logger.info('Load checkpoint from {}'.format(ckpt_path))
if not hasattr(cfg, 'start_over') or cfg.start_over:
start_epoch = 0
else:
optimizer.load_state_dict(ckpt['optimizer'])
start_epoch = ckpt['epoch'] + 1
self.logger.info(f'Load optimizer, start from{start_epoch}')
else:
start_epoch = 0
return start_epoch, model, optimizer
def get_lr(self):
for g in self.optimizer.param_groups:
cur_lr = g['lr']
return cur_lr
def _make_batch_generator(self):
# data load and construct batch generator
self.logger_info("Creating dataset...")
trainset3d_loader = []
for i in range(len(cfg.trainset_3d)):
trainset3d_loader.append(eval(cfg.trainset_3d[i])(transforms.ToTensor(), "train"))
trainset2d_loader = []
for i in range(len(cfg.trainset_2d)):
trainset2d_loader.append(eval(cfg.trainset_2d[i])(transforms.ToTensor(), "train"))
trainset_humandata_loader = []
for i in range(len(cfg.trainset_humandata)):
trainset_humandata_loader.append(eval(cfg.trainset_humandata[i])(transforms.ToTensor(), "train"))
data_strategy = getattr(cfg, 'data_strategy', None)
if data_strategy == 'concat':
print("Using [concat] strategy...")
trainset_loader = MultipleDatasets(trainset3d_loader + trainset2d_loader + trainset_humandata_loader,
make_same_len=False, verbose=True)
elif data_strategy == 'balance':
total_len = getattr(cfg, 'total_data_len', 'auto')
print(f"Using [balance] strategy with total_data_len : {total_len}...")
trainset_loader = MultipleDatasets(trainset3d_loader + trainset2d_loader + trainset_humandata_loader,
make_same_len=True, total_len=total_len, verbose=True)
else:
# original strategy implementation
valid_loader_num = 0
if len(trainset3d_loader) > 0:
trainset3d_loader = [MultipleDatasets(trainset3d_loader, make_same_len=False)]
valid_loader_num += 1
else:
trainset3d_loader = []
if len(trainset2d_loader) > 0:
trainset2d_loader = [MultipleDatasets(trainset2d_loader, make_same_len=False)]
valid_loader_num += 1
else:
trainset2d_loader = []
if len(trainset_humandata_loader) > 0:
trainset_humandata_loader = [MultipleDatasets(trainset_humandata_loader, make_same_len=False)]
valid_loader_num += 1
if valid_loader_num > 1:
trainset_loader = MultipleDatasets(trainset3d_loader + trainset2d_loader + trainset_humandata_loader, make_same_len=True)
else:
trainset_loader = MultipleDatasets(trainset3d_loader + trainset2d_loader + trainset_humandata_loader, make_same_len=False)
self.itr_per_epoch = math.ceil(len(trainset_loader) / cfg.num_gpus / cfg.train_batch_size)
if self.distributed:
self.logger_info(f"Total data length {len(trainset_loader)}.")
rank, world_size = get_dist_info()
self.logger_info("Using distributed data sampler.")
sampler_train = DistributedSampler(trainset_loader, world_size, rank, shuffle=True)
self.batch_generator = DataLoader(dataset=trainset_loader, batch_size=cfg.train_batch_size,
shuffle=False, num_workers=cfg.num_thread, sampler=sampler_train,
pin_memory=True, persistent_workers=True if cfg.num_thread > 0 else False, drop_last=True)
else:
self.batch_generator = DataLoader(dataset=trainset_loader, batch_size=cfg.num_gpus * cfg.train_batch_size,
shuffle=True, num_workers=cfg.num_thread,
pin_memory=True, drop_last=True)
def _make_model(self):
# prepare network
self.logger_info("Creating graph and optimizer...")
model = get_model('train')
if getattr(cfg, 'fine_tune', None) == 'backbone':
print("Fine-tuning [backbone]...")
for module in model.head:
for param in module.parameters():
param.requires_grad = False
for module in model.neck:
for param in module.parameters():
param.requires_grad = False
elif getattr(cfg, 'fine_tune', None) == 'neck_and_head':
print("Fine-tuning [neck and head]...")
for param in model.encoder.parameters():
param.requires_grad = False
elif getattr(cfg, 'fine_tune', None) == 'head':
print("Fine-tuning [head]...")
for param in model.encoder.parameters():
param.requires_grad = False
for module in model.neck:
for param in module.parameters():
param.requires_grad = False
# ddp
if self.distributed:
self.logger_info("Using distributed data parallel.")
model.cuda()
if hasattr(cfg, 'syncbn') and cfg.syncbn:
self.logger_info("Using sync batch norm layers.")
process_groups = get_process_groups()
process_group = process_groups[get_group_idx()]
syncbn_model = torch.nn.SyncBatchNorm.convert_sync_batchnorm(model, process_group)
model = torch.nn.parallel.DistributedDataParallel(
syncbn_model, device_ids=[self.gpu_idx],
find_unused_parameters=True)
else:
model = torch.nn.parallel.DistributedDataParallel(
model, device_ids=[self.gpu_idx],
find_unused_parameters=True)
else:
# dp
model = DataParallel(model).cuda()
optimizer = self.get_optimizer(model)
if hasattr(cfg, "scheduler"):
if cfg.scheduler == 'cos':
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, cfg.end_epoch * self.itr_per_epoch,
eta_min=1e-6)
elif cfg.scheduler == 'step':
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, cfg.step_size, gamma=cfg.gamma,
last_epoch=- 1, verbose=False)
else:
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, cfg.end_epoch * self.itr_per_epoch,
eta_min=getattr(cfg,'min_lr',1e-6))
if cfg.continue_train:
if self.distributed:
start_epoch, model, optimizer = self.load_model(model, optimizer)
else:
start_epoch, model, optimizer = self.load_model(model, optimizer)
else:
start_epoch = 0
model.train()
self.scheduler = scheduler
self.start_epoch = start_epoch
self.model = model
self.optimizer = optimizer
def logger_info(self, info):
if self.distributed:
if is_main_process():
self.logger.info(info)
else:
self.logger.info(info)
class Tester(Base):
def __init__(self, test_epoch=None):
if test_epoch is not None:
self.test_epoch = int(test_epoch)
super(Tester, self).__init__(log_name='test_logs.txt')
def _make_batch_generator(self):
# data load and construct batch generator
self.logger.info("Creating dataset...")
testset_loader = eval(cfg.testset)(transforms.ToTensor(), "test")
batch_generator = DataLoader(dataset=testset_loader, batch_size=cfg.num_gpus * cfg.test_batch_size,
shuffle=False, num_workers=cfg.num_thread, pin_memory=True)
self.testset = testset_loader
self.batch_generator = batch_generator
def _make_model(self):
self.logger.info('Load checkpoint from {}'.format(cfg.pretrained_model_path))
# prepare network
self.logger.info("Creating graph...")
model = get_model('test')
model = DataParallel(model).cuda()
if not getattr(cfg, 'random_init', False):
ckpt = torch.load(cfg.pretrained_model_path, map_location=torch.device('cpu'))
from collections import OrderedDict
new_state_dict = OrderedDict()
for k, v in ckpt['network'].items():
if 'module' not in k:
k = 'module.' + k
k = k.replace('backbone', 'encoder').replace('body_rotation_net', 'body_regressor').replace(
'hand_rotation_net', 'hand_regressor')
new_state_dict[k] = v
self.logger.warning("Attention: Strict=False is set for checkpoint loading. Please check manually.")
model.load_state_dict(new_state_dict, strict=False)
model.eval()
else:
print('Random init!!!!!!!')
self.model = model
def _evaluate(self, outs, cur_sample_idx):
eval_result = self.testset.evaluate(outs, cur_sample_idx)
return eval_result
def _print_eval_result(self, eval_result):
self.testset.print_eval_result(eval_result)
class Demoer(Base):
def __init__(self, test_epoch=None):
if test_epoch is not None:
@@ -334,8 +64,8 @@ class Demoer(Base):
# prepare network
self.logger.info("Creating graph...")
model = get_model('test')
model = DataParallel(model).cuda()
ckpt = torch.load(cfg.pretrained_model_path)
model = DataParallel(model).to(cfg.device)
ckpt = torch.load(cfg.pretrained_model_path, map_location=cfg.device)
from collections import OrderedDict
new_state_dict = OrderedDict()
common/nets/smpler_x.py
+2 -2
View File
@@ -147,9 +147,9 @@ class HandRoI(nn.Module):
self.conv = make_conv_layers([feat_dim//8, feat_dim], kernel=1, stride=1, padding=0, bnrelu_final=False)
def forward(self, img_feat, lhand_bbox, rhand_bbox):
lhand_bbox = torch.cat((torch.arange(lhand_bbox.shape[0]).float().cuda()[:, None], lhand_bbox),
lhand_bbox = torch.cat((torch.arange(lhand_bbox.shape[0]).float().to(cfg.device)[:, None], lhand_bbox),
1) # batch_idx, xmin, ymin, xmax, ymax
rhand_bbox = torch.cat((torch.arange(rhand_bbox.shape[0]).float().cuda()[:, None], rhand_bbox),
rhand_bbox = torch.cat((torch.arange(rhand_bbox.shape[0]).float().to(cfg.device)[:, None], rhand_bbox),
1) # batch_idx, xmin, ymin, xmax, ymax
img_feat = self.deconv(img_feat)
lhand_bbox_roi = lhand_bbox.clone()
common/utils/distribute_utils.py
+1 -1
View File
@@ -7,7 +7,7 @@ import tempfile
import time
import torch
import torch.distributed as dist
from mmcv.runner import get_dist_info
from mmengine.dist import get_dist_info
import random
import numpy as np
import subprocess
common/utils/transforms.py
+6 -6
View File
@@ -80,7 +80,7 @@ def rot6d_to_axis_angle(x):
b3 = torch.cross(b1, b2)
rot_mat = torch.stack((b1, b2, b3), dim=-1) # 3x3 rotation matrix
rot_mat = torch.cat([rot_mat, torch.zeros((batch_size, 3, 1)).cuda().float()], 2) # 3x4 rotation matrix
rot_mat = torch.cat([rot_mat, torch.zeros((batch_size, 3, 1)).to(cfg.device).float()], 2) # 3x4 rotation matrix
axis_angle = tgm.rotation_matrix_to_angle_axis(rot_mat).reshape(-1, 3) # axis-angle
axis_angle[torch.isnan(axis_angle)] = 0.0
return axis_angle
@@ -106,8 +106,8 @@ def soft_argmax_2d(heatmap2d):
accu_x = heatmap2d.sum(dim=(2))
accu_y = heatmap2d.sum(dim=(3))
accu_x = accu_x * torch.arange(width).float().cuda()[None, None, :]
accu_y = accu_y * torch.arange(height).float().cuda()[None, None, :]
accu_x = accu_x * torch.arange(width).float().to(cfg.device)[None, None, :]
accu_y = accu_y * torch.arange(height).float().to(cfg.device)[None, None, :]
accu_x = accu_x.sum(dim=2, keepdim=True)
accu_y = accu_y.sum(dim=2, keepdim=True)
@@ -127,9 +127,9 @@ def soft_argmax_3d(heatmap3d):
accu_y = heatmap3d.sum(dim=(2, 4))
accu_z = heatmap3d.sum(dim=(3, 4))
accu_x = accu_x * torch.arange(width).float().cuda()[None, None, :]
accu_y = accu_y * torch.arange(height).float().cuda()[None, None, :]
accu_z = accu_z * torch.arange(depth).float().cuda()[None, None, :]
accu_x = accu_x * torch.arange(width).float().to(cfg.device)[None, None, :]
accu_y = accu_y * torch.arange(height).float().to(cfg.device)[None, None, :]
accu_z = accu_z * torch.arange(depth).float().to(cfg.device)[None, None, :]
accu_x = accu_x.sum(dim=2, keepdim=True)
accu_y = accu_y.sum(dim=2, keepdim=True)
data/.DS_Store
Vendored
BIN
View File
Binary file not shown.
data/AGORA/AGORA.py
-755
View File
@@ -1,755 +0,0 @@
import os
import os.path as osp
from glob import glob
import numpy as np
from config import cfg
import copy
import json
import pickle
import cv2
import torch
from pycocotools.coco import COCO
from utils.human_models import smpl_x
from utils.preprocessing import load_img, sanitize_bbox, process_bbox, augmentation, process_db_coord, \
process_human_model_output, load_ply, load_obj
from utils.transforms import rigid_align
import tqdm
import random
from humandata import Cache
class AGORA(torch.utils.data.Dataset):
def __init__(self, transform, data_split):
self.transform = transform
self.data_split = data_split
if getattr(cfg, 'eval_on_train', False):
self.data_split = 'eval_train'
print("Evaluate on train set.")
self.data_path = osp.join(cfg.data_dir, 'AGORA', 'data')
self.save_idx = 0
self.resolution = (2160, 3840) # height, width. one of (720, 1280) and (2160, 3840)
if cfg.agora_benchmark == 'agora_model_test' or cfg.agora_benchmark == 'test_only':
self.test_set = 'test'
else:
self.test_set = 'val' # val, test
# AGORA joint set
self.joint_set = {
'joint_num': 127,
'joints_name': \
('Pelvis', 'L_Hip', 'R_Hip', 'Spine_1', 'L_Knee', 'R_Knee', 'Spine_2', 'L_Ankle', 'R_Ankle', 'Spine_3',
'L_Foot', 'R_Foot', 'Neck', 'L_Collar', 'R_Collar', 'Head', 'L_Shoulder', 'R_Shoulder', 'L_Elbow',
'R_Elbow', 'L_Wrist', 'R_Wrist', # body
'Jaw', 'L_Eye_SMPLH', 'R_Eye_SMPLH', # SMPLH
'L_Index_1', 'L_Index_2', 'L_Index_3', 'L_Middle_1', 'L_Middle_2', 'L_Middle_3', 'L_Pinky_1',
'L_Pinky_2', 'L_Pinky_3', 'L_Ring_1', 'L_Ring_2', 'L_Ring_3', 'L_Thumb_1', 'L_Thumb_2', 'L_Thumb_3',
# fingers
'R_Index_1', 'R_Index_2', 'R_Index_3', 'R_Middle_1', 'R_Middle_2', 'R_Middle_3', 'R_Pinky_1',
'R_Pinky_2', 'R_Pinky_3', 'R_Ring_1', 'R_Ring_2', 'R_Ring_3', 'R_Thumb_1', 'R_Thumb_2', 'R_Thumb_3',
# fingers
'Nose', 'R_Eye', 'L_Eye', 'R_Ear', 'L_Ear', # face in body
'L_Big_toe', 'L_Small_toe', 'L_Heel', 'R_Big_toe', 'R_Small_toe', 'R_Heel', # feet
'L_Thumb_4', 'L_Index_4', 'L_Middle_4', 'L_Ring_4', 'L_Pinky_4', # finger tips
'R_Thumb_4', 'R_Index_4', 'R_Middle_4', 'R_Ring_4', 'R_Pinky_4', # finger tips
*['Face_' + str(i) for i in range(5, 56)] # face
),
'flip_pairs': \
((1, 2), (4, 5), (7, 8), (10, 11), (13, 14), (16, 17), (18, 19), (20, 21), # body
(23, 24), # SMPLH
(25, 40), (26, 41), (27, 42), (28, 43), (29, 44), (30, 45), (31, 46), (32, 47), (33, 48), (34, 49),
(35, 50), (36, 51), (37, 52), (38, 53), (39, 54), # fingers
(56, 57), (58, 59), # face in body
(60, 63), (61, 64), (62, 65), # feet
(66, 71), (67, 72), (68, 73), (69, 74), (70, 75), # fingertips
(76, 85), (77, 84), (78, 83), (79, 82), (80, 81), # face eyebrow
(90, 94), (91, 93), # face below nose
(95, 104), (96, 103), (97, 102), (98, 101), (99, 106), (100, 105), # face eyes
(107, 113), (108, 112), (109, 111), (114, 118), (115, 117), # face mouth
(119, 123), (120, 122), (124, 126) # face lip
)
}
self.joint_set['joint_part'] = {
'body': list(range(self.joint_set['joints_name'].index('Pelvis'),
self.joint_set['joints_name'].index('R_Eye_SMPLH') + 1)) + list(
range(self.joint_set['joints_name'].index('Nose'), self.joint_set['joints_name'].index('R_Heel') + 1)),
'lhand': list(range(self.joint_set['joints_name'].index('L_Index_1'),
self.joint_set['joints_name'].index('L_Thumb_3') + 1)) + list(
range(self.joint_set['joints_name'].index('L_Thumb_4'),
self.joint_set['joints_name'].index('L_Pinky_4') + 1)),
'rhand': list(range(self.joint_set['joints_name'].index('R_Index_1'),
self.joint_set['joints_name'].index('R_Thumb_3') + 1)) + list(
range(self.joint_set['joints_name'].index('R_Thumb_4'),
self.joint_set['joints_name'].index('R_Pinky_4') + 1)),
'face': list(range(self.joint_set['joints_name'].index('Face_5'),
self.joint_set['joints_name'].index('Face_55') + 1))}
self.joint_set['root_joint_idx'] = self.joint_set['joints_name'].index('Pelvis')
self.joint_set['lwrist_idx'] = self.joint_set['joints_name'].index('L_Wrist')
self.joint_set['rwrist_idx'] = self.joint_set['joints_name'].index('R_Wrist')
self.joint_set['neck_idx'] = self.joint_set['joints_name'].index('Neck')
# self.datalist = self.load_data()
# load data or cache
self.use_cache = getattr(cfg, 'use_cache', False)
if 'train'in self.data_split or (self.data_split == 'test' and self.test_set == 'val'):
if 'train' in self.data_split:
if getattr(cfg, 'agora_fix_betas', False):
assert getattr(cfg, 'agora_fix_global_orient_transl')
self.annot_path_cache = osp.join(cfg.data_dir, 'cache', f'AGORA_{self.data_split}_fix_betas.npz')
elif getattr(cfg, 'agora_fix_global_orient_transl', False):
self.annot_path_cache = osp.join(cfg.data_dir, 'cache', f'AGORA_{self.data_split}_fix_global_orient_transl.npz')
else:
self.annot_path_cache = osp.join(cfg.data_dir, 'cache', f'AGORA_{self.data_split}.npz')
else:
if getattr(cfg, 'agora_fix_betas', False):
assert getattr(cfg, 'agora_fix_global_orient_transl')
self.annot_path_cache = osp.join(cfg.data_dir, 'cache', 'AGORA_validation_fix_betas.npz')
elif getattr(cfg, 'agora_fix_global_orient_transl', False):
self.annot_path_cache = osp.join(cfg.data_dir, 'cache', 'AGORA_validation_fix_global_orient_transl.npz')
else:
self.annot_path_cache = osp.join(cfg.data_dir, 'cache', 'AGORA_validation.npz')
if self.use_cache and osp.isfile(self.annot_path_cache):
print(f'[{self.__class__.__name__}] loading cache from {self.annot_path_cache}')
datalist = Cache(self.annot_path_cache)
assert datalist.data_strategy == getattr(cfg, 'data_strategy', None), \
f'Cache data strategy {datalist.data_strategy} does not match current data strategy ' \
f'{getattr(cfg, "data_strategy", None)}'
self.datalist = datalist
else:
if self.use_cache:
print(f'[{self.__class__.__name__}] Cache not found, generating cache...')
self.datalist = self.load_data()
if self.use_cache:
print(f'[{self.__class__.__name__}] Caching datalist to {self.annot_path_cache}...')
Cache.save(
self.annot_path_cache,
self.datalist,
data_strategy=getattr(cfg, 'data_strategy', None)
)
else: # test
self.datalist = self.load_data()
def load_data(self):
datalist = []
if 'train' in self.data_split or (self.data_split == 'test' and self.test_set == 'val'):
print('dataset settings:')
print('agora_fix_betas', getattr(cfg, 'agora_fix_betas', False))
print('agora_fix_global_orient_transl', getattr(cfg, 'agora_fix_global_orient_transl', False))
print('agora_valid_root_pose', getattr(cfg, 'agora_valid_root_pose', False))
if 'train' in self.data_split:
if getattr(cfg, 'agora_fix_betas', False):
assert getattr(cfg, 'agora_fix_global_orient_transl')
db = COCO(osp.join(self.data_path, 'AGORA_train_fix_betas.json'))
elif getattr(cfg, 'agora_fix_global_orient_transl', False):
db = COCO(osp.join(self.data_path, 'AGORA_train_fix_global_orient_transl.json'))
else:
db = COCO(osp.join(self.data_path, 'AGORA_train.json'))
else:
if getattr(cfg, 'agora_fix_betas', False):
assert getattr(cfg, 'agora_fix_global_orient_transl')
db = COCO(osp.join(self.data_path, 'AGORA_validation_fix_betas.json'))
elif getattr(cfg, 'agora_fix_global_orient_transl', False):
db = COCO(osp.join(self.data_path, 'AGORA_validation_fix_global_orient_transl.json'))
else:
db = COCO(osp.join(self.data_path, 'AGORA_validation.json'))
i = 0
for aid in tqdm.tqdm(list(db.anns.keys())):
i += 1
if self.data_split == 'train' and i % getattr(cfg, 'AGORA_train_sample_interval', 1) != 0:
continue
ann = db.anns[aid]
image_id = ann['image_id']
img = db.loadImgs(image_id)[0]
if not ann['is_valid']:
continue
joints_2d_path = osp.join(self.data_path, ann['smplx_joints_2d_path'])
joints_3d_path = osp.join(self.data_path, ann['smplx_joints_3d_path'])
verts_path = osp.join(self.data_path, ann['smplx_verts_path'])
smplx_param_path = osp.join(self.data_path, ann['smplx_param_path'])
kid = ann['kid']
gender = ann['gender']
if not osp.exists(smplx_param_path): print(smplx_param_path)
if self.resolution == (720, 1280):
img_shape = self.resolution
img_path = osp.join(self.data_path, img['file_name_1280x720'])
# convert to current resolution
bbox = np.array(ann['bbox']).reshape(2, 2)
bbox[:, 0] = bbox[:, 0] / 3840 * 1280
bbox[:, 1] = bbox[:, 1] / 2160 * 720
bbox = bbox.reshape(4)
if hasattr(cfg, 'bbox_ratio'):
bbox_ratio = cfg.bbox_ratio * 0.833 # agora preprocess is giving 1.2 box padding
else:
bbox_ratio = 1.25
bbox = process_bbox(bbox, img_shape[1], img_shape[0], ratio=bbox_ratio)
if bbox is None:
continue
lhand_bbox = np.array(ann['lhand_bbox']).reshape(2, 2)
lhand_bbox[:, 0] = lhand_bbox[:, 0] / 3840 * 1280
lhand_bbox[:, 1] = lhand_bbox[:, 1] / 2160 * 720
lhand_bbox = lhand_bbox.reshape(4)
lhand_bbox = sanitize_bbox(lhand_bbox, img_shape[1], img_shape[0])
if lhand_bbox is not None:
lhand_bbox[2:] += lhand_bbox[:2] # xywh -> xyxy
rhand_bbox = np.array(ann['rhand_bbox']).reshape(2, 2)
rhand_bbox[:, 0] = rhand_bbox[:, 0] / 3840 * 1280
rhand_bbox[:, 1] = rhand_bbox[:, 1] / 2160 * 720
rhand_bbox = rhand_bbox.reshape(4)
rhand_bbox = sanitize_bbox(rhand_bbox, img_shape[1], img_shape[0])
if rhand_bbox is not None:
rhand_bbox[2:] += rhand_bbox[:2] # xywh -> xyxy
face_bbox = np.array(ann['face_bbox']).reshape(2, 2)
face_bbox[:, 0] = face_bbox[:, 0] / 3840 * 1280
face_bbox[:, 1] = face_bbox[:, 1] / 2160 * 720
face_bbox = face_bbox.reshape(4)
face_bbox = sanitize_bbox(face_bbox, img_shape[1], img_shape[0])
if face_bbox is not None:
face_bbox[2:] += face_bbox[:2] # xywh -> xyxy
data_dict = {'img_path': img_path, 'img_shape': img_shape, 'bbox': bbox, 'lhand_bbox': lhand_bbox,
'rhand_bbox': rhand_bbox, 'face_bbox': face_bbox, 'joints_2d_path': joints_2d_path,
'joints_3d_path': joints_3d_path, 'verts_path': verts_path,
'smplx_param_path': smplx_param_path, 'ann_id': str(aid), 'kid': kid, 'gender': gender}
datalist.append(data_dict)
elif self.resolution == (2160,
3840): # use cropped and resized images. loading 4K images in pytorch dataloader takes too much time...
img_path = osp.join(self.data_path, '3840x2160',
img['file_name_3840x2160'].split('/')[-2] + '_crop',
img['file_name_3840x2160'].split('/')[-1][:-4] + '_ann_id_' + str(aid) + '.png')
json_path = osp.join(self.data_path, '3840x2160',
img['file_name_3840x2160'].split('/')[-2] + '_crop',
img['file_name_3840x2160'].split('/')[-1][:-4] + '_ann_id_' + str(
aid) + '.json')
if not osp.isfile(json_path):
continue
with open(json_path) as f:
crop_resize_info = json.load(f)
img2bb_trans_from_orig = np.array(crop_resize_info['img2bb_trans'], dtype=np.float32)
resized_height, resized_width = crop_resize_info['resized_height'], crop_resize_info[
'resized_width']
img_shape = (resized_height, resized_width)
bbox = np.array([0, 0, resized_width, resized_height], dtype=np.float32)
# transform from original image to crop_and_resize image
lhand_bbox = np.array(ann['lhand_bbox']).reshape(2, 2)
lhand_bbox[1] += lhand_bbox[0] # xywh -> xyxy
lhand_bbox = np.dot(img2bb_trans_from_orig,
np.concatenate((lhand_bbox, np.ones_like(lhand_bbox[:, :1])), 1).transpose(1,
0)).transpose(
1, 0)
lhand_bbox[1] -= lhand_bbox[0] # xyxy -> xywh
lhand_bbox = lhand_bbox.reshape(4)
lhand_bbox = sanitize_bbox(lhand_bbox, self.resolution[1], self.resolution[0])
if lhand_bbox is not None:
lhand_bbox[2:] += lhand_bbox[:2] # xywh -> xyxy
# transform from original image to crop_and_resize image
rhand_bbox = np.array(ann['rhand_bbox']).reshape(2, 2)
rhand_bbox[1] += rhand_bbox[0] # xywh -> xyxy
rhand_bbox = np.dot(img2bb_trans_from_orig,
np.concatenate((rhand_bbox, np.ones_like(rhand_bbox[:, :1])), 1).transpose(1,
0)).transpose(
1, 0)
rhand_bbox[1] -= rhand_bbox[0] # xyxy -> xywh
rhand_bbox = rhand_bbox.reshape(4)
rhand_bbox = sanitize_bbox(rhand_bbox, self.resolution[1], self.resolution[0])
if rhand_bbox is not None:
rhand_bbox[2:] += rhand_bbox[:2] # xywh -> xyxy
# transform from original image to crop_and_resize image
face_bbox = np.array(ann['face_bbox']).reshape(2, 2)
face_bbox[1] += face_bbox[0] # xywh -> xyxy
face_bbox = np.dot(img2bb_trans_from_orig,
np.concatenate((face_bbox, np.ones_like(face_bbox[:, :1])), 1).transpose(1,
0)).transpose(
1, 0)
face_bbox[1] -= face_bbox[0] # xyxy -> xywh
face_bbox = face_bbox.reshape(4)
face_bbox = sanitize_bbox(face_bbox, self.resolution[1], self.resolution[0])
if face_bbox is not None:
face_bbox[2:] += face_bbox[:2] # xywh -> xyxy
data_dict = {'img_path': img_path, 'img_shape': img_shape, 'bbox': bbox, 'lhand_bbox': lhand_bbox,
'rhand_bbox': rhand_bbox, 'face_bbox': face_bbox,
'img2bb_trans_from_orig': img2bb_trans_from_orig, 'joints_2d_path': joints_2d_path,
'joints_3d_path': joints_3d_path, 'verts_path': verts_path,
'smplx_param_path': smplx_param_path, 'ann_id': str(aid), 'kid': kid, 'gender': gender}
datalist.append(data_dict)
print('[AGORA train] original size:', len(db.anns.keys()),
'. Sample interval:', getattr(cfg, 'AGORA_train_sample_interval', 1),
'. Sampled size:', len(datalist))
elif self.data_split == 'test' and self.test_set == 'test':
with open(osp.join(self.data_path, 'AGORA_test_bbox.json')) as f:
bboxs = json.load(f)
for filename in tqdm.tqdm(bboxs.keys()):
if self.resolution == (720, 1280):
img_path = osp.join(self.data_path, 'test', filename)
img_shape = self.resolution
person_num = len(bboxs[filename])
for pid in range(person_num):
# change bbox from (2160,3840) to target resoution
bbox = np.array(bboxs[filename][pid]['bbox']).reshape(2, 2)
bbox[:, 0] = bbox[:, 0] / 3840 * 1280
bbox[:, 1] = bbox[:, 1] / 2160 * 720
bbox = bbox.reshape(4)
if hasattr(cfg, 'bbox_ratio'):
bbox_ratio = cfg.bbox_ratio * 0.833 # agora preprocess is giving 1.2 box padding
else:
bbox_ratio = 1.25
bbox = process_bbox(bbox, img_shape[1], img_shape[0], ratio=bbox_ratio)
if bbox is None:
continue
datalist.append({'img_path': img_path, 'img_shape': img_shape, 'bbox': bbox, 'person_idx': pid})
elif self.resolution == (2160,
3840): # use cropped and resized images. loading 4K images in pytorch dataloader takes too much time...
person_num = len(bboxs[filename])
for pid in range(person_num):
img_path = osp.join(self.data_path, '3840x2160', 'test_crop',
filename[:-4] + '_pid_' + str(pid) + '.png')
json_path = osp.join(self.data_path, '3840x2160', 'test_crop',
filename[:-4] + '_pid_' + str(pid) + '.json')
if not osp.isfile(json_path):
continue
with open(json_path) as f:
crop_resize_info = json.load(f)
img2bb_trans_from_orig = np.array(crop_resize_info['img2bb_trans'], dtype=np.float32)
resized_height, resized_width = crop_resize_info['resized_height'], crop_resize_info[
'resized_width']
img_shape = (resized_height, resized_width)
bbox = np.array([0, 0, resized_width, resized_height], dtype=np.float32)
datalist.append({'img_path': img_path, 'img_shape': img_shape,
'img2bb_trans_from_orig': img2bb_trans_from_orig, 'bbox': bbox,
'person_idx': pid})
if (getattr(cfg, 'data_strategy', None) == 'balance' and self.data_split == 'train') or \
self.data_split == 'eval_train':
print(f"[Agora] Using [balance] strategy with datalist shuffled...")
random.seed(2023)
random.shuffle(datalist)
if self.data_split == "eval_train":
return datalist[:10000]
return datalist
def process_hand_face_bbox(self, bbox, do_flip, img_shape, img2bb_trans):
if bbox is None:
bbox = np.array([0, 0, 1, 1], dtype=np.float32).reshape(2, 2) # dummy value
bbox_valid = float(False) # dummy value
else:
# reshape to top-left (x,y) and bottom-right (x,y)
bbox = bbox.reshape(2, 2)
# flip augmentation
if do_flip:
bbox[:, 0] = img_shape[1] - bbox[:, 0] - 1
bbox[0, 0], bbox[1, 0] = bbox[1, 0].copy(), bbox[0, 0].copy() # xmin <-> xmax swap
# make four points of the bbox
bbox = bbox.reshape(4).tolist()
xmin, ymin, xmax, ymax = bbox
bbox = np.array([[xmin, ymin], [xmax, ymin], [xmax, ymax], [xmin, ymax]], dtype=np.float32).reshape(4, 2)
# affine transformation (crop, rotation, scale)
bbox_xy1 = np.concatenate((bbox, np.ones_like(bbox[:, :1])), 1)
bbox = np.dot(img2bb_trans, bbox_xy1.transpose(1, 0)).transpose(1, 0)[:, :2]
bbox[:, 0] = bbox[:, 0] / cfg.input_img_shape[1] * cfg.output_hm_shape[2]
bbox[:, 1] = bbox[:, 1] / cfg.input_img_shape[0] * cfg.output_hm_shape[1]
# make box a rectangle without rotation
xmin = np.min(bbox[:, 0]);
xmax = np.max(bbox[:, 0]);
ymin = np.min(bbox[:, 1]);
ymax = np.max(bbox[:, 1]);
bbox = np.array([xmin, ymin, xmax, ymax], dtype=np.float32)
bbox_valid = float(True)
bbox = bbox.reshape(2, 2)
return bbox, bbox_valid
def __len__(self):
return len(self.datalist)
def __getitem__(self, idx):
data = copy.deepcopy(self.datalist[idx])
img_path, img_shape, bbox = data['img_path'], data['img_shape'], data['bbox']
# image load
img = load_img(img_path)
# affine transform
img, img2bb_trans, bb2img_trans, rot, do_flip = augmentation(img, bbox, self.data_split)
img = self.transform(img.astype(np.float32)) / 255.
if self.data_split == 'train':
# gt load
with open(data['joints_2d_path']) as f:
joint_img = np.array(json.load(f)).reshape(-1, 2)
if self.resolution == (2160, 3840):
joint_img[:, :2] = np.dot(data['img2bb_trans_from_orig'],
np.concatenate((joint_img, np.ones_like(joint_img[:, :1])), 1).transpose(
1, 0)).transpose(1,
0) # transform from original image to crop_and_resize image
joint_img[:, 0] = joint_img[:, 0] / 3840 * self.resolution[1]
joint_img[:, 1] = joint_img[:, 1] / 2160 * self.resolution[0]
with open(data['joints_3d_path']) as f:
joint_cam = np.array(json.load(f)).reshape(-1, 3)
with open(data['smplx_param_path'], 'rb') as f:
smplx_param = pickle.load(f, encoding='latin1')
# hand and face bbox transform
lhand_bbox, rhand_bbox, face_bbox = data['lhand_bbox'], data['rhand_bbox'], data['face_bbox']
lhand_bbox, lhand_bbox_valid = self.process_hand_face_bbox(lhand_bbox, do_flip, img_shape, img2bb_trans)
rhand_bbox, rhand_bbox_valid = self.process_hand_face_bbox(rhand_bbox, do_flip, img_shape, img2bb_trans)
face_bbox, face_bbox_valid = self.process_hand_face_bbox(face_bbox, do_flip, img_shape, img2bb_trans)
if do_flip:
lhand_bbox, rhand_bbox = rhand_bbox, lhand_bbox
lhand_bbox_valid, rhand_bbox_valid = rhand_bbox_valid, lhand_bbox_valid
lhand_bbox_center = (lhand_bbox[0] + lhand_bbox[1]) / 2.;
rhand_bbox_center = (rhand_bbox[0] + rhand_bbox[1]) / 2.;
face_bbox_center = (face_bbox[0] + face_bbox[1]) / 2.
lhand_bbox_size = lhand_bbox[1] - lhand_bbox[0];
rhand_bbox_size = rhand_bbox[1] - rhand_bbox[0];
face_bbox_size = face_bbox[1] - face_bbox[0];
# coordinates
joint_cam = joint_cam - joint_cam[self.joint_set['root_joint_idx'], None, :] # root-relative
joint_cam[self.joint_set['joint_part']['lhand'], :] = joint_cam[self.joint_set['joint_part']['lhand'],
:] - joint_cam[self.joint_set['lwrist_idx'], None,
:] # left hand root-relative
joint_cam[self.joint_set['joint_part']['rhand'], :] = joint_cam[self.joint_set['joint_part']['rhand'],
:] - joint_cam[self.joint_set['rwrist_idx'], None,
:] # right hand root-relative
joint_cam[self.joint_set['joint_part']['face'], :] = joint_cam[self.joint_set['joint_part']['face'],
:] - joint_cam[self.joint_set['neck_idx'], None,
:] # face root-relative
joint_img = np.concatenate((joint_img[:, :2], joint_cam[:, 2:]), 1) # x, y, depth
joint_img[self.joint_set['joint_part']['body'], 2] = (joint_cam[self.joint_set['joint_part'][
'body'], 2].copy() / (
cfg.body_3d_size / 2) + 1) / 2. * \
cfg.output_hm_shape[0] # body depth discretize
joint_img[self.joint_set['joint_part']['lhand'], 2] = (joint_cam[self.joint_set['joint_part'][
'lhand'], 2].copy() / (
cfg.hand_3d_size / 2) + 1) / 2. * \
cfg.output_hm_shape[0] # left hand depth discretize
joint_img[self.joint_set['joint_part']['rhand'], 2] = (joint_cam[self.joint_set['joint_part'][
'rhand'], 2].copy() / (
cfg.hand_3d_size / 2) + 1) / 2. * \
cfg.output_hm_shape[0] # right hand depth discretize
joint_img[self.joint_set['joint_part']['face'], 2] = (joint_cam[self.joint_set['joint_part'][
'face'], 2].copy() / (
cfg.face_3d_size / 2) + 1) / 2. * \
cfg.output_hm_shape[0] # face depth discretize
joint_valid = np.ones_like(joint_img[:, :1])
# alr ra when passed into this function
joint_img, joint_cam_ra, _, joint_valid, joint_trunc = process_db_coord(joint_img, joint_cam, joint_valid,
do_flip, img_shape,
self.joint_set['flip_pairs'],
img2bb_trans, rot,
self.joint_set['joints_name'],
smpl_x.joints_name)
# reverse ra
joint_cam_wo_ra = joint_cam_ra.copy()
joint_cam_wo_ra[smpl_x.joint_part['lhand'], :] = joint_cam_wo_ra[smpl_x.joint_part['lhand'], :] \
+ joint_cam_wo_ra[smpl_x.lwrist_idx, None, :] # left hand root-relative
joint_cam_wo_ra[smpl_x.joint_part['rhand'], :] = joint_cam_wo_ra[smpl_x.joint_part['rhand'], :] \
+ joint_cam_wo_ra[smpl_x.rwrist_idx, None, :] # right hand root-relative
joint_cam_wo_ra[smpl_x.joint_part['face'], :] = joint_cam_wo_ra[smpl_x.joint_part['face'], :] \
+ joint_cam_wo_ra[smpl_x.neck_idx, None,: ] # face root-relative
# smplx parameters
root_pose = np.array(smplx_param['global_orient'], dtype=np.float32).reshape(
-1) # rotation to world coordinate
body_pose = np.array(smplx_param['body_pose'], dtype=np.float32).reshape(-1)
# use adapted shape for adults
if getattr(cfg, 'agora_fix_betas', False) and not data['kid']:
shape = np.array(smplx_param['betas_neutral'], dtype=np.float32).reshape(-1)[:10]
else:
shape = np.array(smplx_param['betas'], dtype=np.float32).reshape(-1)[:10] # bug?
lhand_pose = np.array(smplx_param['left_hand_pose'], dtype=np.float32).reshape(-1)
rhand_pose = np.array(smplx_param['right_hand_pose'], dtype=np.float32).reshape(-1)
jaw_pose = np.array(smplx_param['jaw_pose'], dtype=np.float32).reshape(-1)
expr = np.array(smplx_param['expression'], dtype=np.float32).reshape(-1)
trans = np.array(smplx_param['transl'], dtype=np.float32).reshape(-1) # translation to world coordinate
cam_param = {'focal': cfg.focal,
'princpt': cfg.princpt} # put random camera paraemter as we do not use coordinates from smplx parameters
smplx_param = {'root_pose': root_pose, 'body_pose': body_pose, 'shape': shape,
'lhand_pose': lhand_pose, 'lhand_valid': True,
'rhand_pose': rhand_pose, 'rhand_valid': True,
'jaw_pose': jaw_pose, 'expr': expr, 'face_valid': True,
'trans': trans}
_, _, _, smplx_pose, smplx_shape, smplx_expr, smplx_pose_valid, _, smplx_expr_valid, _ = process_human_model_output(
smplx_param, cam_param, do_flip, img_shape, img2bb_trans, rot, 'smplx')
smplx_pose_valid = np.tile(smplx_pose_valid[:, None], (1, 3)).reshape(-1)
if not getattr(cfg, 'agora_valid_root_pose', False):
smplx_pose_valid[:3] = 0 # global orient of the provided parameter is a rotation to world coordinate system. I want camera coordinate system.
smplx_shape_valid = True
inputs = {'img': img}
targets = {'joint_img': joint_img, 'joint_cam': joint_cam_wo_ra, #from annot
'smplx_joint_img': joint_img, 'smplx_joint_cam': joint_cam_ra, #_smplx_joint_cam, # from smplx param w/ ra
'smplx_pose': smplx_pose, 'smplx_shape': smplx_shape, 'smplx_expr': smplx_expr,
'lhand_bbox_center': lhand_bbox_center, 'lhand_bbox_size': lhand_bbox_size,
'rhand_bbox_center': rhand_bbox_center, 'rhand_bbox_size': rhand_bbox_size,
'face_bbox_center': face_bbox_center, 'face_bbox_size': face_bbox_size}
meta_info = {'joint_valid': joint_valid, 'joint_trunc': joint_trunc,
'smplx_joint_valid': joint_valid, 'smplx_joint_trunc': joint_trunc,
'smplx_pose_valid': smplx_pose_valid, 'smplx_shape_valid': float(smplx_shape_valid),
'smplx_expr_valid': float(smplx_expr_valid), 'is_3D': float(True),
'lhand_bbox_valid': lhand_bbox_valid, 'rhand_bbox_valid': rhand_bbox_valid, 'face_bbox_valid': face_bbox_valid}
return inputs, targets, meta_info
else:
# load crop and resize information (for the 4K setting)
if self.resolution == (2160, 3840):
img2bb_trans = np.dot(
np.concatenate((img2bb_trans,
np.array([0, 0, 1], dtype=np.float32).reshape(1, 3))),
np.concatenate((data['img2bb_trans_from_orig'],
np.array([0, 0, 1], dtype=np.float32).reshape(1, 3)))
)
bb2img_trans = np.linalg.inv(img2bb_trans)[:2, :]
img2bb_trans = img2bb_trans[:2, :]
if self.test_set == 'val':
# gt load
with open(data['verts_path']) as f:
verts = np.array(json.load(f)).reshape(-1, 3)
with open(data['smplx_param_path'], 'rb') as f:
smplx_param = pickle.load(f, encoding='latin1')
transl = np.array(smplx_param['transl'], dtype=np.float32).reshape(-1)
inputs = {'img': img}
targets = {'smplx_mesh_cam': verts}
meta_info = {'bb2img_trans': bb2img_trans, 'img_path': img_path, 'gt_smplx_transl':transl}
else:
inputs = {'img': img}
targets = {'smplx_mesh_cam': np.zeros((smpl_x.vertex_num, 3), dtype=np.float32)} # dummy vertex
meta_info = {'bb2img_trans': bb2img_trans, 'img_path': img_path}
return inputs, targets, meta_info
def evaluate(self, outs, cur_sample_idx):
annots = self.datalist
sample_num = len(outs)
eval_result = {'pa_mpvpe_all': [], 'pa_mpvpe_l_hand': [], 'pa_mpvpe_r_hand': [], 'pa_mpvpe_hand': [], 'pa_mpvpe_face': [],
'mpvpe_all': [], 'mpvpe_l_hand': [], 'mpvpe_r_hand': [], 'mpvpe_hand': [], 'mpvpe_face': []}
vis = getattr(cfg, 'vis', False)
vis_save_dir = cfg.vis_dir
if getattr(cfg, 'vis', False):
import csv
csv_file = f'{cfg.vis_dir}/agora_smplx_error.csv'
file = open(csv_file, 'a', newline='')
writer = csv.writer(file)
for n in range(sample_num):
annot = annots[cur_sample_idx + n]
out = outs[n]
mesh_gt = out['smplx_mesh_cam_target']
mesh_out = out['smplx_mesh_cam']
# MPVPE from all vertices
mesh_out_align = mesh_out - np.dot(smpl_x.J_regressor, mesh_out)[smpl_x.J_regressor_idx['pelvis'], None,
:] + np.dot(smpl_x.J_regressor, mesh_gt)[smpl_x.J_regressor_idx['pelvis'], None,
:]
mpvpe_all = np.sqrt(np.sum((mesh_out_align - mesh_gt) ** 2, 1)).mean() * 1000
eval_result['mpvpe_all'].append(mpvpe_all)
mesh_out_align = rigid_align(mesh_out, mesh_gt)
pa_mpvpe_all = np.sqrt(np.sum((mesh_out_align - mesh_gt) ** 2, 1)).mean() * 1000
eval_result['pa_mpvpe_all'].append(pa_mpvpe_all)
# MPVPE from hand vertices
mesh_gt_lhand = mesh_gt[smpl_x.hand_vertex_idx['left_hand'], :]
mesh_out_lhand = mesh_out[smpl_x.hand_vertex_idx['left_hand'], :]
mesh_gt_rhand = mesh_gt[smpl_x.hand_vertex_idx['right_hand'], :]
mesh_out_rhand = mesh_out[smpl_x.hand_vertex_idx['right_hand'], :]
mesh_out_lhand_align = mesh_out_lhand - np.dot(smpl_x.J_regressor, mesh_out)[
smpl_x.J_regressor_idx['lwrist'], None, :] + np.dot(
smpl_x.J_regressor, mesh_gt)[smpl_x.J_regressor_idx['lwrist'], None, :]
mesh_out_rhand_align = mesh_out_rhand - np.dot(smpl_x.J_regressor, mesh_out)[
smpl_x.J_regressor_idx['rwrist'], None, :] + np.dot(
smpl_x.J_regressor, mesh_gt)[smpl_x.J_regressor_idx['rwrist'], None, :]
eval_result['mpvpe_l_hand'].append(np.sqrt(
np.sum((mesh_out_lhand_align - mesh_gt_lhand) ** 2, 1)).mean() * 1000)
eval_result['mpvpe_r_hand'].append(np.sqrt(
np.sum((mesh_out_rhand_align - mesh_gt_rhand) ** 2, 1)).mean() * 1000)
eval_result['mpvpe_hand'].append((np.sqrt(
np.sum((mesh_out_lhand_align - mesh_gt_lhand) ** 2, 1)).mean() * 1000 + np.sqrt(
np.sum((mesh_out_rhand_align - mesh_gt_rhand) ** 2, 1)).mean() * 1000) / 2.)
mesh_out_lhand_align = rigid_align(mesh_out_lhand, mesh_gt_lhand)
mesh_out_rhand_align = rigid_align(mesh_out_rhand, mesh_gt_rhand)
eval_result['pa_mpvpe_l_hand'].append(np.sqrt(
np.sum((mesh_out_lhand_align - mesh_gt_lhand) ** 2, 1)).mean() * 1000)
eval_result['pa_mpvpe_r_hand'].append(np.sqrt(
np.sum((mesh_out_rhand_align - mesh_gt_rhand) ** 2, 1)).mean() * 1000)
eval_result['pa_mpvpe_hand'].append((np.sqrt(
np.sum((mesh_out_lhand_align - mesh_gt_lhand) ** 2, 1)).mean() * 1000 + np.sqrt(
np.sum((mesh_out_rhand_align - mesh_gt_rhand) ** 2, 1)).mean() * 1000) / 2.)
# MPVPE from face vertices
mesh_gt_face = mesh_gt[smpl_x.face_vertex_idx, :]
mesh_out_face = mesh_out[smpl_x.face_vertex_idx, :]
mesh_out_face_align = mesh_out_face - np.dot(smpl_x.J_regressor, mesh_out)[smpl_x.J_regressor_idx['neck'],
None, :] + np.dot(smpl_x.J_regressor, mesh_gt)[
smpl_x.J_regressor_idx['neck'], None, :]
eval_result['mpvpe_face'].append(
np.sqrt(np.sum((mesh_out_face_align - mesh_gt_face) ** 2, 1)).mean() * 1000)
mesh_out_face_align = rigid_align(mesh_out_face, mesh_gt_face)
eval_result['pa_mpvpe_face'].append(
np.sqrt(np.sum((mesh_out_face_align - mesh_gt_face) ** 2, 1)).mean() * 1000)
if vis:
img_path = out['img_path']
rel_img_path = img_path.split('..')[-1]
smplx_pred = {}
smplx_pred['global_orient'] = out['smplx_root_pose'].reshape(-1,3)
smplx_pred['body_pose'] = out['smplx_body_pose'].reshape(-1,3)
smplx_pred['left_hand_pose'] = out['smplx_lhand_pose'].reshape(-1,3)
smplx_pred['right_hand_pose'] = out['smplx_rhand_pose'].reshape(-1,3)
smplx_pred['jaw_pose'] = out['smplx_jaw_pose'].reshape(-1,3)
smplx_pred['leye_pose'] = np.zeros((1, 3))
smplx_pred['reye_pose'] = np.zeros((1, 3))
smplx_pred['betas'] = out['smplx_shape'].reshape(-1,10)
smplx_pred['expression'] = out['smplx_expr'].reshape(-1,10)
smplx_pred['transl'] = out['gt_smplx_transl'].reshape(-1,3)
smplx_pred['img_path'] = rel_img_path
npz_path = os.path.join(cfg.vis_dir, f'{self.save_idx}.npz')
np.savez(npz_path, **smplx_pred)
# save img path and error
new_line = [self.save_idx, rel_img_path, mpvpe_all, pa_mpvpe_all]
# Append the new line to the CSV file
writer.writerow(new_line)
self.save_idx += 1
# save_obj(out['smplx_mesh_cam'], smpl_x.face, str(cur_sample_idx + n) + '.obj')
# save results for the official evaluation codes/server
save_name = annot['img_path'].split('/')[-1][:-4]
if self.data_split == 'test' and self.test_set == 'test':
if self.resolution == (2160, 3840):
save_name = save_name.split('_pid')[0]
elif self.data_split == 'test' and self.test_set == 'val':
if self.resolution == (2160, 3840):
save_name = save_name.split('_ann_id')[0]
else:
save_name = save_name.split('_1280x720')[0]
if 'person_idx' in annot:
person_idx = annot['person_idx']
else:
exist_result_path = glob(osp.join(cfg.result_dir, 'AGORA', save_name + '*'))
if len(exist_result_path) == 0:
person_idx = 0
else:
last_person_idx = max(
[int(name.split('personId_')[1].split('.pkl')[0]) for name in exist_result_path])
person_idx = last_person_idx + 1
save_name += '_personId_' + str(person_idx) + '.pkl'
joint_proj = out['smplx_joint_proj']
joint_proj[:, 0] = joint_proj[:, 0] / cfg.output_hm_shape[2] * cfg.input_img_shape[1]
joint_proj[:, 1] = joint_proj[:, 1] / cfg.output_hm_shape[1] * cfg.input_img_shape[0]
joint_proj = np.concatenate((joint_proj, np.ones_like(joint_proj[:, :1])), 1)
joint_proj = np.dot(out['bb2img_trans'], joint_proj.transpose(1, 0)).transpose(1, 0)
joint_proj[:, 0] = joint_proj[:, 0] / self.resolution[1] * 3840 # restore to original resolution
joint_proj[:, 1] = joint_proj[:, 1] / self.resolution[0] * 2160 # restore to original resolution
save_dict = {'params':
{'transl': out['cam_trans'].reshape(1, -1),
'global_orient': out['smplx_root_pose'].reshape(1, -1),
'body_pose': out['smplx_body_pose'].reshape(1, -1),
'left_hand_pose': out['smplx_lhand_pose'].reshape(1, -1),
'right_hand_pose': out['smplx_rhand_pose'].reshape(1, -1),
'reye_pose': np.zeros((1, 3)),
'leye_pose': np.zeros((1, 3)),
'jaw_pose': out['smplx_jaw_pose'].reshape(1, -1),
'expression': out['smplx_expr'].reshape(1, -1),
'betas': out['smplx_shape'].reshape(1, -1)},
'joints': joint_proj.reshape(1, -1, 2)
}
os.makedirs(osp.join(cfg.result_dir, 'predictions'), exist_ok=True)
with open(osp.join(cfg.result_dir, 'predictions', save_name), 'wb') as f:
pickle.dump(save_dict, f)
if getattr(cfg, 'vis', False):
file.close()
return eval_result
def print_eval_result(self, eval_result):
print('AGORA test results are dumped at: ' + osp.join(cfg.result_dir, 'predictions'))
if self.data_split == 'test' and self.test_set == 'test': # do not print. just submit the results to the official evaluation server
return
print('======AGORA-val======')
print(f'{cfg.vis_dir}')
print('PA MPVPE (All): %.2f mm' % np.mean(eval_result['pa_mpvpe_all']))
print('PA MPVPE (L-Hands): %.2f mm' % np.mean(eval_result['pa_mpvpe_l_hand']))
print('PA MPVPE (R-Hands): %.2f mm' % np.mean(eval_result['pa_mpvpe_r_hand']))
print('PA MPVPE (Hands): %.2f mm' % np.mean(eval_result['pa_mpvpe_hand']))
print('PA MPVPE (Face): %.2f mm' % np.mean(eval_result['pa_mpvpe_face']))
print()
print('MPVPE (All): %.2f mm' % np.mean(eval_result['mpvpe_all']))
print('MPVPE (L-Hands): %.2f mm' % np.mean(eval_result['mpvpe_l_hand']))
print('MPVPE (R-Hands): %.2f mm' % np.mean(eval_result['mpvpe_r_hand']))
print('MPVPE (Hands): %.2f mm' % np.mean(eval_result['mpvpe_hand']))
print('MPVPE (Face): %.2f mm' % np.mean(eval_result['mpvpe_face']))
print()
print(f"{np.mean(eval_result['pa_mpvpe_all'])},{np.mean(eval_result['pa_mpvpe_l_hand'])},{np.mean(eval_result['pa_mpvpe_r_hand'])},{np.mean(eval_result['pa_mpvpe_hand'])},{np.mean(eval_result['pa_mpvpe_face'])},"
f"{np.mean(eval_result['mpvpe_all'])},{np.mean(eval_result['mpvpe_l_hand'])},{np.mean(eval_result['mpvpe_r_hand'])},{np.mean(eval_result['mpvpe_hand'])},{np.mean(eval_result['mpvpe_face'])}")
print()
f = open(os.path.join(cfg.result_dir, 'result.txt'), 'w')
f.write(f'AGORA-val dataset: \n')
f.write('PA MPVPE (All): %.2f mm\n' % np.mean(eval_result['pa_mpvpe_all']))
f.write('PA MPVPE (L-Hands): %.2f mm' % np.mean(eval_result['pa_mpvpe_l_hand']))
f.write('PA MPVPE (R-Hands): %.2f mm' % np.mean(eval_result['pa_mpvpe_r_hand']))
f.write('PA MPVPE (Hands): %.2f mm\n' % np.mean(eval_result['pa_mpvpe_hand']))
f.write('PA MPVPE (Face): %.2f mm\n' % np.mean(eval_result['pa_mpvpe_face']))
f.write('MPVPE (All): %.2f mm\n' % np.mean(eval_result['mpvpe_all']))
f.write('MPVPE (L-Hands): %.2f mm' % np.mean(eval_result['mpvpe_l_hand']))
f.write('MPVPE (R-Hands): %.2f mm' % np.mean(eval_result['mpvpe_r_hand']))
f.write('MPVPE (Hands): %.2f mm' % np.mean(eval_result['mpvpe_hand']))
f.write('MPVPE (Face): %.2f mm\n' % np.mean(eval_result['mpvpe_face']))
f.write(f"{np.mean(eval_result['pa_mpvpe_all'])},{np.mean(eval_result['pa_mpvpe_l_hand'])},{np.mean(eval_result['pa_mpvpe_r_hand'])},{np.mean(eval_result['pa_mpvpe_hand'])},{np.mean(eval_result['pa_mpvpe_face'])},"
f"{np.mean(eval_result['mpvpe_all'])},{np.mean(eval_result['mpvpe_l_hand'])},{np.mean(eval_result['mpvpe_r_hand'])},{np.mean(eval_result['mpvpe_hand'])},{np.mean(eval_result['mpvpe_face'])}")
if getattr(cfg, 'eval_on_train', False):
import csv
csv_file = f'{cfg.root_dir}/output/agora_eval_on_train.csv'
exp_id = cfg.exp_name.split('_')[1]
new_line = [exp_id,np.mean(eval_result['pa_mpvpe_all']),np.mean(eval_result['pa_mpvpe_l_hand']),np.mean(eval_result['pa_mpvpe_r_hand']),np.mean(eval_result['pa_mpvpe_hand']),np.mean(eval_result['pa_mpvpe_face']),
np.mean(eval_result['mpvpe_all']),np.mean(eval_result['mpvpe_l_hand']),np.mean(eval_result['mpvpe_r_hand']),np.mean(eval_result['mpvpe_hand']),np.mean(eval_result['mpvpe_face'])]
# Append the new line to the CSV file
with open(csv_file, 'a', newline='') as file:
writer = csv.writer(file)
writer.writerow(new_line)
data/ARCTIC/ARCTIC.py
-51
View File
@@ -1,51 +0,0 @@
import os
import os.path as osp
import numpy as np
import torch
import cv2
import json
import copy
from pycocotools.coco import COCO
from config import cfg
from utils.human_models import smpl_x
from utils.preprocessing import load_img, process_bbox, augmentation, process_db_coord, process_human_model_output, \
get_fitting_error_3D
from utils.transforms import world2cam, cam2pixel, rigid_align
from humandata import HumanDataset
class ARCTIC(HumanDataset):
def __init__(self, transform, data_split):
super(ARCTIC, self).__init__(transform, data_split)
if getattr(cfg, 'eval_on_train', False):
self.data_split = 'eval_train'
print("Evaluate on train set.")
self.use_cache = getattr(cfg, 'use_cache', False)
self.annot_path_cache = osp.join(cfg.data_dir, 'cache', f'arctic_{self.data_split}.npz')
self.img_shape = None # (h, w)
self.cam_param = {}
if self.use_cache and osp.isfile(self.annot_path_cache):
print(f'[{self.__class__.__name__}] loading cache from {self.annot_path_cache}')
self.datalist = self.load_cache(self.annot_path_cache)
else:
if self.use_cache:
print(f'[{self.__class__.__name__}] Cache not found, generating cache...')
if 'train' in data_split:
filename = getattr(cfg, 'filename', 'p1_train.npz')
else:
filename = getattr(cfg, 'filename', 'p1_val.npz')
self.img_dir = osp.join(cfg.data_dir, 'ARCTIC')
self.annot_path = osp.join(cfg.data_dir, 'preprocessed_datasets', filename)
# load data
self.datalist = self.load_data(
train_sample_interval=getattr(cfg, f'{self.__class__.__name__}_train_sample_interval', 1),
test_sample_interval=getattr(cfg, f'{self.__class__.__name__}_test_sample_interval', 10))
if self.use_cache:
self.save_cache(self.annot_path_cache, self.datalist)
data/BEDLAM/BEDLAM.py
-45
View File
@@ -1,45 +0,0 @@
import os
import os.path as osp
import numpy as np
import torch
import cv2
import json
import copy
from pycocotools.coco import COCO
from config import cfg
from utils.human_models import smpl_x
from utils.preprocessing import load_img, process_bbox, augmentation, process_db_coord, process_human_model_output, \
get_fitting_error_3D
from utils.transforms import world2cam, cam2pixel, rigid_align
from humandata import HumanDataset
class BEDLAM(HumanDataset):
def __init__(self, transform, data_split):
super(BEDLAM, self).__init__(transform, data_split)
self.use_cache = getattr(cfg, 'use_cache', False)
self.annot_path_cache = osp.join(cfg.data_dir, 'cache', 'bedlam_train.npz')
self.img_shape = None # (h, w)
self.cam_param = {}
if self.use_cache and osp.isfile(self.annot_path_cache):
print(f'[{self.__class__.__name__}] loading cache from {self.annot_path_cache}')
self.datalist = self.load_cache(self.annot_path_cache)
else:
if self.use_cache:
print(f'[{self.__class__.__name__}] Cache not found, generating cache...')
if self.data_split == 'train':
filename = getattr(cfg, 'filename', 'bedlam_train.npz')
else:
raise ValueError('BEDLAM test set is not support')
self.img_dir = osp.join(cfg.data_dir, 'BEDLAM')
self.annot_path = osp.join(cfg.data_dir, 'preprocessed_datasets', filename)
# load data
self.datalist = self.load_data(
train_sample_interval=getattr(cfg, f'{self.__class__.__name__}_train_sample_interval', 1))
if self.use_cache:
self.save_cache(self.annot_path_cache, self.datalist)
data/Behave/Behave.py
-50
View File
@@ -1,50 +0,0 @@
import os
import os.path as osp
import numpy as np
import torch
import cv2
import json
import copy
from pycocotools.coco import COCO
from config import cfg
from utils.human_models import smpl_x
from utils.preprocessing import load_img, process_bbox, augmentation, process_db_coord, process_human_model_output, \
get_fitting_error_3D
from utils.transforms import world2cam, cam2pixel, rigid_align
from humandata import HumanDataset
class Behave(HumanDataset):
def __init__(self, transform, data_split):
super(Behave, self).__init__(transform, data_split)
pre_prc_file_train = 'behave_train_230516_231_downsampled.npz'
pre_prc_file_test = 'behave_test_230516_090_downsampled.npz'
if self.data_split == 'train':
filename = getattr(cfg, 'filename', pre_prc_file_train)
else:
filename = getattr(cfg, 'filename', pre_prc_file_test)
self.img_dir = osp.join(cfg.data_dir, 'Behave')
self.annot_path = osp.join(cfg.data_dir, 'preprocessed_datasets', filename)
self.annot_path_cache = osp.join(cfg.data_dir, 'cache', filename)
self.use_cache = getattr(cfg, 'use_cache', False)
self.img_shape = (1536, 2048) # (h, w)
self.cam_param = {}
# check image shape
img_path = osp.join(self.img_dir, np.load(self.annot_path)['image_path'][0])
img_shape = cv2.imread(img_path).shape[:2]
assert self.img_shape == img_shape, 'image shape is incorrect: {} vs {}'.format(self.img_shape, img_shape)
# load data or cache
if self.use_cache and osp.isfile(self.annot_path_cache):
print(f'[{self.__class__.__name__}] loading cache from {self.annot_path_cache}')
self.datalist = self.load_cache(self.annot_path_cache)
else:
if self.use_cache:
print(f'[{self.__class__.__name__}] Cache not found, generating cache...')
self.datalist = self.load_data(
train_sample_interval=getattr(cfg, f'{self.__class__.__name__}_train_sample_interval', 1))
if self.use_cache:
self.save_cache(self.annot_path_cache, self.datalist)
data/CHI3D/CHI3D.py
-56
View File
@@ -1,56 +0,0 @@
import os
import os.path as osp
import numpy as np
import torch
import cv2
import json
import copy
from pycocotools.coco import COCO
from config import cfg
from utils.human_models import smpl_x
from utils.preprocessing import load_img, process_bbox, augmentation, process_db_coord, process_human_model_output, \
get_fitting_error_3D
from utils.transforms import world2cam, cam2pixel, rigid_align
from humandata import HumanDataset
class CHI3D(HumanDataset):
def __init__(self, transform, data_split):
super(CHI3D, self).__init__(transform, data_split)
self.use_cache = getattr(cfg, 'use_cache', False)
self.annot_path_cache = osp.join(cfg.data_dir, 'cache', 'CHI3D_train_230511_1492.npz')
self.img_shape = (900, 900) # (h, w)
self.cam_param = {}
# load data or cache
if self.use_cache and osp.isfile(self.annot_path_cache):
print(f'[{self.__class__.__name__}] loading cache from {self.annot_path_cache}')
self.datalist = self.load_cache(self.annot_path_cache)
else:
if self.use_cache:
print(f'[{self.__class__.__name__}] Cache not found, generating cache...')
self.datalist = []
for pre_prc_file in ['CHI3D_train_230511_1492_0.npz','CHI3D_train_230511_1492_1.npz']:
if self.data_split == 'train':
filename = getattr(cfg, 'filename', pre_prc_file)
else:
raise ValueError('CHI3D test set is not support')
self.img_dir = cfg.data_dir # CHI3D included
self.annot_path = osp.join(cfg.data_dir, 'preprocessed_datasets', filename)
# check image shape
img_path = osp.join(self.img_dir, np.load(self.annot_path)['image_path'][0])
img_shape = cv2.imread(img_path).shape[:2]
assert self.img_shape == img_shape, 'image shape is incorrect: {} vs {}'.format(self.img_shape, img_shape)
# load data
datalist_slice = self.load_data(
train_sample_interval=getattr(cfg, f'{self.__class__.__name__}_train_sample_interval', 1))
self.datalist.extend(datalist_slice)
if self.use_cache:
self.save_cache(self.annot_path_cache, self.datalist)
data/CrowdPose/CrowdPose.py
-47
View File
@@ -1,47 +0,0 @@
import os
import os.path as osp
import numpy as np
import torch
import cv2
import json
import copy
from pycocotools.coco import COCO
from config import cfg
from utils.human_models import smpl_x
from utils.preprocessing import load_img, process_bbox, augmentation, process_db_coord, process_human_model_output, \
get_fitting_error_3D
from utils.transforms import world2cam, cam2pixel, rigid_align
from humandata import HumanDataset
class CrowdPose(HumanDataset):
def __init__(self, transform, data_split):
super(CrowdPose, self).__init__(transform, data_split)
self.datalist = []
pre_prc_file = 'crowdpose_neural_annot_train_new.npz'
if self.data_split == 'train':
filename = getattr(cfg, 'filename', pre_prc_file)
else:
raise ValueError('CrowdPose test set is not support')
self.img_dir = osp.join(cfg.data_dir, 'CrowdPose')
self.annot_path = osp.join(cfg.data_dir, 'preprocessed_datasets', filename)
self.annot_path_cache = osp.join(cfg.data_dir, 'cache', filename)
self.use_cache = getattr(cfg, 'use_cache', False)
self.img_shape = None
self.cam_param = {}
print("Various image shape in CrowdPose dataset.")
# load data or cache
if self.use_cache and osp.isfile(self.annot_path_cache):
print(f'[{self.__class__.__name__}] loading cache from {self.annot_path_cache}')
self.datalist = self.load_cache(self.annot_path_cache)
else:
if self.use_cache:
print(f'[{self.__class__.__name__}] Cache not found, generating cache...')
self.datalist = self.load_data(
train_sample_interval=getattr(cfg, f'{self.__class__.__name__}_train_sample_interval', 1))
if self.use_cache:
self.save_cache(self.annot_path_cache, self.datalist)
data/EHF/EHF.py
-372
View File
@@ -1,372 +0,0 @@
import os
import os.path as osp
from glob import glob
import numpy as np
from config import cfg
import copy
import json
import cv2
import torch
from pycocotools.coco import COCO
from utils.human_models import smpl_x
from utils.preprocessing import load_img, process_bbox, augmentation, load_ply
from utils.transforms import rigid_align
class EHF(torch.utils.data.Dataset):
def __init__(self, transform, data_split):
self.transform = transform
self.data_split = data_split
# self.data_path = osp.join('..', 'data', 'EHF', 'data')
self.data_path = osp.join(cfg.data_dir, 'EHF', 'data')
self.datalist = self.load_data()
self.cam_param = {'R': [-2.98747896, 0.01172457, -0.05704687]}
self.cam_param['R'], _ = cv2.Rodrigues(np.array(self.cam_param['R']))
self.save_idx = 0
def load_data(self):
datalist = []
db = COCO(osp.join(self.data_path, 'EHF.json'))
for aid in db.anns.keys():
ann = db.anns[aid]
image_id = ann['image_id']
img = db.loadImgs(image_id)[0]
img_shape = (img['height'], img['width'])
img_path = osp.join(self.data_path, img['file_name'])
bbox = ann['body_bbox']
bbox = process_bbox(bbox, img['width'], img['height'], ratio=getattr(cfg, 'bbox_ratio', 1.25))
if bbox is None:
continue
lhand_bbox = np.array(ann['lefthand_bbox']).reshape(4)
if hasattr(cfg, 'bbox_ratio'):
lhand_bbox = process_bbox(lhand_bbox, img['width'], img['height'], ratio=cfg.bbox_ratio)
lhand_bbox[2:] += lhand_bbox[:2] # xywh -> xyxy
rhand_bbox = np.array(ann['righthand_bbox']).reshape(4)
if hasattr(cfg, 'bbox_ratio'):
rhand_bbox = process_bbox(rhand_bbox, img['width'], img['height'], ratio=cfg.bbox_ratio)
rhand_bbox[2:] += rhand_bbox[:2] # xywh -> xyxy
face_bbox = np.array(ann['face_bbox']).reshape(4)
if hasattr(cfg, 'bbox_ratio'):
face_bbox = process_bbox(face_bbox, img['width'], img['height'], ratio=cfg.bbox_ratio)
face_bbox[2:] += face_bbox[:2] # xywh -> xyxy
mesh_gt_path = osp.join(self.data_path, img['file_name'].split('_')[0] + '_align.ply')
data_dict = {'img_path': img_path, 'img_shape': img_shape, 'bbox': bbox, 'lhand_bbox': lhand_bbox,
'rhand_bbox': rhand_bbox, 'face_bbox': face_bbox, 'mesh_gt_path': mesh_gt_path}
datalist.append(data_dict)
return datalist
def process_hand_face_bbox(self, bbox, do_flip, img_shape, img2bb_trans):
if bbox is None:
bbox = np.zeros((2, 2), dtype=np.float32) # dummy value
bbox_valid = float(False) # dummy value
else:
# reshape to top-left (x,y) and bottom-right (x,y)
bbox = bbox.reshape(2, 2)
# flip augmentation
if do_flip:
bbox[:, 0] = img_shape[1] - bbox[:, 0] - 1
bbox[0, 0], bbox[1, 0] = bbox[1, 0].copy(), bbox[0, 0].copy() # xmin <-> xmax swap
# make four points of the bbox
bbox = bbox.reshape(4).tolist()
xmin, ymin, xmax, ymax = bbox
bbox = np.array([[xmin, ymin], [xmax, ymin], [xmax, ymax], [xmin, ymax]], dtype=np.float32).reshape(4, 2)
# affine transformation (crop, rotation, scale)
bbox_xy1 = np.concatenate((bbox, np.ones_like(bbox[:, :1])), 1)
bbox = np.dot(img2bb_trans, bbox_xy1.transpose(1, 0)).transpose(1, 0)[:, :2]
bbox[:, 0] = bbox[:, 0] / cfg.input_img_shape[1] * cfg.output_hm_shape[1]
bbox[:, 1] = bbox[:, 1] / cfg.input_img_shape[0] * cfg.output_hm_shape[0]
# make box a rectangle without rotation
xmin = np.min(bbox[:, 0]);
xmax = np.max(bbox[:, 0]);
ymin = np.min(bbox[:, 1]);
ymax = np.max(bbox[:, 1]);
bbox = np.array([xmin, ymin, xmax, ymax], dtype=np.float32)
bbox_valid = float(True)
bbox = bbox.reshape(2, 2)
return bbox, bbox_valid
def __len__(self):
return len(self.datalist)
def __getitem__(self, idx):
data = copy.deepcopy(self.datalist[idx])
img_path, img_shape, bbox, mesh_gt_path = data['img_path'], data['img_shape'], data['bbox'], data[
'mesh_gt_path']
# image load
img = load_img(img_path)
# affine transform
img, img2bb_trans, bb2img_trans, rot, do_flip = augmentation(img, bbox, self.data_split)
img = self.transform(img.astype(np.float32)) / 255.
# hand and face bbox transform
lhand_bbox, lhand_bbox_valid = self.process_hand_face_bbox(data['lhand_bbox'], do_flip, img_shape, img2bb_trans)
rhand_bbox, rhand_bbox_valid = self.process_hand_face_bbox(data['rhand_bbox'], do_flip, img_shape, img2bb_trans)
face_bbox, face_bbox_valid = self.process_hand_face_bbox(data['face_bbox'], do_flip, img_shape, img2bb_trans)
if do_flip:
lhand_bbox, rhand_bbox = rhand_bbox, lhand_bbox
lhand_bbox_valid, rhand_bbox_valid = rhand_bbox_valid, lhand_bbox_valid
lhand_bbox_center = (lhand_bbox[0] + lhand_bbox[1]) / 2.;
rhand_bbox_center = (rhand_bbox[0] + rhand_bbox[1]) / 2.;
face_bbox_center = (face_bbox[0] + face_bbox[1]) / 2.
lhand_bbox_size = lhand_bbox[1] - lhand_bbox[0];
rhand_bbox_size = rhand_bbox[1] - rhand_bbox[0];
face_bbox_size = face_bbox[1] - face_bbox[0];
# mesh gt load
mesh_gt = load_ply(mesh_gt_path)
inputs = {'img': img}
targets = {'smplx_mesh_cam': mesh_gt, 'lhand_bbox_center': lhand_bbox_center,
'rhand_bbox_center': rhand_bbox_center, 'face_bbox_center': face_bbox_center,
'lhand_bbox_size': lhand_bbox_size, 'rhand_bbox_size': rhand_bbox_size,
'face_bbox_size': face_bbox_size}
meta_info = {'bb2img_trans': bb2img_trans, 'lhand_bbox_valid': float(True), 'rhand_bbox_valid': float(True),
'face_bbox_valid': float(True),
'img_path': img_path}
return inputs, targets, meta_info
def evaluate(self, outs, cur_sample_idx):
annots = self.datalist
sample_num = len(outs)
eval_result = {'pa_mpvpe_all': [], 'pa_mpvpe_l_hand': [], 'pa_mpvpe_r_hand': [], 'pa_mpvpe_hand': [], 'pa_mpvpe_face': [],
'mpvpe_all': [], 'mpvpe_l_hand': [], 'mpvpe_r_hand': [], 'mpvpe_hand': [], 'mpvpe_face': [],
'pa_mpjpe_body': [], 'pa_mpjpe_l_hand': [], 'pa_mpjpe_r_hand': [], 'pa_mpjpe_hand': []}
if getattr(cfg, 'vis', False):
import csv
csv_file = f'{cfg.vis_dir}/ehf_smplx_error.csv'
file = open(csv_file, 'a', newline='')
writer = csv.writer(file)
for n in range(sample_num):
annot = annots[cur_sample_idx + n]
ann_id = annot['img_path'].split('/')[-1].split('_')[0]
# print(annot['img_path'])
# ann_id = annot['ann_id']
out = outs[n]
# MPVPE from all vertices
mesh_gt = np.dot(self.cam_param['R'], out['smplx_mesh_cam_target'].transpose(1, 0)).transpose(1, 0)
mesh_out = out['smplx_mesh_cam']
# mesh_gt_align = rigid_align(mesh_gt, mesh_out)
# print(mesh_out.shape)
mesh_out_align = rigid_align(mesh_out, mesh_gt)
pa_mpvpe_all = np.sqrt(np.sum((mesh_out_align - mesh_gt) ** 2, 1)).mean() * 1000
eval_result['pa_mpvpe_all'].append(pa_mpvpe_all)
mesh_out_align = mesh_out - np.dot(smpl_x.J_regressor, mesh_out)[smpl_x.J_regressor_idx['pelvis'], None,
:] + np.dot(smpl_x.J_regressor, mesh_gt)[smpl_x.J_regressor_idx['pelvis'], None,
:]
mpvpe_all = np.sqrt(np.sum((mesh_out_align - mesh_gt) ** 2, 1)).mean() * 1000
eval_result['mpvpe_all'].append(mpvpe_all)
# MPVPE from hand vertices
mesh_gt_lhand = mesh_gt[smpl_x.hand_vertex_idx['left_hand'], :]
mesh_out_lhand = mesh_out[smpl_x.hand_vertex_idx['left_hand'], :]
mesh_out_lhand_align = rigid_align(mesh_out_lhand, mesh_gt_lhand)
mesh_gt_rhand = mesh_gt[smpl_x.hand_vertex_idx['right_hand'], :]
mesh_out_rhand = mesh_out[smpl_x.hand_vertex_idx['right_hand'], :]
mesh_out_rhand_align = rigid_align(mesh_out_rhand, mesh_gt_rhand)
eval_result['pa_mpvpe_l_hand'].append(np.sqrt(
np.sum((mesh_out_lhand_align - mesh_gt_lhand) ** 2, 1)).mean() * 1000)
eval_result['pa_mpvpe_r_hand'].append(np.sqrt(
np.sum((mesh_out_rhand_align - mesh_gt_rhand) ** 2, 1)).mean() * 1000)
eval_result['pa_mpvpe_hand'].append((np.sqrt(
np.sum((mesh_out_lhand_align - mesh_gt_lhand) ** 2, 1)).mean() * 1000 + np.sqrt(
np.sum((mesh_out_rhand_align - mesh_gt_rhand) ** 2, 1)).mean() * 1000) / 2.)
mesh_out_lhand_align = mesh_out_lhand - np.dot(smpl_x.J_regressor, mesh_out)[
smpl_x.J_regressor_idx['lwrist'], None, :] + np.dot(
smpl_x.J_regressor, mesh_gt)[smpl_x.J_regressor_idx['lwrist'], None, :]
mesh_out_rhand_align = mesh_out_rhand - np.dot(smpl_x.J_regressor, mesh_out)[
smpl_x.J_regressor_idx['rwrist'], None, :] + np.dot(
smpl_x.J_regressor, mesh_gt)[smpl_x.J_regressor_idx['rwrist'], None, :]
eval_result['mpvpe_l_hand'].append(np.sqrt(
np.sum((mesh_out_lhand_align - mesh_gt_lhand) ** 2, 1)).mean() * 1000)
eval_result['mpvpe_r_hand'].append(np.sqrt(
np.sum((mesh_out_rhand_align - mesh_gt_rhand) ** 2, 1)).mean() * 1000)
eval_result['mpvpe_hand'].append((np.sqrt(
np.sum((mesh_out_lhand_align - mesh_gt_lhand) ** 2, 1)).mean() * 1000 + np.sqrt(
np.sum((mesh_out_rhand_align - mesh_gt_rhand) ** 2, 1)).mean() * 1000) / 2.)
# MPVPE from face vertices
mesh_gt_face = mesh_gt[smpl_x.face_vertex_idx, :]
mesh_out_face = mesh_out[smpl_x.face_vertex_idx, :]
mesh_out_face_align = rigid_align(mesh_out_face, mesh_gt_face)
eval_result['pa_mpvpe_face'].append(
np.sqrt(np.sum((mesh_out_face_align - mesh_gt_face) ** 2, 1)).mean() * 1000)
mesh_out_face_align = mesh_out_face - np.dot(smpl_x.J_regressor, mesh_out)[smpl_x.J_regressor_idx['neck'],
None, :] + np.dot(smpl_x.J_regressor, mesh_gt)[
smpl_x.J_regressor_idx['neck'], None, :]
eval_result['mpvpe_face'].append(
np.sqrt(np.sum((mesh_out_face_align - mesh_gt_face) ** 2, 1)).mean() * 1000)
# MPJPE from body joints
joint_gt_body = np.dot(smpl_x.j14_regressor, mesh_gt)
joint_out_body = np.dot(smpl_x.j14_regressor, mesh_out)
joint_out_body_align = rigid_align(joint_out_body, joint_gt_body)
eval_result['pa_mpjpe_body'].append(
np.sqrt(np.sum((joint_out_body_align - joint_gt_body) ** 2, 1)).mean() * 1000)
# MPJPE from hand joints
joint_gt_lhand = np.dot(smpl_x.orig_hand_regressor['left'], mesh_gt)
joint_out_lhand = np.dot(smpl_x.orig_hand_regressor['left'], mesh_out)
joint_out_lhand_align = rigid_align(joint_out_lhand, joint_gt_lhand)
joint_gt_rhand = np.dot(smpl_x.orig_hand_regressor['right'], mesh_gt)
joint_out_rhand = np.dot(smpl_x.orig_hand_regressor['right'], mesh_out)
joint_out_rhand_align = rigid_align(joint_out_rhand, joint_gt_rhand)
eval_result['pa_mpjpe_l_hand'].append(np.sqrt(
np.sum((joint_out_lhand_align - joint_gt_lhand) ** 2, 1)).mean() * 1000)
eval_result['pa_mpjpe_r_hand'].append(np.sqrt(
np.sum((joint_out_rhand_align - joint_gt_rhand) ** 2, 1)).mean() * 1000)
eval_result['pa_mpjpe_hand'].append((np.sqrt(
np.sum((joint_out_lhand_align - joint_gt_lhand) ** 2, 1)).mean() * 1000 + np.sqrt(
np.sum((joint_out_rhand_align - joint_gt_rhand) ** 2, 1)).mean() * 1000) / 2.)
vis = cfg.vis
if vis:
# save_folder = cfg.vis_dir
# kpt_save_folder = os.path.join(save_folder, 'KPT')
# os.makedirs(kpt_save_folder, exist_ok=True)
# mesh_save_folder = os.path.join(save_folder, 'mesh_origin')
# os.makedirs(mesh_save_folder, exist_ok=True)
# # from utils.vis import vis_keypoints, render_mesh, save_obj
# img = (out['img'].transpose(1, 2, 0)[:, :, ::-1] * 255).copy()
# joint_img = out['joint_img'].copy()
# joint_img[:, 0] = joint_img[:, 0] / cfg.output_hm_shape[2] * cfg.input_img_shape[1]
# joint_img[:, 1] = joint_img[:, 1] / cfg.output_hm_shape[1] * cfg.input_img_shape[0]
# for j in range(len(joint_img)):
# cv2.circle(img, (int(joint_img[j][0]), int(joint_img[j][1])), 3, (0, 0, 255), -1)
# lhand_bbox = out['lhand_bbox'].reshape(2, 2).copy()
# cv2.rectangle(img, (int(lhand_bbox[0][0]), int(lhand_bbox[0][1])),
# (int(lhand_bbox[1][0]), int(lhand_bbox[1][1])), (255, 0, 0), 3)
# rhand_bbox = out['rhand_bbox'].reshape(2, 2).copy()
# cv2.rectangle(img, (int(rhand_bbox[0][0]), int(rhand_bbox[0][1])),
# (int(rhand_bbox[1][0]), int(rhand_bbox[1][1])), (255, 0, 0), 3)
# face_bbox = out['face_bbox'].reshape(2, 2).copy()
# cv2.rectangle(img, (int(face_bbox[0][0]), int(face_bbox[0][1])),
# (int(face_bbox[1][0]), int(face_bbox[1][1])), (255, 0, 0), 3)
# cv2.imwrite(os.path.join(kpt_save_folder, str(cur_sample_idx + n) + '.jpg'), img)
# vis_img = img.copy()
# focal = [cfg.focal[0] / cfg.input_body_shape[1] * cfg.input_img_shape[1],
# cfg.focal[1] / cfg.input_body_shape[0] * cfg.input_img_shape[0]]
# princpt = [cfg.princpt[0] / cfg.input_body_shape[1] * cfg.input_img_shape[1],
# cfg.princpt[1] / cfg.input_body_shape[0] * cfg.input_img_shape[0]]
# rendered_img = render_mesh(vis_img, mesh_out, smpl_x.face, {'focal': focal, 'princpt': princpt})
# vis_img = img.copy()
# # rendered_img_gt = render_mesh(vis_img, mesh_gt_align, smpl_x.face, {'focal': focal, 'princpt': princpt})
# cv2.imwrite(os.path.join(mesh_save_folder, f'{ann_id}_render.jpg'), rendered_img)
# # cv2.imwrite(os.path.join(mesh_save_folder, f'{ann_id}_render_gt.jpg'), rendered_img_gt)
# cv2.imwrite(os.path.join(mesh_save_folder, f'{ann_id}.jpg'), vis_img)
# np.save(os.path.join(mesh_save_folder, f'{ann_id}.npy'), mesh_out)
# save smplx param
smplx_pred = {}
smplx_pred['global_orient'] = out['smplx_root_pose'].reshape(-1,3)
smplx_pred['body_pose'] = out['smplx_body_pose'].reshape(-1,3)
smplx_pred['left_hand_pose'] = out['smplx_lhand_pose'].reshape(-1,3)
smplx_pred['right_hand_pose'] = out['smplx_rhand_pose'].reshape(-1,3)
smplx_pred['jaw_pose'] = out['smplx_jaw_pose'].reshape(-1,3)
smplx_pred['leye_pose'] = np.zeros((1, 3))
smplx_pred['reye_pose'] = np.zeros((1, 3))
smplx_pred['betas'] = out['smplx_shape'].reshape(-1,10)
smplx_pred['expression'] = out['smplx_expr'].reshape(-1,10)
smplx_pred['transl'] = out['cam_trans'].reshape(-1,3)
np.savez(os.path.join(cfg.vis_dir, f'{self.save_idx}.npz'), **smplx_pred)
# save img path and error
img_path = out['img_path']
rel_img_path = img_path.split('..')[-1]
new_line = [self.save_idx, rel_img_path, mpvpe_all, pa_mpvpe_all]
# Append the new line to the CSV file
writer.writerow(new_line)
self.save_idx += 1
# save_obj(out['smplx_mesh_cam'], smpl_x.face, str(cur_sample_idx + n) + '.obj')
if getattr(cfg, 'vis', False):
file.close()
return eval_result
def print_eval_result(self, eval_result):
print('======EHF======')
print('PA MPVPE (All): %.2f mm' % np.mean(eval_result['pa_mpvpe_all']))
print('PA MPVPE (L-Hands): %.2f mm' % np.mean(eval_result['pa_mpvpe_l_hand']))
print('PA MPVPE (R-Hands): %.2f mm' % np.mean(eval_result['pa_mpvpe_r_hand']))
print('PA MPVPE (Hands): %.2f mm' % np.mean(eval_result['pa_mpvpe_hand']))
print('PA MPVPE (Face): %.2f mm' % np.mean(eval_result['pa_mpvpe_face']))
print()
print('MPVPE (All): %.2f mm' % np.mean(eval_result['mpvpe_all']))
print('MPVPE (L-Hands): %.2f mm' % np.mean(eval_result['mpvpe_l_hand']))
print('MPVPE (R-Hands): %.2f mm' % np.mean(eval_result['mpvpe_r_hand']))
print('MPVPE (Hands): %.2f mm' % np.mean(eval_result['mpvpe_hand']))
print('MPVPE (Face): %.2f mm' % np.mean(eval_result['mpvpe_face']))
print()
print('PA MPJPE (Body): %.2f mm' % np.mean(eval_result['pa_mpjpe_body']))
print('PA MPJPE (L-Hands): %.2f mm' % np.mean(eval_result['pa_mpjpe_l_hand']))
print('PA MPJPE (R-Hands): %.2f mm' % np.mean(eval_result['pa_mpjpe_r_hand']))
print('PA MPJPE (Hands): %.2f mm' % np.mean(eval_result['pa_mpjpe_hand']))
print()
print(f"{np.mean(eval_result['pa_mpvpe_all'])},{np.mean(eval_result['pa_mpvpe_l_hand'])},{np.mean(eval_result['pa_mpvpe_r_hand'])},{np.mean(eval_result['pa_mpvpe_hand'])},{np.mean(eval_result['pa_mpvpe_face'])},"
f"{np.mean(eval_result['mpvpe_all'])},{np.mean(eval_result['mpvpe_l_hand'])},{np.mean(eval_result['mpvpe_r_hand'])},{np.mean(eval_result['mpvpe_hand'])},{np.mean(eval_result['mpvpe_face'])}")
print()
f = open(os.path.join(cfg.result_dir, 'result.txt'), 'w')
f.write(f'EHF dataset: \n')
f.write('PA MPVPE (All): %.2f mm\n' % np.mean(eval_result['pa_mpvpe_all']))
f.write('PA MPVPE (L-Hands): %.2f mm' % np.mean(eval_result['pa_mpvpe_l_hand']))
f.write('PA MPVPE (R-Hands): %.2f mm' % np.mean(eval_result['pa_mpvpe_r_hand']))
f.write('PA MPVPE (Hands): %.2f mm\n' % np.mean(eval_result['pa_mpvpe_hand']))
f.write('PA MPVPE (Face): %.2f mm\n' % np.mean(eval_result['pa_mpvpe_face']))
f.write('MPVPE (All): %.2f mm\n' % np.mean(eval_result['mpvpe_all']))
f.write('MPVPE (L-Hands): %.2f mm' % np.mean(eval_result['mpvpe_l_hand']))
f.write('MPVPE (R-Hands): %.2f mm' % np.mean(eval_result['mpvpe_r_hand']))
f.write('MPVPE (Hands): %.2f mm' % np.mean(eval_result['mpvpe_hand']))
f.write('MPVPE (Face): %.2f mm\n' % np.mean(eval_result['mpvpe_face']))
f.write('PA MPJPE (Body): %.2f mm\n' % np.mean(eval_result['pa_mpjpe_body']))
f.write('PA MPJPE (L-Hands): %.2f mm' % np.mean(eval_result['pa_mpjpe_l_hand']))
f.write('PA MPJPE (R-Hands): %.2f mm' % np.mean(eval_result['pa_mpjpe_r_hand']))
f.write('PA MPJPE (Hands): %.2f mm\n' % np.mean(eval_result['pa_mpjpe_hand']))
f.write(f"{np.mean(eval_result['pa_mpvpe_all'])},{np.mean(eval_result['pa_mpvpe_l_hand'])},{np.mean(eval_result['pa_mpvpe_r_hand'])},{np.mean(eval_result['pa_mpvpe_hand'])},{np.mean(eval_result['pa_mpvpe_face'])},"
f"{np.mean(eval_result['mpvpe_all'])},{np.mean(eval_result['mpvpe_l_hand'])},{np.mean(eval_result['mpvpe_r_hand'])},{np.mean(eval_result['mpvpe_hand'])},{np.mean(eval_result['mpvpe_face'])}")
# for i in range(len(eval_result['pa_mpvpe_all'])):
# f.write(f'{i+1:02d}.jpg\n')
# f.write('PA MPVPE (All): %.2f mm\n' % eval_result['pa_mpvpe_all'][i])
# f.write('PA MPVPE (Hands): %.2f mm\n' % eval_result['pa_mpvpe_hand'][i])
# f.write('PA MPVPE (Face): %.2f mm\n' % eval_result['pa_mpvpe_face'][i])
# f.write('MPVPE (All): %.2f mm\n' % eval_result['mpvpe_all'][i])
# f.write('MPVPE (Hands): %.2f mm\n' % eval_result['mpvpe_hand'][i])
# f.write('MPVPE (Face): %.2f mm\n' % eval_result['mpvpe_face'][i])
# f.write('PA MPJPE (Body): %.2f mm\n' % eval_result['pa_mpjpe_body'][i])
# f.write('PA MPJPE (Hands): %.2f mm\n' % eval_result['pa_mpjpe_hand'][i])
data/EgoBody_Egocentric/EgoBody_Egocentric.py
-53
View File
@@ -1,53 +0,0 @@
import os
import os.path as osp
import numpy as np
import torch
import cv2
import json
import copy
from pycocotools.coco import COCO
from config import cfg
from utils.human_models import smpl_x
from utils.preprocessing import load_img, process_bbox, augmentation, process_db_coord, process_human_model_output, \
get_fitting_error_3D
from utils.transforms import world2cam, cam2pixel, rigid_align
from humandata import HumanDataset
class EgoBody_Egocentric(HumanDataset):
def __init__(self, transform, data_split):
super(EgoBody_Egocentric, self).__init__(transform, data_split)
if getattr(cfg, 'eval_on_train', False):
self.data_split = 'eval_train'
print("Evaluate on train set.")
if 'train' in self.data_split:
filename = getattr(cfg, 'filename', 'egobody_egocentric_train_230425_065_fix_betas.npz')
else:
filename = getattr(cfg, 'filename', 'egobody_egocentric_test_230425_043_fix_betas.npz')
self.use_betas_neutral = getattr(cfg, 'egobody_fix_betas', False)
self.img_dir = osp.join(cfg.data_dir, 'EgoBody')
self.annot_path = osp.join(cfg.data_dir, 'preprocessed_datasets', filename)
self.annot_path_cache = osp.join(cfg.data_dir, 'cache', filename)
self.use_cache = getattr(cfg, 'use_cache', False)
self.img_shape = (1080, 1920) # (h, w)
self.cam_param = {}
# check image shape
img_path = osp.join(self.img_dir, np.load(self.annot_path)['image_path'][0])
img_shape = cv2.imread(img_path).shape[:2]
assert self.img_shape == img_shape, 'image shape is incorrect: {} vs {}'.format(self.img_shape, img_shape)
# load data or cache
if self.use_cache and osp.isfile(self.annot_path_cache):
print(f'[{self.__class__.__name__}] loading cache from {self.annot_path_cache}')
self.datalist = self.load_cache(self.annot_path_cache)
else:
if self.use_cache:
print(f'[{self.__class__.__name__}] Cache not found, generating cache...')
self.datalist = self.load_data(
train_sample_interval=getattr(cfg, f'{self.__class__.__name__}_train_sample_interval', 1))
if self.use_cache:
self.save_cache(self.annot_path_cache, self.datalist)
data/EgoBody_Kinect/EgoBody_Kinect.py
-50
View File
@@ -1,50 +0,0 @@
import os
import os.path as osp
import numpy as np
import torch
import cv2
import json
import copy
from pycocotools.coco import COCO
from config import cfg
from utils.human_models import smpl_x
from utils.preprocessing import load_img, process_bbox, augmentation, process_db_coord, process_human_model_output, \
get_fitting_error_3D
from utils.transforms import world2cam, cam2pixel, rigid_align
from humandata import HumanDataset
class EgoBody_Kinect(HumanDataset):
def __init__(self, transform, data_split):
super(EgoBody_Kinect, self).__init__(transform, data_split)
if self.data_split == 'train':
filename = getattr(cfg, 'filename', 'egobody_kinect_train_230503_065_fix_betas.npz')
else:
filename = getattr(cfg, 'filename', 'egobody_kinect_test_230503_043_fix_betas.npz')
self.use_betas_neutral = getattr(cfg, 'egobody_fix_betas', False)
self.img_dir = osp.join(cfg.data_dir, 'EgoBody')
self.annot_path = osp.join(cfg.data_dir, 'preprocessed_datasets', filename)
self.annot_path_cache = osp.join(cfg.data_dir, 'cache', filename)
self.use_cache = getattr(cfg, 'use_cache', False)
self.img_shape = (1080, 1920) # (h, w)
self.cam_param = {}
# check image shape
img_path = osp.join(self.img_dir, np.load(self.annot_path)['image_path'][0])
img_shape = cv2.imread(img_path).shape[:2]
assert self.img_shape == img_shape, 'image shape is incorrect: {} vs {}'.format(self.img_shape, img_shape)
# load data or cache
if self.use_cache and osp.isfile(self.annot_path_cache):
print(f'[{self.__class__.__name__}] loading cache from {self.annot_path_cache}')
self.datalist = self.load_cache(self.annot_path_cache)
else:
if self.use_cache:
print(f'[{self.__class__.__name__}] Cache not found, generating cache...')
self.datalist = self.load_data(
train_sample_interval=getattr(cfg, f'{self.__class__.__name__}_{self.data_split}_sample_interval', 1))
if self.use_cache:
self.save_cache(self.annot_path_cache, self.datalist)
data/FIT3D/FIT3D.py
-58
View File
@@ -1,58 +0,0 @@
import os
import os.path as osp
import numpy as np
import torch
import cv2
import json
import copy
from pycocotools.coco import COCO
from config import cfg
from utils.human_models import smpl_x
from utils.preprocessing import load_img, process_bbox, augmentation, process_db_coord, process_human_model_output, \
get_fitting_error_3D
from utils.transforms import world2cam, cam2pixel, rigid_align
from humandata import HumanDataset
class FIT3D(HumanDataset):
def __init__(self, transform, data_split):
super(FIT3D, self).__init__(transform, data_split)
self.use_cache = getattr(cfg, 'use_cache', False)
self.annot_path_cache = osp.join(cfg.data_dir, 'cache', 'FIT3D_train_230511_1504.npz')
self.img_shape = (900, 900) # (h, w)
self.cam_param = {}
# load data or cache
if self.use_cache and osp.isfile(self.annot_path_cache):
print(f'[{self.__class__.__name__}] loading cache from {self.annot_path_cache}')
self.datalist = self.load_cache(self.annot_path_cache)
else:
if self.use_cache:
print(f'[{self.__class__.__name__}] Cache not found, generating cache...')
self.datalist = []
for pre_prc_file in ['FIT3D_train_230511_1504_0.npz','FIT3D_train_230511_1504_1.npz',
'FIT3D_train_230511_1504_2.npz', 'FIT3D_train_230511_1504_3.npz',
'FIT3D_train_230511_1504_4.npz', 'FIT3D_train_230511_1504_5.npz']:
if self.data_split == 'train':
filename = getattr(cfg, 'filename', pre_prc_file)
else:
raise ValueError('FIT3D test set is not support')
self.img_dir = cfg.data_dir # FIT3D included
self.annot_path = osp.join(cfg.data_dir, 'preprocessed_datasets', filename)
# check image shape
img_path = osp.join(self.img_dir, np.load(self.annot_path)['image_path'][0])
img_shape = cv2.imread(img_path).shape[:2]
assert self.img_shape == img_shape, 'image shape is incorrect: {} vs {}'.format(self.img_shape, img_shape)
# load data
datalist_slice = self.load_data(
train_sample_interval=getattr(cfg, f'{self.__class__.__name__}_train_sample_interval', 1))
self.datalist.extend(datalist_slice)
if self.use_cache:
self.save_cache(self.annot_path_cache, self.datalist)
data/GTA_Human2/GTA_Human2.py
-47
View File
@@ -1,47 +0,0 @@
import os
import os.path as osp
import numpy as np
import torch
import cv2
import json
import copy
from pycocotools.coco import COCO
from config import cfg
from utils.human_models import smpl_x
from utils.preprocessing import load_img, process_bbox, augmentation, process_db_coord, process_human_model_output, \
get_fitting_error_3D
from utils.transforms import world2cam, cam2pixel, rigid_align
from humandata import HumanDataset
class GTA_Human2(HumanDataset):
def __init__(self, transform, data_split):
super(GTA_Human2, self).__init__(transform, data_split)
filename = 'gta_human2multiple_230406_04000_0.npz'
self.img_dir = osp.join(cfg.data_dir, 'GTA_Human2')
self.annot_path = osp.join(cfg.data_dir, 'preprocessed_datasets', filename)
self.annot_path_cache = osp.join(cfg.data_dir, 'cache', filename)
self.use_cache = getattr(cfg, 'use_cache', False)
self.img_shape = (1080, 1920) # (h, w)
self.cam_param = {
'focal': (1158.0337, 1158.0337), # (fx, fy)
'princpt': (960, 540) # (cx, cy)
}
# check image shape
img_path = osp.join(self.img_dir, np.load(self.annot_path)['image_path'][0])
img_shape = cv2.imread(img_path).shape[:2]
assert self.img_shape == img_shape, 'image shape is incorrect: {} vs {}'.format(self.img_shape, img_shape)
# load data or cache
if self.use_cache and osp.isfile(self.annot_path_cache):
print(f'[{self.__class__.__name__}] loading cache from {self.annot_path_cache}')
self.datalist = self.load_cache(self.annot_path_cache)
else:
if self.use_cache:
print(f'[{self.__class__.__name__}] Cache not found, generating cache...')
self.datalist = self.load_data(
train_sample_interval=getattr(cfg, f'{self.__class__.__name__}_train_sample_interval', 1))
if self.use_cache:
self.save_cache(self.annot_path_cache, self.datalist)
data/Human36M/Human36M.py
-277
View File
@@ -1,277 +0,0 @@
import os
import os.path as osp
import numpy as np
import torch
import cv2
import json
import copy
from pycocotools.coco import COCO
from config import cfg
from utils.human_models import smpl_x
from utils.preprocessing import load_img, process_bbox, augmentation, process_db_coord, process_human_model_output, get_fitting_error_3D
from utils.transforms import world2cam, cam2pixel, rigid_align
import random
from humandata import Cache
class Human36M(torch.utils.data.Dataset):
def __init__(self, transform, data_split):
self.transform = transform
self.data_split = data_split
self.img_dir = osp.join(cfg.data_dir, 'Human36M', 'images')
self.annot_path = osp.join(cfg.data_dir, 'Human36M', 'annotations')
self.action_name = ['Directions', 'Discussion', 'Eating', 'Greeting', 'Phoning', 'Posing', 'Purchases', 'Sitting', 'SittingDown', 'Smoking', 'Photo', 'Waiting', 'Walking', 'WalkDog', 'WalkTogether']
# H36M joint set
self.joint_set = {'joint_num': 17,
'joints_name': ('Pelvis', 'R_Hip', 'R_Knee', 'R_Ankle', 'L_Hip', 'L_Knee', 'L_Ankle', 'Torso', 'Neck', 'Head', 'Head_top', 'L_Shoulder', 'L_Elbow', 'L_Wrist', 'R_Shoulder', 'R_Elbow', 'R_Wrist'),
'flip_pairs': ( (1, 4), (2, 5), (3, 6), (14, 11), (15, 12), (16, 13) ),
'eval_joint': (1, 2, 3, 4, 5, 6, 8, 10, 11, 12, 13, 14, 15, 16),
'regressor': np.load(osp.join(cfg.data_dir, 'Human36M', 'J_regressor_h36m_smplx.npy'))
}
self.joint_set['root_joint_idx'] = self.joint_set['joints_name'].index('Pelvis')
# self.datalist = self.load_data()
# load data or cache
self.use_cache = getattr(cfg, 'use_cache', False)
self.annot_path_cache = osp.join(cfg.data_dir, 'cache', f'Human36M_{data_split}.npz')
if self.use_cache and osp.isfile(self.annot_path_cache):
print(f'[{self.__class__.__name__}] loading cache from {self.annot_path_cache}')
datalist = Cache(self.annot_path_cache)
assert datalist.data_strategy == getattr(cfg, 'data_strategy', None), \
f'Cache data strategy {datalist.data_strategy} does not match current data strategy ' \
f'{getattr(cfg, "data_strategy", None)}'
self.datalist = datalist
else:
if self.use_cache:
print(f'[{self.__class__.__name__}] Cache not found, generating cache...')
self.datalist = self.load_data()
if self.use_cache:
print(f'[{self.__class__.__name__}] Caching datalist to {self.annot_path_cache}...')
Cache.save(
self.annot_path_cache,
self.datalist,
data_strategy=getattr(cfg, 'data_strategy', None)
)
def get_subsampling_ratio(self):
if self.data_split == 'train':
return 5
elif self.data_split == 'test':
return 64
else:
assert 0, print('Unknown subset')
def get_subject(self):
if self.data_split == 'train':
subject = [1,5,6,7,8]
elif self.data_split == 'test':
subject = [9,11]
else:
assert 0, print("Unknown subset")
return subject
def load_data(self):
subject_list = self.get_subject()
sampling_ratio = self.get_subsampling_ratio()
# aggregate annotations from each subject
db = COCO()
cameras = {}
joints = {}
smplx_params = {}
for subject in subject_list:
# data load
with open(osp.join(self.annot_path, 'Human36M_subject' + str(subject) + '_data.json'),'r') as f:
annot = json.load(f)
if len(db.dataset) == 0:
for k,v in annot.items():
db.dataset[k] = v
else:
for k,v in annot.items():
db.dataset[k] += v
# camera load
with open(osp.join(self.annot_path, 'Human36M_subject' + str(subject) + '_camera.json'),'r') as f:
cameras[str(subject)] = json.load(f)
# joint coordinate load
with open(osp.join(self.annot_path, 'Human36M_subject' + str(subject) + '_joint_3d.json'),'r') as f:
joints[str(subject)] = json.load(f)
# smplx parameter load
with open(osp.join(self.annot_path, 'Human36M_subject' + str(subject) + '_SMPLX_NeuralAnnot.json'),'r') as f:
smplx_params[str(subject)] = json.load(f)
db.createIndex()
datalist = []
i = 0
for aid in db.anns.keys():
i += 1
if self.data_split == 'train' and i % getattr(cfg, 'Human36M_train_sample_interval', 1) != 0:
continue
ann = db.anns[aid]
image_id = ann['image_id']
img = db.loadImgs(image_id)[0]
img_path = osp.join(self.img_dir, img['file_name'])
img_shape = (img['height'], img['width'])
# check subject and frame_idx
frame_idx = img['frame_idx'];
if frame_idx % sampling_ratio != 0:
continue
# smplx parameter
subject = img['subject']; action_idx = img['action_idx']; subaction_idx = img['subaction_idx']; frame_idx = img['frame_idx']; cam_idx = img['cam_idx'];
smplx_param = smplx_params[str(subject)][str(action_idx)][str(subaction_idx)][str(frame_idx)]
# camera parameter
cam_param = cameras[str(subject)][str(cam_idx)]
R,t,f,c = np.array(cam_param['R'], dtype=np.float32), np.array(cam_param['t'], dtype=np.float32), np.array(cam_param['f'], dtype=np.float32), np.array(cam_param['c'], dtype=np.float32)
cam_param = {'R': R, 't': t, 'focal': f, 'princpt': c}
# only use frontal camera following previous works (HMR and SPIN)
if self.data_split == 'test' and str(cam_idx) != '4':
continue
# project world coordinate to cam, image coordinate space
joint_world = np.array(joints[str(subject)][str(action_idx)][str(subaction_idx)][str(frame_idx)], dtype=np.float32)
joint_cam = world2cam(joint_world, R, t)
joint_img = cam2pixel(joint_cam, f, c)[:,:2]
joint_valid = np.ones((self.joint_set['joint_num'],1))
bbox = process_bbox(np.array(ann['bbox']), img['width'], img['height'], ratio=getattr(cfg, 'bbox_ratio', 1.25))
if bbox is None: continue
datalist.append({
'img_path': img_path,
'img_shape': img_shape,
'bbox': bbox,
'joint_img': joint_img,
'joint_cam': joint_cam,
'joint_valid': joint_valid,
'smplx_param': smplx_param,
'cam_param': cam_param})
if self.data_split == 'train':
print('[Human36M train] original size:', len(db.anns.keys()),
'. Sample interval:', getattr(cfg, 'Human36M_train_sample_interval', 1),
'. Sampled size:', len(datalist))
if getattr(cfg, 'data_strategy', None) == 'balance' and self.data_split == 'train':
print(f'[Human36M] Using [balance] strategy with datalist shuffled...')
random.shuffle(datalist)
return datalist
def __len__(self):
return len(self.datalist)
def __getitem__(self, idx):
data = copy.deepcopy(self.datalist[idx])
img_path, img_shape, bbox, cam_param = data['img_path'], data['img_shape'], data['bbox'], data['cam_param']
# img
img = load_img(img_path)
img, img2bb_trans, bb2img_trans, rot, do_flip = augmentation(img, bbox, self.data_split)
img = self.transform(img.astype(np.float32))/255.
if self.data_split == 'train':
# h36m gt
joint_cam = data['joint_cam']
joint_cam = (joint_cam - joint_cam[self.joint_set['root_joint_idx'],None,:]) / 1000 # root-relative. milimeter to meter.
joint_img = data['joint_img']
joint_img = np.concatenate((joint_img[:,:2], joint_cam[:,2:]),1) # x, y, depth
joint_img[:,2] = (joint_img[:,2] / (cfg.body_3d_size / 2) + 1)/2. * cfg.output_hm_shape[0] # discretize depth
joint_img, joint_cam, joint_cam_ra, joint_valid, joint_trunc = process_db_coord(joint_img, joint_cam, data['joint_valid'], do_flip, img_shape, self.joint_set['flip_pairs'], img2bb_trans, rot, self.joint_set['joints_name'], smpl_x.joints_name)
# smplx coordinates and parameters
smplx_param = data['smplx_param']
cam_param['t'] /= 1000 # milimeter to meter
smplx_joint_img, smplx_joint_cam, smplx_joint_trunc, smplx_pose, smplx_shape, smplx_expr, \
smplx_pose_valid, smplx_joint_valid, smplx_expr_valid, smplx_mesh_cam_orig = \
process_human_model_output(smplx_param, cam_param, do_flip, img_shape, img2bb_trans, rot, 'smplx')
# reverse ra
smplx_joint_cam_wo_ra = smplx_joint_cam.copy()
smplx_joint_cam_wo_ra[smpl_x.joint_part['lhand'], :] = smplx_joint_cam_wo_ra[smpl_x.joint_part['lhand'], :] \
+ smplx_joint_cam_wo_ra[smpl_x.lwrist_idx, None, :] # left hand root-relative
smplx_joint_cam_wo_ra[smpl_x.joint_part['rhand'], :] = smplx_joint_cam_wo_ra[smpl_x.joint_part['rhand'], :] \
+ smplx_joint_cam_wo_ra[smpl_x.rwrist_idx, None, :] # right hand root-relative
smplx_joint_cam_wo_ra[smpl_x.joint_part['face'], :] = smplx_joint_cam_wo_ra[smpl_x.joint_part['face'], :] \
+ smplx_joint_cam_wo_ra[smpl_x.neck_idx, None,: ] # face root-relative
# SMPLX pose parameter validity
for name in ('L_Ankle', 'R_Ankle', 'L_Wrist', 'R_Wrist'):
smplx_pose_valid[smpl_x.orig_joints_name.index(name)] = 0
smplx_pose_valid = np.tile(smplx_pose_valid[:,None], (1,3)).reshape(-1)
# SMPLX joint coordinate validity
for name in ('L_Big_toe', 'L_Small_toe', 'L_Heel', 'R_Big_toe', 'R_Small_toe', 'R_Heel'):
smplx_joint_valid[smpl_x.joints_name.index(name)] = 0
smplx_joint_valid = smplx_joint_valid[:,None]
smplx_joint_trunc = smplx_joint_valid * smplx_joint_trunc
smplx_shape_valid = True
# dummy hand/face bbox
dummy_center = np.zeros((2), dtype=np.float32)
dummy_size = np.zeros((2), dtype=np.float32)
inputs = {'img': img}
targets = {'joint_img': smplx_joint_img, 'smplx_joint_img': smplx_joint_img,
'joint_cam': smplx_joint_cam_wo_ra, 'smplx_joint_cam': smplx_joint_cam,
'smplx_pose': smplx_pose, 'smplx_shape': smplx_shape, 'smplx_expr': smplx_expr,
'lhand_bbox_center': dummy_center, 'lhand_bbox_size': dummy_size,
'rhand_bbox_center': dummy_center, 'rhand_bbox_size': dummy_size,
'face_bbox_center': dummy_center, 'face_bbox_size': dummy_size}
meta_info = {'joint_valid': smplx_joint_valid, 'joint_trunc': smplx_joint_trunc,
'smplx_joint_valid': smplx_joint_valid, 'smplx_joint_trunc': smplx_joint_trunc,
'smplx_pose_valid': smplx_pose_valid, 'smplx_shape_valid': float(smplx_shape_valid),
'smplx_expr_valid': float(smplx_expr_valid), 'is_3D': float(True),
'lhand_bbox_valid': float(False), 'rhand_bbox_valid': float(False),
'face_bbox_valid': float(False)}
return inputs, targets, meta_info
else:
inputs = {'img': img}
targets = {}
meta_info = {}
return inputs, targets, meta_info
def evaluate(self, outs, cur_sample_idx):
annots = self.datalist
sample_num = len(outs)
eval_result = {'mpjpe': [], 'pa_mpjpe': []}
for n in range(sample_num):
annot = annots[cur_sample_idx + n]
out = outs[n]
# h36m joint from gt mesh
joint_gt = annot['joint_cam']
joint_gt = joint_gt - joint_gt[self.joint_set['root_joint_idx'],None] # root-relative
joint_gt = joint_gt[self.joint_set['eval_joint'],:]
# h36m joint from param mesh
mesh_out = out['smpl_mesh_cam'] * 1000 # meter to milimeter
joint_out = np.dot(self.joint_set['regressor'], mesh_out) # meter to milimeter
joint_out = joint_out - joint_out[self.joint_set['root_joint_idx'],None] # root-relative
joint_out = joint_out[self.joint_set['eval_joint'],:]
joint_out_aligned = rigid_align(joint_out, joint_gt)
eval_result['mpjpe'].append(np.sqrt(np.sum((joint_out - joint_gt)**2,1)).mean())
eval_result['pa_mpjpe'].append(np.sqrt(np.sum((joint_out_aligned - joint_gt)**2,1)).mean())
vis = False
if vis:
from utils.vis import vis_keypoints, vis_mesh, save_obj
filename = annot['img_path'].split('/')[-1][:-4]
img = load_img(annot['img_path'])[:,:,::-1]
img = vis_mesh(img, mesh_out_img, 0.5)
cv2.imwrite(filename + '.jpg', img)
save_obj(mesh_out, smpl_x.face, filename + '.obj')
return eval_result
def print_eval_result(self, eval_result):
print('MPJPE: %.2f mm' % np.mean(eval_result['mpjpe']))
print('PA MPJPE: %.2f mm' % np.mean(eval_result['pa_mpjpe']))
data/HumanArt/HumanArt.py
View File
data/HumanSC3D/HumanSC3D.py
-57
View File
@@ -1,57 +0,0 @@
import os
import os.path as osp
import numpy as np
import torch
import cv2
import json
import copy
from pycocotools.coco import COCO
from config import cfg
from utils.human_models import smpl_x
from utils.preprocessing import load_img, process_bbox, augmentation, process_db_coord, process_human_model_output, \
get_fitting_error_3D
from utils.transforms import world2cam, cam2pixel, rigid_align
from humandata import HumanDataset
class HumanSC3D(HumanDataset):
def __init__(self, transform, data_split):
super(HumanSC3D, self).__init__(transform, data_split)
self.use_cache = getattr(cfg, 'use_cache', False)
self.annot_path_cache = osp.join(cfg.data_dir, 'cache', 'HumanSC3D_train_230511_2752.npz')
self.img_shape = (900, 900) # (h, w)
self.cam_param = {}
# load data or cache
if self.use_cache and osp.isfile(self.annot_path_cache):
print(f'[{self.__class__.__name__}] loading cache from {self.annot_path_cache}')
self.datalist = self.load_cache(self.annot_path_cache)
else:
if self.use_cache:
print(f'[{self.__class__.__name__}] Cache not found, generating cache...')
self.datalist = []
for pre_prc_file in ['HumanSC3D_train_230511_2752_0.npz','HumanSC3D_train_230511_2752_1.npz',
'HumanSC3D_train_230511_2752_2.npz']:
if self.data_split == 'train':
filename = getattr(cfg, 'filename', pre_prc_file)
else:
raise ValueError('HumanSC3D test set is not support')
self.img_dir = cfg.data_dir # HumanSC3D included
self.annot_path = osp.join(cfg.data_dir, 'preprocessed_datasets', filename)
# check image shape
img_path = osp.join(self.img_dir, np.load(self.annot_path)['image_path'][0])
img_shape = cv2.imread(img_path).shape[:2]
assert self.img_shape == img_shape, 'image shape is incorrect: {} vs {}'.format(self.img_shape, img_shape)
# load data
datalist_slice = self.load_data(
train_sample_interval=getattr(cfg, f'{self.__class__.__name__}_train_sample_interval', 1))
self.datalist.extend(datalist_slice)
if self.use_cache:
self.save_cache(self.annot_path_cache, self.datalist)
data/InstaVariety/InstaVariety.py
-51
View File
@@ -1,51 +0,0 @@
import os
import os.path as osp
import numpy as np
import torch
import cv2
import json
import copy
from pycocotools.coco import COCO
from config import cfg
from utils.human_models import smpl_x
from utils.preprocessing import load_img, process_bbox, augmentation, process_db_coord, process_human_model_output, \
get_fitting_error_3D
from utils.transforms import world2cam, cam2pixel, rigid_align
from humandata import HumanDataset
class InstaVariety(HumanDataset):
def __init__(self, transform, data_split):
super(InstaVariety, self).__init__(transform, data_split)
self.datalist = []
pre_prc_file = 'insta_variety_neural_annot_train.npz'
if self.data_split == 'train':
filename = getattr(cfg, 'filename', pre_prc_file)
else:
raise ValueError('InstaVariety test set is not support')
self.img_dir = osp.join(cfg.data_dir, 'InstaVariety')
self.annot_path = osp.join(cfg.data_dir, 'preprocessed_datasets', filename)
self.annot_path_cache = osp.join(cfg.data_dir, 'cache', filename)
self.use_cache = getattr(cfg, 'use_cache', False)
self.img_shape = (224,224) # (h, w)
self.cam_param = {}
# check image shape
img_path = osp.join(self.img_dir, np.load(self.annot_path)['image_path'][0])
img_shape = cv2.imread(img_path).shape[:2]
assert self.img_shape == img_shape, 'image shape is incorrect: {} vs {}'.format(self.img_shape, img_shape)
# load data or cache
if self.use_cache and osp.isfile(self.annot_path_cache):
print(f'[{self.__class__.__name__}] loading cache from {self.annot_path_cache}')
self.datalist = self.load_cache(self.annot_path_cache)
else:
if self.use_cache:
print(f'[{self.__class__.__name__}] Cache not found, generating cache...')
self.datalist = self.load_data(
train_sample_interval=getattr(cfg, f'{self.__class__.__name__}_train_sample_interval', 1))
if self.use_cache:
self.save_cache(self.annot_path_cache, self.datalist)
data/LSPET/LSPET.py
-43
View File
@@ -1,43 +0,0 @@
import os
import os.path as osp
import numpy as np
import torch
import cv2
import json
import copy
from pycocotools.coco import COCO
from config import cfg
from utils.human_models import smpl_x
from utils.preprocessing import load_img, process_bbox, augmentation, process_db_coord, process_human_model_output, \
get_fitting_error_3D
from utils.transforms import world2cam, cam2pixel, rigid_align
from humandata import HumanDataset
class LSPET(HumanDataset):
def __init__(self, transform, data_split):
super(LSPET, self).__init__(transform, data_split)
if self.data_split == 'train':
filename = getattr(cfg, 'filename', 'eft_lspet.npz')
else:
raise ValueError('LSPET test set is not support')
self.img_dir = osp.join(cfg.data_dir, 'LSPET')
self.annot_path = osp.join(cfg.data_dir, 'preprocessed_datasets', filename)
self.annot_path_cache = osp.join(cfg.data_dir, 'cache', filename)
self.use_cache = getattr(cfg, 'use_cache', False)
self.img_shape = None # (h, w)
self.cam_param = {}
# load data or cache
if self.use_cache and osp.isfile(self.annot_path_cache):
print(f'[{self.__class__.__name__}] loading cache from {self.annot_path_cache}')
self.datalist = self.load_cache(self.annot_path_cache)
else:
if self.use_cache:
print(f'[{self.__class__.__name__}] Cache not found, generating cache...')
self.datalist = self.load_data(
train_sample_interval=getattr(cfg, f'{self.__class__.__name__}_train_sample_interval', 1))
if self.use_cache:
self.save_cache(self.annot_path_cache, self.datalist)
data/MPII/MPII.py
-185
View File
@@ -1,185 +0,0 @@
import os
import os.path as osp
import numpy as np
from config import cfg
import copy
import json
import cv2
import torch
from pycocotools.coco import COCO
from utils.human_models import smpl_x
from utils.preprocessing import load_img, process_bbox, augmentation, process_db_coord, \
process_human_model_output
import random
from humandata import Cache
# from utils.vis import vis_keypoints, vis_mesh, save_obj
class MPII(torch.utils.data.Dataset):
def __init__(self, transform, data_split):
self.transform = transform
self.data_split = data_split
self.img_path = osp.join(cfg.data_dir, 'MPII', 'data')
self.annot_path = osp.join(cfg.data_dir, 'MPII', 'data', 'annotations')
# mpii skeleton
self.joint_set = {
'joint_num': 16,
'joints_name': ('R_Ankle', 'R_Knee', 'R_Hip', 'L_Hip', 'L_Knee', 'L_Ankle', 'Pelvis', 'Thorax', 'Neck', 'Head_top', 'R_Wrist', 'R_Elbow', 'R_Shoulder', 'L_Shoulder', 'L_Elbow', 'L_Wrist'),
'flip_pairs': ( (0,5), (1,4), (2,3), (10,15), (11,14), (12,13) ),
}
# self.datalist = self.load_data()
# load data or cache
self.use_cache = getattr(cfg, 'use_cache', False)
self.annot_path_cache = osp.join(cfg.data_dir, 'cache', f'MPII_{data_split}.npz')
if self.use_cache and osp.isfile(self.annot_path_cache):
print(f'[{self.__class__.__name__}] loading cache from {self.annot_path_cache}')
datalist = Cache(self.annot_path_cache)
assert datalist.data_strategy == getattr(cfg, 'data_strategy', None), \
f'Cache data strategy {datalist.data_strategy} does not match current data strategy ' \
f'{getattr(cfg, "data_strategy", None)}'
self.datalist = datalist
else:
if self.use_cache:
print(f'[{self.__class__.__name__}] Cache not found, generating cache...')
self.datalist = self.load_data()
if self.use_cache:
print(f'[{self.__class__.__name__}] Caching datalist to {self.annot_path_cache}...')
Cache.save(
self.annot_path_cache,
self.datalist,
data_strategy=getattr(cfg, 'data_strategy', None)
)
def load_data(self):
db = COCO(osp.join(self.annot_path, 'train.json'))
with open(osp.join(self.annot_path, 'MPII_train_SMPLX_NeuralAnnot.json')) as f:
smplx_params = json.load(f)
datalist = []
i = 0
for aid in db.anns.keys():
i += 1
if self.data_split == 'train' and i % getattr(cfg, 'MPII_train_sample_interval', 1) != 0:
continue
ann = db.anns[aid]
img = db.loadImgs(ann['image_id'])[0]
imgname = img['file_name']
img_path = osp.join(self.img_path, imgname)
# bbox
bbox = process_bbox(ann['bbox'], img['width'], img['height'], ratio=getattr(cfg, 'bbox_ratio', 1.25))
if bbox is None: continue
# joint coordinates
joint_img = np.array(ann['keypoints'], dtype=np.float32).reshape(-1,3)
joint_valid = joint_img[:,2:].copy()
joint_img[:,2] = 0
# smplx parameter
if str(aid) in smplx_params:
smplx_param = smplx_params[str(aid)]
else:
smplx_param = None
datalist.append({
'img_path': img_path,
'img_shape': (img['height'], img['width']),
'bbox': bbox,
'joint_img': joint_img,
'joint_valid': joint_valid,
'smplx_param': smplx_param
})
if self.data_split == 'train':
print('[MPII train] original size:', len(db.anns.keys()),
'. Sample interval:', getattr(cfg, 'MPII_train_sample_interval', 1),
'. Sampled size:', len(datalist))
if getattr(cfg, 'data_strategy', None) == 'balance' and self.data_split == 'train':
print(f'[MPII] Using [balance] strategy with datalist shuffled...')
random.shuffle(datalist)
return datalist
def __len__(self):
return len(self.datalist)
def __getitem__(self, idx):
data = copy.deepcopy(self.datalist[idx])
img_path, img_shape, bbox = data['img_path'], data['img_shape'], data['bbox']
# image load and affine transform
img = load_img(img_path)
img, img2bb_trans, bb2img_trans, rot, do_flip = augmentation(img, bbox, self.data_split)
img = self.transform(img.astype(np.float32))/255.
# mpii gt
dummy_coord = np.zeros((self.joint_set['joint_num'],3), dtype=np.float32)
joint_img = data['joint_img']
joint_img = np.concatenate((joint_img[:,:2], np.zeros_like(joint_img[:,:1])),1) # x, y, dummy depth
joint_img, joint_cam, joint_cam_ra, joint_valid, joint_trunc = process_db_coord(joint_img, dummy_coord, data['joint_valid'], do_flip, img_shape, self.joint_set['flip_pairs'], img2bb_trans, rot, self.joint_set['joints_name'], smpl_x.joints_name)
# smplx coordinates and parameters
smplx_param = data['smplx_param']
if smplx_param is not None:
smplx_joint_img, smplx_joint_cam, smplx_joint_trunc, smplx_pose, smplx_shape, smplx_expr, smplx_pose_valid, smplx_joint_valid, smplx_expr_valid, smplx_mesh_cam_orig = process_human_model_output(smplx_param['smplx_param'], smplx_param['cam_param'], do_flip, img_shape, img2bb_trans, rot, 'smplx')
is_valid_fit = True
else:
# dummy values
smplx_joint_img = np.zeros((smpl_x.joint_num,3), dtype=np.float32)
smplx_joint_cam = np.zeros((smpl_x.joint_num,3), dtype=np.float32)
smplx_joint_trunc = np.zeros((smpl_x.joint_num,1), dtype=np.float32)
smplx_joint_valid = np.zeros((smpl_x.joint_num), dtype=np.float32)
smplx_pose = np.zeros((smpl_x.orig_joint_num*3), dtype=np.float32)
smplx_shape = np.zeros((smpl_x.shape_param_dim), dtype=np.float32)
smplx_expr = np.zeros((smpl_x.expr_code_dim), dtype=np.float32)
smplx_pose_valid = np.zeros((smpl_x.orig_joint_num), dtype=np.float32)
smplx_expr_valid = False
is_valid_fit = False
# SMPLX pose parameter validity
for name in ('L_Ankle', 'R_Ankle', 'L_Wrist', 'R_Wrist'):
smplx_pose_valid[smpl_x.orig_joints_name.index(name)] = 0
smplx_pose_valid = np.tile(smplx_pose_valid[:,None], (1,3)).reshape(-1)
# SMPLX joint coordinate validity
for name in ('L_Big_toe', 'L_Small_toe', 'L_Heel', 'R_Big_toe', 'R_Small_toe', 'R_Heel'):
smplx_joint_valid[smpl_x.joints_name.index(name)] = 0
smplx_joint_valid = smplx_joint_valid[:,None]
smplx_joint_trunc = smplx_joint_valid * smplx_joint_trunc
# make zero mask for invalid fit
if not is_valid_fit:
smplx_pose_valid[:] = 0
smplx_joint_valid[:] = 0
smplx_joint_trunc[:] = 0
smplx_shape_valid = False
else:
smplx_shape_valid = True
# dummy hand/face bbox
dummy_center = np.zeros((2), dtype=np.float32)
dummy_size = np.zeros((2), dtype=np.float32)
inputs = {'img': img}
targets = {'joint_img': joint_img, 'smplx_joint_img': smplx_joint_img,
'joint_cam': joint_cam, 'smplx_joint_cam': smplx_joint_cam,
'smplx_pose': smplx_pose, 'smplx_shape': smplx_shape, 'smplx_expr': smplx_expr,
'lhand_bbox_center': dummy_center, 'lhand_bbox_size': dummy_size,
'rhand_bbox_center': dummy_center, 'rhand_bbox_size': dummy_size,
'face_bbox_center': dummy_center, 'face_bbox_size': dummy_size}
meta_info = {'joint_valid': joint_valid, 'joint_trunc': joint_trunc,
'smplx_joint_valid': smplx_joint_valid,
'smplx_joint_trunc': smplx_joint_trunc, 'smplx_pose_valid': smplx_pose_valid,
'smplx_shape_valid': float(smplx_shape_valid),
'smplx_expr_valid': float(smplx_expr_valid), 'is_3D': float(False),
'lhand_bbox_valid': float(False), 'rhand_bbox_valid': float(False),
'face_bbox_valid': float(False)}
return inputs, targets, meta_info
data/MPI_INF_3DHP/MPI_INF_3DHP.py
-56
View File
@@ -1,56 +0,0 @@
import os
import os.path as osp
import numpy as np
import torch
import cv2
import json
import copy
from pycocotools.coco import COCO
from config import cfg
from utils.human_models import smpl_x
from utils.preprocessing import load_img, process_bbox, augmentation, process_db_coord, process_human_model_output, \
get_fitting_error_3D
from utils.transforms import world2cam, cam2pixel, rigid_align
from humandata import HumanDataset
class MPI_INF_3DHP(HumanDataset):
def __init__(self, transform, data_split):
super(MPI_INF_3DHP, self).__init__(transform, data_split)
if data_split != 'train':
raise NotImplementedError('MPI_INF_3DHP test set is not supported')
self.use_cache = getattr(cfg, 'use_cache', False)
self.annot_path_cache = osp.join(cfg.data_dir, 'cache', 'mpi_inf_3dhp_neural_annot.npz')
self.img_shape = None # (h, w)
self.cam_param = {}
# load data or cache
if self.use_cache and osp.isfile(self.annot_path_cache):
print(f'[{self.__class__.__name__}] loading cache from {self.annot_path_cache}')
self.datalist = self.load_cache(self.annot_path_cache)
else:
if self.use_cache:
print(f'[{self.__class__.__name__}] Cache not found, generating cache...')
self.datalist = []
for pre_prc_file in ['mpi_inf_3dhp_neural_annot_part1.npz', 'mpi_inf_3dhp_neural_annot_part2.npz',
'mpi_inf_3dhp_neural_annot_part3.npz']:
if self.data_split == 'train':
filename = getattr(cfg, 'filename', pre_prc_file)
else:
raise ValueError('MPI_INF_3DHP test set is not support')
self.img_dir = osp.join(cfg.data_dir, 'MPI_INF_3DHP')
self.annot_path = osp.join(cfg.data_dir, 'preprocessed_datasets', filename)
# load data
datalist_slice = self.load_data(
train_sample_interval=getattr(cfg, f'{self.__class__.__name__}_train_sample_interval', 1))
self.datalist.extend(datalist_slice)
if self.use_cache:
self.save_cache(self.annot_path_cache, self.datalist)
data/MSCOCO/MSCOCO.py
-468
View File
@@ -1,468 +0,0 @@
import os
import os.path as osp
import numpy as np
from config import cfg
import copy
import json
import cv2
import torch
from pycocotools.coco import COCO
from utils.human_models import smpl_x
from utils.preprocessing import load_img, process_bbox, augmentation, process_db_coord, process_human_model_output
import random
from humandata import Cache
class MSCOCO(torch.utils.data.Dataset):
def __init__(self, transform, data_split):
self.transform = transform
self.data_split = data_split
if os.path.exists(osp.join(cfg.data_dir, 'MSCOCO', 'images')):
self.img_path = osp.join(cfg.data_dir, 'MSCOCO', 'images')
self.annot_path = osp.join(cfg.data_dir, 'MSCOCO', 'annotations')
else:
self.img_path = osp.join(cfg.data_dir, 'coco')
self.annot_path = osp.join(cfg.data_dir, 'coco', 'annotations')
# mscoco joint set
self.joint_set = {
'joint_num': 134, # body 24 (23 + pelvis), lhand 21, rhand 21, face 68
'joints_name': \
(
'Nose', 'L_Eye', 'R_Eye', 'L_Ear', 'R_Ear', 'L_Shoulder', 'R_Shoulder', 'L_Elbow', 'R_Elbow', 'L_Wrist',
'R_Wrist', 'L_Hip', 'R_Hip', 'L_Knee', 'R_Knee', 'L_Ankle', 'R_Ankle', 'Pelvis', 'L_Big_toe',
'L_Small_toe', 'L_Heel', 'R_Big_toe', 'R_Small_toe', 'R_Heel', # body part
'L_Wrist_Hand', 'L_Thumb_1', 'L_Thumb_2', 'L_Thumb_3', 'L_Thumb_4', 'L_Index_1', 'L_Index_2',
'L_Index_3', 'L_Index_4', 'L_Middle_1', 'L_Middle_2', 'L_Middle_3', 'L_Middle_4', 'L_Ring_1',
'L_Ring_2', 'L_Ring_3', 'L_Ring_4', 'L_Pinky_1', 'L_Pinky_2', 'L_Pinky_3', 'L_Pinky_4', # left hand
'R_Wrist_Hand', 'R_Thumb_1', 'R_Thumb_2', 'R_Thumb_3', 'R_Thumb_4', 'R_Index_1', 'R_Index_2',
'R_Index_3', 'R_Index_4', 'R_Middle_1', 'R_Middle_2', 'R_Middle_3', 'R_Middle_4', 'R_Ring_1',
'R_Ring_2', 'R_Ring_3', 'R_Ring_4', 'R_Pinky_1', 'R_Pinky_2', 'R_Pinky_3', 'R_Pinky_4', # right hand
*['Face_' + str(i) for i in range(56, 73)], # face contour
*['Face_' + str(i) for i in range(5, 56)] # face
),
'flip_pairs': \
((1, 2), (3, 4), (5, 6), (7, 8), (9, 10), (11, 12), (13, 14), (15, 16), (18, 21), (19, 22), (20, 23),
# body part
(24, 45), (25, 46), (26, 47), (27, 48), (28, 49), (29, 50), (30, 51), (31, 52), (32, 53), (33, 54),
(34, 55), (35, 56), (36, 57), (37, 58), (38, 59), (39, 60), (40, 61), (41, 62), (42, 63), (43, 64),
(44, 65), # hand part
(66, 82), (67, 81), (68, 80), (69, 79), (70, 78), (71, 77), (72, 76), (73, 75), # face contour
(83, 92), (84, 91), (85, 90), (86, 89), (87, 88), # face eyebrow
(97, 101), (98, 100), # face below nose
(102, 111), (103, 110), (104, 109), (105, 108), (106, 113), (107, 112), # face eyes
(114, 120), (115, 119), (116, 118), (121, 125), (122, 124), # face mouth
(126, 130), (127, 129), (131, 133) # face lip
)
}
# self.datalist = self.load_data()
# load data or cache
self.use_cache = getattr(cfg, 'use_cache', False)
self.annot_path_cache = osp.join(cfg.data_dir, 'cache', f'MSCOCO_{data_split}.npz')
if self.use_cache and osp.isfile(self.annot_path_cache):
print(f'[{self.__class__.__name__}] loading cache from {self.annot_path_cache}')
datalist = Cache(self.annot_path_cache)
assert datalist.data_strategy == getattr(cfg, 'data_strategy', None), \
f'Cache data strategy {datalist.data_strategy} does not match current data strategy ' \
f'{getattr(cfg, "data_strategy", None)}'
self.datalist = datalist
else:
if self.use_cache:
print(f'[{self.__class__.__name__}] Cache not found, generating cache...')
self.datalist = self.load_data()
if self.use_cache:
print(f'[{self.__class__.__name__}] Caching datalist to {self.annot_path_cache}...')
Cache.save(
self.annot_path_cache,
self.datalist,
data_strategy=getattr(cfg, 'data_strategy', None)
)
def merge_joint(self, joint_img, feet_img, lhand_img, rhand_img, face_img):
# pelvis
lhip_idx = self.joint_set['joints_name'].index('L_Hip')
rhip_idx = self.joint_set['joints_name'].index('R_Hip')
pelvis = (joint_img[lhip_idx, :] + joint_img[rhip_idx, :]) * 0.5
pelvis[2] = joint_img[lhip_idx, 2] * joint_img[rhip_idx, 2] # joint_valid
pelvis = pelvis.reshape(1, 3)
# feet
lfoot = feet_img[:3, :]
rfoot = feet_img[3:, :]
joint_img = np.concatenate((joint_img, pelvis, lfoot, rfoot, lhand_img, rhand_img, face_img)).astype(
np.float32) # self.joint_set['joint_num'], 3
return joint_img
def load_data(self):
if self.data_split == 'train':
db = COCO(osp.join(self.annot_path, 'coco_wholebody_train_v1.0.json'))
smplx_json_path = osp.join(self.annot_path, 'MSCOCO_train_SMPLX_all_NeuralAnnot.json') # MSCOCO_train_SMPLX.json
with open(smplx_json_path) as f:
print(f'load SMPLX parameters from {smplx_json_path}')
smplx_params = json.load(f)
else:
db = COCO(osp.join(self.annot_path, 'coco_wholebody_val_v1.0.json'))
# train mode
if self.data_split == 'train':
datalist = []
i = 0
for aid in db.anns.keys():
i += 1
if self.data_split == 'train' and i % getattr(cfg, 'MSCOCO_train_sample_interval', 1) != 0:
continue
ann = db.anns[aid]
img = db.loadImgs(ann['image_id'])[0]
imgname = osp.join('train2017', img['file_name'])
img_path = osp.join(self.img_path, imgname)
# exclude the samples that are crowd or have few visible keypoints
if ann['iscrowd'] or (ann['num_keypoints'] == 0): continue
# bbox
bbox = process_bbox(ann['bbox'], img['width'], img['height'], ratio=getattr(cfg, 'bbox_ratio', 1.25))
if bbox is None: continue
# joint coordinates
joint_img = np.array(ann['keypoints'], dtype=np.float32).reshape(-1, 3)
foot_img = np.array(ann['foot_kpts'], dtype=np.float32).reshape(-1, 3)
lhand_img = np.array(ann['lefthand_kpts'], dtype=np.float32).reshape(-1, 3)
rhand_img = np.array(ann['righthand_kpts'], dtype=np.float32).reshape(-1, 3)
face_img = np.array(ann['face_kpts'], dtype=np.float32).reshape(-1, 3)
joint_img = self.merge_joint(joint_img, foot_img, lhand_img, rhand_img, face_img)
joint_valid = (joint_img[:, 2].copy().reshape(-1, 1) > 0).astype(np.float32)
joint_img[:, 2] = 0
# use body annotation to fill hand/face annotation
for body_name, part_name in (
('L_Wrist', 'L_Wrist_Hand'), ('R_Wrist', 'R_Wrist_Hand'), ('Nose', 'Face_18')):
if joint_valid[self.joint_set['joints_name'].index(part_name), 0] == 0:
joint_img[self.joint_set['joints_name'].index(part_name)] = joint_img[
self.joint_set['joints_name'].index(body_name)]
joint_valid[self.joint_set['joints_name'].index(part_name)] = joint_valid[
self.joint_set['joints_name'].index(body_name)]
# hand/face bbox
if ann['lefthand_valid']:
lhand_bbox = np.array(ann['lefthand_box']).reshape(4)
if hasattr(cfg, 'bbox_ratio'):
lhand_bbox = process_bbox(lhand_bbox, img['width'], img['height'], ratio=cfg.bbox_ratio)
if lhand_bbox is not None:
lhand_bbox[2:] += lhand_bbox[:2] # xywh -> xyxy
else:
lhand_bbox = None
if ann['righthand_valid']:
rhand_bbox = np.array(ann['righthand_box']).reshape(4)
if hasattr(cfg, 'bbox_ratio'):
rhand_bbox = process_bbox(rhand_bbox, img['width'], img['height'], ratio=cfg.bbox_ratio)
if rhand_bbox is not None:
rhand_bbox[2:] += rhand_bbox[:2] # xywh -> xyxy
else:
rhand_bbox = None
if ann['face_valid']:
face_bbox = np.array(ann['face_box']).reshape(4)
if hasattr(cfg, 'bbox_ratio'):
face_bbox = process_bbox(face_bbox, img['width'], img['height'], ratio=cfg.bbox_ratio)
if face_bbox is not None:
face_bbox[2:] += face_bbox[:2] # xywh -> xyxy
else:
face_bbox = None
if str(aid) in smplx_params:
smplx_param = smplx_params[str(aid)]
if 'lhand_valid' not in smplx_param['smplx_param']:
smplx_param['smplx_param']['lhand_valid'] = ann['lefthand_valid']
smplx_param['smplx_param']['rhand_valid'] = ann['righthand_valid']
smplx_param['smplx_param']['face_valid'] = ann['face_valid']
else:
smplx_param = None
data_dict = {'img_path': img_path, 'img_shape': (img['height'], img['width']), 'bbox': bbox,
'joint_img': joint_img, 'joint_valid': joint_valid, 'smplx_param': smplx_param,
'lhand_bbox': lhand_bbox, 'rhand_bbox': rhand_bbox, 'face_bbox': face_bbox}
datalist.append(data_dict)
print('[MSCOCO train] original size:', len(db.anns.keys()),
'. Sample interval:', getattr(cfg, 'MSCOCO_train_sample_interval', 1),
'. Sampled size:', len(datalist))
if getattr(cfg, 'data_strategy', None) == 'balance':
print(f"[MSCOCO] Using [balance] strategy with datalist shuffled...")
random.shuffle(datalist)
return datalist
# test mode
else:
datalist = []
for aid in db.anns.keys():
ann = db.anns[aid]
img = db.loadImgs(ann['image_id'])[0]
imgname = osp.join('val2017', img['file_name'])
img_path = osp.join(self.img_path, imgname)
# bbox
bbox = process_bbox(ann['bbox'], img['width'], img['height'], ratio=getattr(cfg, 'bbox_ratio', 1.25))
if bbox is None: continue
# hand/face bbox
if ann['lefthand_valid']:
lhand_bbox = np.array(ann['lefthand_box']).reshape(4)
lhand_bbox[2:] += lhand_bbox[:2] # xywh -> xyxy
else:
lhand_bbox = None
if ann['righthand_valid']:
rhand_bbox = np.array(ann['righthand_box']).reshape(4)
rhand_bbox[2:] += rhand_bbox[:2] # xywh -> xyxy
else:
rhand_bbox = None
if ann['face_valid']:
face_bbox = np.array(ann['face_box']).reshape(4)
face_bbox[2:] += face_bbox[:2] # xywh -> xyxy
else:
face_bbox = None
data_dict = {'img_path': img_path, 'ann_id': aid, 'img_shape': (img['height'], img['width']),
'bbox': bbox, 'lhand_bbox': lhand_bbox, 'rhand_bbox': rhand_bbox, 'face_bbox': face_bbox}
datalist.append(data_dict)
return datalist
def process_hand_face_bbox(self, bbox, do_flip, img_shape, img2bb_trans):
if bbox is None:
bbox = np.array([0, 0, 1, 1], dtype=np.float32).reshape(2, 2) # dummy value
bbox_valid = float(False) # dummy value
else:
# reshape to top-left (x,y) and bottom-right (x,y)
bbox = bbox.reshape(2, 2)
# flip augmentation
if do_flip:
bbox[:, 0] = img_shape[1] - bbox[:, 0] - 1
bbox[0, 0], bbox[1, 0] = bbox[1, 0].copy(), bbox[0, 0].copy() # xmin <-> xmax swap
# make four points of the bbox
bbox = bbox.reshape(4).tolist()
xmin, ymin, xmax, ymax = bbox
bbox = np.array([[xmin, ymin], [xmax, ymin], [xmax, ymax], [xmin, ymax]], dtype=np.float32).reshape(4, 2)
# affine transformation (crop, rotation, scale)
bbox_xy1 = np.concatenate((bbox, np.ones_like(bbox[:, :1])), 1)
bbox = np.dot(img2bb_trans, bbox_xy1.transpose(1, 0)).transpose(1, 0)[:, :2]
bbox[:, 0] = bbox[:, 0] / cfg.input_img_shape[1] * cfg.output_hm_shape[2]
bbox[:, 1] = bbox[:, 1] / cfg.input_img_shape[0] * cfg.output_hm_shape[1]
# make box a rectangle without rotation
xmin = np.min(bbox[:, 0]);
xmax = np.max(bbox[:, 0]);
ymin = np.min(bbox[:, 1]);
ymax = np.max(bbox[:, 1]);
bbox = np.array([xmin, ymin, xmax, ymax], dtype=np.float32)
bbox_valid = float(True)
bbox = bbox.reshape(2, 2)
return bbox, bbox_valid
def __len__(self):
return len(self.datalist)
def __getitem__(self, idx):
data = copy.deepcopy(self.datalist[idx])
# train mode
if self.data_split == 'train':
img_path, img_shape = data['img_path'], data['img_shape']
# image load
img = load_img(img_path)
bbox = data['bbox']
img, img2bb_trans, bb2img_trans, rot, do_flip = augmentation(img, bbox, self.data_split)
img = self.transform(img.astype(np.float32)) / 255.
# hand and face bbox transform
lhand_bbox, lhand_bbox_valid = self.process_hand_face_bbox(data['lhand_bbox'], do_flip, img_shape,
img2bb_trans)
rhand_bbox, rhand_bbox_valid = self.process_hand_face_bbox(data['rhand_bbox'], do_flip, img_shape,
img2bb_trans)
face_bbox, face_bbox_valid = self.process_hand_face_bbox(data['face_bbox'], do_flip, img_shape,
img2bb_trans)
if do_flip:
lhand_bbox, rhand_bbox = rhand_bbox, lhand_bbox
lhand_bbox_valid, rhand_bbox_valid = rhand_bbox_valid, lhand_bbox_valid
lhand_bbox_center = (lhand_bbox[0] + lhand_bbox[1]) / 2.;
rhand_bbox_center = (rhand_bbox[0] + rhand_bbox[1]) / 2.;
face_bbox_center = (face_bbox[0] + face_bbox[1]) / 2.
lhand_bbox_size = lhand_bbox[1] - lhand_bbox[0];
rhand_bbox_size = rhand_bbox[1] - rhand_bbox[0];
face_bbox_size = face_bbox[1] - face_bbox[0];
# coco gt
dummy_coord = np.zeros((self.joint_set['joint_num'], 3), dtype=np.float32)
joint_img = data['joint_img']
joint_img = np.concatenate((joint_img[:, :2], np.zeros_like(joint_img[:, :1])), 1) # x, y, dummy depth
joint_img, joint_cam, joint_cam_ra, joint_valid, joint_trunc = process_db_coord(joint_img, dummy_coord,
data['joint_valid'], do_flip, img_shape,
self.joint_set['flip_pairs'],
img2bb_trans, rot,
self.joint_set['joints_name'],
smpl_x.joints_name)
# smplx coordinates and parameters
smplx_param = data['smplx_param']
if smplx_param is not None:
smplx_joint_img, smplx_joint_cam, smplx_joint_trunc, smplx_pose, smplx_shape, smplx_expr, smplx_pose_valid, smplx_joint_valid, smplx_expr_valid, smplx_mesh_cam_orig \
= process_human_model_output(smplx_param['smplx_param'], smplx_param['cam_param'], do_flip,
img_shape, img2bb_trans, rot, 'smplx')
is_valid_fit = True
else:
# dummy values
smplx_joint_img = np.zeros((smpl_x.joint_num, 3), dtype=np.float32)
smplx_joint_cam = np.zeros((smpl_x.joint_num, 3), dtype=np.float32)
smplx_joint_trunc = np.zeros((smpl_x.joint_num, 1), dtype=np.float32)
smplx_joint_valid = np.zeros((smpl_x.joint_num), dtype=np.float32)
smplx_pose = np.zeros((smpl_x.orig_joint_num * 3), dtype=np.float32)
smplx_shape = np.zeros((smpl_x.shape_param_dim), dtype=np.float32)
smplx_expr = np.zeros((smpl_x.expr_code_dim), dtype=np.float32)
smplx_pose_valid = np.zeros((smpl_x.orig_joint_num), dtype=np.float32)
smplx_expr_valid = False
is_valid_fit = False
# SMPLX pose parameter validity
smplx_pose_valid = np.tile(smplx_pose_valid[:, None], (1, 3)).reshape(-1)
# SMPLX joint coordinate validity
smplx_joint_valid = smplx_joint_valid[:, None]
smplx_joint_trunc = smplx_joint_valid * smplx_joint_trunc
# make zero mask for invalid fit
if not is_valid_fit:
smplx_pose_valid[:] = 0
smplx_joint_valid[:] = 0
smplx_joint_trunc[:] = 0
smplx_shape_valid = False
else:
smplx_shape_valid = True
inputs = {'img': img}
targets = {'joint_img': joint_img, 'joint_cam': joint_cam, 'smplx_joint_img': smplx_joint_img,
'smplx_joint_cam': smplx_joint_cam,
'smplx_pose': smplx_pose, 'smplx_shape': smplx_shape, 'smplx_expr': smplx_expr,
'lhand_bbox_center': lhand_bbox_center,
'lhand_bbox_size': lhand_bbox_size, 'rhand_bbox_center': rhand_bbox_center,
'rhand_bbox_size': rhand_bbox_size,
'face_bbox_center': face_bbox_center, 'face_bbox_size': face_bbox_size}
meta_info = {'joint_valid': joint_valid, 'joint_trunc': joint_trunc, 'smplx_joint_valid': smplx_joint_valid,
'smplx_joint_trunc': smplx_joint_trunc,
'smplx_pose_valid': smplx_pose_valid, 'smplx_shape_valid': float(smplx_shape_valid),
'smplx_expr_valid': float(smplx_expr_valid), 'is_3D': float(False),
# 'lhand_bbox_valid': float(False), 'rhand_bbox_valid': float(False),
# 'face_bbox_valid': float(False)}
'lhand_bbox_valid': lhand_bbox_valid,
'rhand_bbox_valid': rhand_bbox_valid, 'face_bbox_valid': face_bbox_valid}
return inputs, targets, meta_info
# test mode
else:
img_path, img_shape = data['img_path'], data['img_shape']
# image load
img = load_img(img_path)
bbox = data['bbox']
img, img2bb_trans, bb2img_trans, rot, do_flip = augmentation(img, bbox, self.data_split)
img = self.transform(img.astype(np.float32)) / 255.
inputs = {'img': img}
targets = {}
meta_info = {'bb2img_trans': bb2img_trans}
return inputs, targets, meta_info
def evaluate(self, outs, cur_sample_idx):
annots = self.datalist
sample_num = len(outs)
for n in range(sample_num):
annot = annots[cur_sample_idx + n]
ann_id = annot['ann_id']
out = outs[n]
if annot['lhand_bbox'] is not None:
lhand_bbox = out['lhand_bbox'].copy().reshape(2, 2)
lhand_bbox = np.concatenate((lhand_bbox, np.ones((2, 1))), 1)
lhand_bbox = np.dot(out['bb2img_trans'], lhand_bbox.transpose(1, 0)).transpose(1, 0)[:, :2]
if annot['rhand_bbox'] is not None:
rhand_bbox = out['rhand_bbox'].copy().reshape(2, 2)
rhand_bbox = np.concatenate((rhand_bbox, np.ones((2, 1))), 1)
rhand_bbox = np.dot(out['bb2img_trans'], rhand_bbox.transpose(1, 0)).transpose(1, 0)[:, :2]
if annot['face_bbox'] is not None:
face_bbox = out['face_bbox'].copy().reshape(2, 2)
face_bbox = np.concatenate((face_bbox, np.ones((2, 1))), 1)
face_bbox = np.dot(out['bb2img_trans'], face_bbox.transpose(1, 0)).transpose(1, 0)[:, :2]
vis = False
if vis:
img_path = annot['img_path']
"""
img = (out['img'].transpose(1,2,0)[:,:,::-1] * 255).copy()
joint_img = out['joint_img'].copy()
joint_img[:,0] = joint_img[:,0] / cfg.output_hm_shape[2] * cfg.input_img_shape[1]
joint_img[:,1] = joint_img[:,1] / cfg.output_hm_shape[1] * cfg.input_img_shape[0]
for j in range(len(joint_img)):
if j in smpl_x.pos_joint_part['body']:
cv2.circle(img, (int(joint_img[j][0]), int(joint_img[j][1])), 3, (0,0,255), -1)
lhand_bbox = out['lhand_bbox'].reshape(2,2).copy()
cv2.rectangle(img, (int(lhand_bbox[0][0]), int(lhand_bbox[0][1])), (int(lhand_bbox[1][0]), int(lhand_bbox[1][1])), (255,0,0), 3)
rhand_bbox = out['rhand_bbox'].reshape(2,2).copy()
cv2.rectangle(img, (int(rhand_bbox[0][0]), int(rhand_bbox[0][1])), (int(rhand_bbox[1][0]), int(rhand_bbox[1][1])), (255,0,0), 3)
face_bbox = out['face_bbox'].reshape(2,2).copy()
cv2.rectangle(img, (int(face_bbox[0][0]), int(face_bbox[0][1])), (int(face_bbox[1][0]), int(face_bbox[1][1])), (255,0,0), 3)
cv2.imwrite(str(ann_id) + '.jpg', img)
"""
# save_obj(out['smplx_mesh_cam'], smpl_x.face, img_id + '_' + str(ann_id) + '.obj')
"""
img = load_img(img_path)[:,:,::-1]
bbox = annot['bbox']
focal = list(cfg.focal)
princpt = list(cfg.princpt)
focal[0] = focal[0] / cfg.input_body_shape[1] * bbox[2]
focal[1] = focal[1] / cfg.input_body_shape[0] * bbox[3]
princpt[0] = princpt[0] / cfg.input_body_shape[1] * bbox[2] + bbox[0]
princpt[1] = princpt[1] / cfg.input_body_shape[0] * bbox[3] + bbox[1]
img = render_mesh(img, out['smplx_mesh_cam'], smpl_x.face, {'focal': focal, 'princpt': princpt})
#img = cv2.resize(img, (512,512))
cv2.imwrite(img_id + '_' + str(ann_id) + '.jpg', img)
"""
bbox = annot['bbox']
focal = list(cfg.focal)
princpt = list(cfg.princpt)
focal[0] = focal[0] / cfg.input_body_shape[1] * bbox[2]
focal[1] = focal[1] / cfg.input_body_shape[0] * bbox[3]
princpt[0] = princpt[0] / cfg.input_body_shape[1] * bbox[2] + bbox[0]
princpt[1] = princpt[1] / cfg.input_body_shape[0] * bbox[3] + bbox[1]
param_save = {'smplx_param': {'root_pose': out['smplx_root_pose'].tolist(),
'body_pose': out['smplx_body_pose'].tolist(),
'lhand_pose': out['smplx_lhand_pose'].tolist(),
'rhand_pose': out['smplx_rhand_pose'].tolist(),
'jaw_pose': out['smplx_jaw_pose'].tolist(),
'shape': out['smplx_shape'].tolist(), 'expr': out['smplx_expr'].tolist(),
'trans': out['cam_trans'].tolist()},
'cam_param': {'focal': focal, 'princpt': princpt}
}
with open(str(ann_id) + '.json', 'w') as f:
json.dump(param_save, f)
return {}
def print_eval_result(self, eval_result):
return
data/MTP/MTP.py
-45
View File
@@ -1,45 +0,0 @@
import os
import os.path as osp
import numpy as np
import torch
import cv2
import json
import copy
from pycocotools.coco import COCO
from config import cfg
from utils.human_models import smpl_x
from utils.preprocessing import load_img, process_bbox, augmentation, process_db_coord, process_human_model_output, \
get_fitting_error_3D
from utils.transforms import world2cam, cam2pixel, rigid_align
from humandata import HumanDataset
class MTP(HumanDataset):
def __init__(self, transform, data_split):
super(MTP, self).__init__(transform, data_split)
pre_prc_file = 'mtp_smplx_train.npz'
if self.data_split == 'train':
filename = getattr(cfg, 'filename', pre_prc_file)
else:
raise ValueError('MTP test set is not support')
self.img_dir = osp.join(cfg.data_dir, 'MTP')
self.annot_path = osp.join(cfg.data_dir, 'preprocessed_datasets', filename)
self.annot_path_cache = osp.join(cfg.data_dir, 'cache', filename)
self.use_cache = getattr(cfg, 'use_cache', False)
self.img_shape = None
self.cam_param = {}
print("Various image shape in MTP dataset.")
# load data or cache
if self.use_cache and osp.isfile(self.annot_path_cache):
print(f'[{self.__class__.__name__}] loading cache from {self.annot_path_cache}')
self.datalist = self.load_cache(self.annot_path_cache)
else:
if self.use_cache:
print(f'[{self.__class__.__name__}] Cache not found, generating cache...')
self.datalist = self.load_data(
train_sample_interval=getattr(cfg, f'{self.__class__.__name__}_train_sample_interval', 1))
if self.use_cache:
self.save_cache(self.annot_path_cache, self.datalist)
data/MuCo/MuCo.py
-48
View File
@@ -1,48 +0,0 @@
import os
import os.path as osp
import numpy as np
import torch
import cv2
import json
import copy
from pycocotools.coco import COCO
from config import cfg
from utils.human_models import smpl_x
from utils.preprocessing import load_img, process_bbox, augmentation, process_db_coord, process_human_model_output, \
get_fitting_error_3D
from utils.transforms import world2cam, cam2pixel, rigid_align
from humandata import HumanDataset
class MuCo(HumanDataset):
def __init__(self, transform, data_split):
super(MuCo, self).__init__(transform, data_split)
if self.data_split == 'train':
filename = getattr(cfg, 'filename', 'muco3dhp_train.npz')
else:
raise ValueError('MoCo test set is not support')
self.img_dir = osp.join(cfg.data_dir, 'MuCo')
self.annot_path = osp.join(cfg.data_dir, 'preprocessed_datasets', filename)
self.annot_path_cache = osp.join(cfg.data_dir, 'cache', filename)
self.use_cache = getattr(cfg, 'use_cache', False)
self.img_shape = (1024, 1024) # (h, w)
self.cam_param = {}
# check image shape
img_path = osp.join(self.img_dir, np.load(self.annot_path)['image_path'][0])
img_shape = cv2.imread(img_path).shape[:2]
assert self.img_shape == img_shape, 'image shape is incorrect: {} vs {}'.format(self.img_shape, img_shape)
# load data or cache
if self.use_cache and osp.isfile(self.annot_path_cache):
print(f'[{self.__class__.__name__}] loading cache from {self.annot_path_cache}')
self.datalist = self.load_cache(self.annot_path_cache)
else:
if self.use_cache:
print(f'[{self.__class__.__name__}] Cache not found, generating cache...')
self.datalist = self.load_data(
train_sample_interval=getattr(cfg, f'{self.__class__.__name__}_train_sample_interval', 1))
if self.use_cache:
self.save_cache(self.annot_path_cache, self.datalist)
data/OCHuman/OCHuman.py
-47
View File
@@ -1,47 +0,0 @@
import os
import os.path as osp
import numpy as np
import torch
import cv2
import json
import copy
from pycocotools.coco import COCO
from config import cfg
from utils.human_models import smpl_x
from utils.preprocessing import load_img, process_bbox, augmentation, process_db_coord, process_human_model_output, \
get_fitting_error_3D
from utils.transforms import world2cam, cam2pixel, rigid_align
from humandata import HumanDataset
class OCHuman(HumanDataset):
def __init__(self, transform, data_split):
super(OCHuman, self).__init__(transform, data_split)
self.datalist = []
pre_prc_file = 'eft_ochuman.npz'
if self.data_split == 'train':
filename = getattr(cfg, 'filename', pre_prc_file)
else:
raise ValueError('OCHuman test set is not support')
self.img_dir = osp.join(cfg.data_dir, 'OCHuman')
self.annot_path = osp.join(cfg.data_dir, 'preprocessed_datasets', filename)
self.annot_path_cache = osp.join(cfg.data_dir, 'cache', filename)
self.use_cache = getattr(cfg, 'use_cache', False)
self.img_shape = None
self.cam_param = {}
print("Various image shape in OCHuman dataset.")
# load data or cache
if self.use_cache and osp.isfile(self.annot_path_cache):
print(f'[{self.__class__.__name__}] loading cache from {self.annot_path_cache}')
self.datalist = self.load_cache(self.annot_path_cache)
else:
if self.use_cache:
print(f'[{self.__class__.__name__}] Cache not found, generating cache...')
self.datalist = self.load_data(
train_sample_interval=getattr(cfg, f'{self.__class__.__name__}_train_sample_interval', 1))
if self.use_cache:
self.save_cache(self.annot_path_cache, self.datalist)
data/PROX/PROX.py
-48
View File
@@ -1,48 +0,0 @@
import os
import os.path as osp
import numpy as np
import torch
import cv2
import json
import copy
from pycocotools.coco import COCO
from config import cfg
from utils.human_models import smpl_x
from utils.preprocessing import load_img, process_bbox, augmentation, process_db_coord, process_human_model_output, \
get_fitting_error_3D
from utils.transforms import world2cam, cam2pixel, rigid_align
from humandata import HumanDataset
class PROX(HumanDataset):
def __init__(self, transform, data_split):
super(PROX, self).__init__(transform, data_split)
if self.data_split == 'train':
filename = getattr(cfg, 'filename', 'prox_train_smplx_new.npz')
else:
raise ValueError('PROX test set is not support')
self.img_dir = osp.join(cfg.data_dir, 'PROX')
self.annot_path = osp.join(cfg.data_dir, 'preprocessed_datasets', filename)
self.annot_path_cache = osp.join(cfg.data_dir, 'cache', filename)
self.use_cache = getattr(cfg, 'use_cache', False)
self.img_shape = (1080, 1920) # (h, w)
self.cam_param = {}
# check image shape
img_path = osp.join(self.img_dir, np.load(self.annot_path)['image_path'][0])
img_shape = cv2.imread(img_path).shape[:2]
assert self.img_shape == img_shape, 'image shape is incorrect: {} vs {}'.format(self.img_shape, img_shape)
# load data or cache
if self.use_cache and osp.isfile(self.annot_path_cache):
print(f'[{self.__class__.__name__}] loading cache from {self.annot_path_cache}')
self.datalist = self.load_cache(self.annot_path_cache)
else:
if self.use_cache:
print(f'[{self.__class__.__name__}] Cache not found, generating cache...')
self.datalist = self.load_data(
train_sample_interval=getattr(cfg, f'{self.__class__.__name__}_train_sample_interval', 1))
if self.use_cache:
self.save_cache(self.annot_path_cache, self.datalist)
data/PW3D/PW3D.py
-223
View File
@@ -1,223 +0,0 @@
import os
import os.path as osp
import numpy as np
import torch
import cv2
import json
import copy
from pycocotools.coco import COCO
from config import cfg
from utils.human_models import smpl_x, smpl
from utils.preprocessing import load_img, process_bbox, augmentation, process_human_model_output, process_db_coord
from utils.transforms import rigid_align
import numpy as np
import random
class PW3D(torch.utils.data.Dataset):
def __init__(self, transform, data_split):
self.transform = transform
self.data_split = data_split
self.data_path = osp.join(cfg.data_dir, 'PW3D', 'data')
# 3dpw skeleton
self.joint_set = {
'joint_num': smpl_x.joint_num,
'joints_name': smpl_x.joints_name,
'flip_pairs': smpl_x.flip_pairs}
self.datalist = self.load_data()
def load_data(self):
db = COCO(osp.join(self.data_path, '3DPW_' + self.data_split + '.json'))
datalist = []
i = 0
if getattr(cfg, 'eval_on_train', False):
self.data_split = 'eval_train'
print("Evaluate on train set.")
for aid in db.anns.keys():
i += 1
if 'train' in self.data_split and i % getattr(cfg, 'PW3D_train_sample_interval', 1) != 0:
continue
ann = db.anns[aid]
image_id = ann['image_id']
img = db.loadImgs(image_id)[0]
sequence_name = img['sequence']
img_name = img['file_name']
img_path = osp.join(self.data_path, 'imageFiles', sequence_name, img_name)
cam_param = {k: np.array(v, dtype=np.float32) for k,v in img['cam_param'].items()}
smpl_param = ann['smpl_param']
bbox = process_bbox(np.array(ann['bbox']), img['width'], img['height'], ratio=getattr(cfg, 'bbox_ratio', 1.25))
if bbox is None: continue
data_dict = {'img_path': img_path, 'ann_id': aid, 'img_shape': (img['height'], img['width']),
'bbox': bbox, 'smpl_param': smpl_param, 'cam_param': cam_param}
datalist.append(data_dict)
if self.data_split == 'train':
print('[PW3D train] original size:', len(db.anns.keys()),
'. Sample interval:', getattr(cfg, 'PW3D_train_sample_interval', 1),
'. Sampled size:', len(datalist))
if (getattr(cfg, 'data_strategy', None) == 'balance' and self.data_split == 'train') or \
self.data_split == 'eval_train':
print(f'[PW3D] Using [balance] strategy with datalist shuffled...')
random.seed(2023)
random.shuffle(datalist)
if self.data_split == "eval_train":
return datalist[:10000]
return datalist
def __len__(self):
return len(self.datalist)
def __getitem__(self, idx):
data = copy.deepcopy(self.datalist[idx])
img_path, img_shape = data['img_path'], data['img_shape']
# img
img = load_img(img_path)
bbox, smpl_param, cam_param = data['bbox'], data['smpl_param'], data['cam_param']
img, img2bb_trans, bb2img_trans, rot, do_flip = augmentation(img, bbox, self.data_split)
img = self.transform(img.astype(np.float32))/255.
cam_param = data['cam_param']
if self.data_split == 'train':
smplx_param = {}
smplx_param['root_pose'] = np.array(smpl_param['pose']).reshape(-1,3)[:1, :]
smplx_param['body_pose'] = np.array(smpl_param['pose']).reshape(-1,3)[1:22, :]
smplx_param['trans'] = np.array(smpl_param['trans']).reshape(-1,3)
smplx_param['shape'] = np.zeros(10, dtype=np.float32) # drop smpl betas for smplx
# smpl coordinates
smplx_joint_img, smplx_joint_cam, smplx_joint_trunc, smplx_pose, smplx_shape, smplx_expr, \
smplx_pose_valid, smplx_joint_valid, smplx_expr_valid, smplx_mesh_cam_orig = process_human_model_output(
smplx_param, cam_param, do_flip, img_shape, img2bb_trans, rot, 'smplx',
joint_img=None)
# reverse ra
smplx_joint_cam_wo_ra = smplx_joint_cam.copy()
smplx_joint_cam_wo_ra[smpl_x.joint_part['lhand'], :] = smplx_joint_cam_wo_ra[smpl_x.joint_part['lhand'], :] \
+ smplx_joint_cam_wo_ra[smpl_x.lwrist_idx, None, :] # left hand root-relative
smplx_joint_cam_wo_ra[smpl_x.joint_part['rhand'], :] = smplx_joint_cam_wo_ra[smpl_x.joint_part['rhand'], :] \
+ smplx_joint_cam_wo_ra[smpl_x.rwrist_idx, None, :] # right hand root-relative
smplx_joint_cam_wo_ra[smpl_x.joint_part['face'], :] = smplx_joint_cam_wo_ra[smpl_x.joint_part['face'], :] \
+ smplx_joint_cam_wo_ra[smpl_x.neck_idx, None,: ] # face root-relative
smplx_pose_valid = np.tile(smplx_pose_valid[:, None], (1, 3)).reshape(-1)
smplx_joint_valid = smplx_joint_valid[:, None]
smplx_joint_trunc = smplx_joint_valid * smplx_joint_trunc
# smpl coordinates
smpl_joint_img, _, _, _, _, _ = process_human_model_output(
smpl_param, cam_param, do_flip, img_shape, img2bb_trans, rot, 'smpl',
joint_img=None)
joint_img = np.zeros_like(smplx_joint_img)
joint_img[:22] = smpl_joint_img[:22, :]
# dummy hand/face bbox
dummy_center = np.zeros((2), dtype=np.float32)
dummy_size = np.zeros((2), dtype=np.float32)
inputs = {'img': img}
targets = {'joint_img': joint_img, 'smplx_joint_img': joint_img,
'joint_cam': smplx_joint_cam_wo_ra, 'smplx_joint_cam': smplx_joint_cam,
'smplx_pose': smplx_pose, 'smplx_shape': smplx_shape, 'smplx_expr': smplx_expr,
'lhand_bbox_center': dummy_center, 'lhand_bbox_size': dummy_size,
'rhand_bbox_center': dummy_center, 'rhand_bbox_size': dummy_size,
'face_bbox_center': dummy_center, 'face_bbox_size': dummy_size}
meta_info = {'joint_valid': smplx_joint_valid, 'joint_trunc': smplx_joint_trunc,
'smplx_joint_valid': smplx_joint_valid, 'smplx_joint_trunc': smplx_joint_trunc,
'smplx_pose_valid': smplx_pose_valid, 'smplx_shape_valid': float(False),
'smplx_expr_valid': float(smplx_expr_valid), 'is_3D': float(True),
'lhand_bbox_valid': float(False), 'rhand_bbox_valid': float(False),
'face_bbox_valid': float(False)}
return inputs, targets, meta_info
else:
# smpl coordinates
smpl_joint_img, smpl_joint_cam, smpl_joint_trunc, smpl_pose, smpl_shape, smpl_mesh_cam_orig = process_human_model_output(smpl_param, cam_param, do_flip, img_shape, img2bb_trans, rot, 'smpl')
inputs = {'img': img}
targets = {'smpl_mesh_cam': smpl_mesh_cam_orig}
meta_info = {}
return inputs, targets, meta_info
def evaluate(self, outs, cur_sample_idx):
annots = self.datalist
sample_num = len(outs)
eval_result = {'mpjpe_body': [], 'pa_mpjpe_body': [], }
## smpl/smplx -> lsp
# ['left_hip', 'right_hip', 'left_knee', 'right_knee', 'left_ankle',
# 'right_ankle', 'neck', 'head', 'left_shoulder', 'right_shoulder',
# 'left_elbow', 'right_elbow', 'left_wrist', 'right_wrist']
joint_mapper = [1, 2, 4, 5, 7, 8, 12, 15, 16, 17, 18, 19, 20, 21]
for n in range(sample_num):
out = outs[n]
# MPVPE from all vertices
mesh_gt = out['smpl_mesh_cam_target']
mesh_out = out['smplx_mesh_cam']
# MPJPE from body joints
mesh_out_align = mesh_out - np.dot(smpl_x.J_regressor, mesh_out)[smpl_x.J_regressor_idx['pelvis'], None, :] \
+ np.dot(smpl.joint_regressor, mesh_gt)[smpl.root_joint_idx, None, :]
# only test 14 keypoints
joint_gt_body = np.dot(smpl.joint_regressor, mesh_gt)[joint_mapper, :]
joint_out_body = np.dot(smpl_x.J_regressor, mesh_out)[joint_mapper, :]
joint_out_body_root_align = np.dot(smpl_x.J_regressor, mesh_out_align)[joint_mapper, :]
eval_result['mpjpe_body'].append(
np.sqrt(np.sum((joint_out_body_root_align - joint_gt_body) ** 2, 1)).mean() * 1000)
# PAMPJPE from body joints
joint_out_body_pa_align = rigid_align(joint_out_body, joint_gt_body)
eval_result['pa_mpjpe_body'].append(
np.sqrt(np.sum((joint_out_body_pa_align - joint_gt_body) ** 2, 1)).mean() * 1000)
return eval_result
def print_eval_result(self, eval_result):
print('======3DPW-test======')
print('MPJPE (Body): %.2f mm' % np.mean(eval_result['mpjpe_body']))
print('PA MPJPE (Body): %.2f mm' % np.mean(eval_result['pa_mpjpe_body']))
print()
print(f"{np.mean(eval_result['mpjpe_body'])},{np.mean(eval_result['pa_mpjpe_body'])}")
print()
f = open(os.path.join(cfg.result_dir, 'result.txt'), 'w')
f.write(f'3DPW-test dataset: \n')
f.write('MPJPE (Body): %.2f mm\n' % np.mean(eval_result['mpjpe_body']))
f.write('PA MPJPE (Body): %.2f mm\n' % np.mean(eval_result['pa_mpjpe_body']))
f.write(f"{np.mean(eval_result['mpjpe_body'])},{np.mean(eval_result['pa_mpjpe_body'])}")
if getattr(cfg, 'eval_on_train', False):
import csv
csv_file = f'{cfg.root_dir}/output/pw3d_eval_on_train.csv'
exp_id = cfg.exp_name.split('_')[1]
new_line = [exp_id,np.mean(eval_result['mpjpe_body']), np.mean(eval_result['pa_mpjpe_body'])]
# Append the new line to the CSV file
with open(csv_file, 'a', newline='') as file:
writer = csv.writer(file)
writer.writerow(new_line)
data/PoseTrack/PoseTrack.py
-47
View File
@@ -1,47 +0,0 @@
import os
import os.path as osp
import numpy as np
import torch
import cv2
import json
import copy
from pycocotools.coco import COCO
from config import cfg
from utils.human_models import smpl_x
from utils.preprocessing import load_img, process_bbox, augmentation, process_db_coord, process_human_model_output, \
get_fitting_error_3D
from utils.transforms import world2cam, cam2pixel, rigid_align
from humandata import HumanDataset
class PoseTrack(HumanDataset):
def __init__(self, transform, data_split):
super(PoseTrack, self).__init__(transform, data_split)
self.datalist = []
pre_prc_file = 'eft_posetrack.npz'
if self.data_split == 'train':
filename = getattr(cfg, 'filename', pre_prc_file)
else:
raise ValueError('PoseTrack test set is not support')
self.img_dir = osp.join(cfg.data_dir, 'PoseTrack/data/images')
self.annot_path = osp.join(cfg.data_dir, 'preprocessed_datasets', filename)
self.annot_path_cache = osp.join(cfg.data_dir, 'cache', filename)
self.use_cache = getattr(cfg, 'use_cache', False)
self.img_shape = None
self.cam_param = {}
print("Various image shape in PoseTrack dataset.")
# load data or cache
if self.use_cache and osp.isfile(self.annot_path_cache):
print('loading cache from {}'.format(self.annot_path_cache))
self.datalist = self.load_cache(self.annot_path_cache)
else:
if self.use_cache:
print(f'[{self.__class__.__name__}] Cache not found, generating cache...')
self.datalist = self.load_data(
train_sample_interval=getattr(cfg, f'{self.__class__.__name__}_train_sample_interval', 1))
if self.use_cache:
self.save_cache(self.annot_path_cache, self.datalist)
data/RICH/RICH.py
-46
View File
@@ -1,46 +0,0 @@
import os
import os.path as osp
import numpy as np
import torch
import cv2
import json
import copy
from pycocotools.coco import COCO
from config import cfg
from utils.human_models import smpl_x
from utils.preprocessing import load_img, process_bbox, augmentation, process_db_coord, process_human_model_output, \
get_fitting_error_3D
from utils.transforms import world2cam, cam2pixel, rigid_align
from humandata import HumanDataset
class RICH(HumanDataset):
def __init__(self, transform, data_split):
super(RICH, self).__init__(transform, data_split)
self.use_cache = getattr(cfg, 'use_cache', False)
self.annot_path_cache = osp.join(cfg.data_dir, 'cache', 'rich_train_fix_betas.npz')
self.img_shape = None # (h, w)
self.cam_param = {}
if self.use_cache and osp.isfile(self.annot_path_cache):
print(f'[{self.__class__.__name__}] loading cache from {self.annot_path_cache}')
self.datalist = self.load_cache(self.annot_path_cache)
else:
if self.use_cache:
print(f'[{self.__class__.__name__}] Cache not found, generating cache...')
if self.data_split == 'train':
filename = getattr(cfg, 'filename', 'rich_train_fix_betas.npz')
else:
raise ValueError('RICH test set is not support')
self.img_dir = osp.join(cfg.data_dir, 'RICH')
self.annot_path = osp.join(cfg.data_dir, 'preprocessed_datasets', filename)
# load data
self.datalist = self.load_data(
train_sample_interval=getattr(cfg, f'{self.__class__.__name__}_train_sample_interval', 1))
if self.use_cache:
self.save_cache(self.annot_path_cache, self.datalist)
data/RenBody/RenBody.py
-60
View File
@@ -1,60 +0,0 @@
import os
import os.path as osp
import numpy as np
import torch
import cv2
import json
import copy
from pycocotools.coco import COCO
from config import cfg
from utils.human_models import smpl_x
from utils.preprocessing import load_img, process_bbox, augmentation, process_db_coord, process_human_model_output, \
get_fitting_error_3D
from utils.transforms import world2cam, cam2pixel, rigid_align
from humandata import HumanDataset
# issue: 4 IndexError: index 432000 is out of bounds for axis 0 with size 432000 (bbox = bbox_xywh[i][:4])
class RenBody(HumanDataset):
def __init__(self, transform, data_split):
super(RenBody, self).__init__(transform, data_split)
self.use_cache = getattr(cfg, 'use_cache', False)
if self.data_split == 'train':
self.annot_path_cache = osp.join(cfg.data_dir, 'cache', 'renbody_train_230525_399_ds10_fix_betas.npz')
else:
self.annot_path_cache = osp.join(cfg.data_dir, 'cache', 'renbody_test_230525_78_ds10_fix_betas.npz')
self.img_shape = None # (h, w)
self.cam_param = {}
if self.use_cache and osp.isfile(self.annot_path_cache):
print(f'[{self.__class__.__name__}] loading cache from {self.annot_path_cache}')
self.datalist = self.load_cache(self.annot_path_cache)
else:
if self.use_cache:
print(f'[{self.__class__.__name__}] Cache not found, generating cache...')
# load data or cache
self.datalist = []
for idx in range(10):
if self.data_split == 'train':
pre_prc_file_train = f'renbody_train_230525_399_{idx}.npz'
filename = getattr(cfg, 'filename', pre_prc_file_train)
else:
if idx > 1: continue
pre_prc_file_test = f'renbody_test_230525_78_{idx}.npz'
filename = getattr(cfg, 'filename', pre_prc_file_test)
self.img_dir = osp.join(cfg.data_dir, 'RenBody')
self.annot_path = osp.join(cfg.data_dir, 'preprocessed_datasets', filename)
# load data
datalist_slice = self.load_data(
train_sample_interval=getattr(cfg, f'{self.__class__.__name__}_train_sample_interval', 1),
test_sample_interval=getattr(cfg, f'{self.__class__.__name__}_test_sample_interval', 1))
self.datalist.extend(datalist_slice)
if self.use_cache:
self.save_cache(self.annot_path_cache, self.datalist)
data/RenBody_HiRes/RenBody_HiRes.py
-61
View File
@@ -1,61 +0,0 @@
import os
import os.path as osp
import numpy as np
import torch
import cv2
import json
import copy
from pycocotools.coco import COCO
from config import cfg
from utils.human_models import smpl_x
from utils.preprocessing import load_img, process_bbox, augmentation, process_db_coord, process_human_model_output, \
get_fitting_error_3D
from utils.transforms import world2cam, cam2pixel, rigid_align
from humandata import HumanDataset
class RenBody_HiRes(HumanDataset):
def __init__(self, transform, data_split):
super(RenBody_HiRes, self).__init__(transform, data_split)
self.datalist = []
if getattr(cfg, 'eval_on_train', False):
self.data_split = 'eval_train'
print("Evaluate on train set.")
self.use_cache = getattr(cfg, 'use_cache', False)
self.annot_path_cache = osp.join(cfg.data_dir, 'cache', f'renbody_{self.data_split}_highrescam_230517_399_fix_betas.npz')
self.img_shape = None # (h, w)
self.cam_param = {}
# load data or cache
if self.use_cache and osp.isfile(self.annot_path_cache):
print(f'[{self.__class__.__name__}] loading cache from {self.annot_path_cache}')
self.datalist = self.load_cache(self.annot_path_cache)
else:
if self.use_cache:
print(f'[{self.__class__.__name__}] Cache not found, generating cache...')
for idx in range(2):
if 'train' in self.data_split:
pre_prc_file_train = f'renbody_train_highrescam_230517_399_{idx}_fix_betas.npz'
filename = getattr(cfg, 'filename', pre_prc_file_train)
else:
if idx > 0: continue
pre_prc_file_test = f'renbody_test_highrescam_230517_78_{idx}_fix_betas.npz'
filename = getattr(cfg, 'filename', pre_prc_file_test)
self.img_dir = osp.join(cfg.data_dir, 'RenBody')
self.annot_path = osp.join(cfg.data_dir, 'preprocessed_datasets', filename)
if self.use_cache:
print(f'[{self.__class__.__name__}] Cache not found, generating cache...')
data_split = self.load_data(
train_sample_interval=getattr(cfg, f'{self.__class__.__name__}_{self.data_split}_sample_interval', 1),
test_sample_interval=getattr(cfg, f'{self.__class__.__name__}_{self.data_split}_sample_interval', 10))
self.datalist.extend(data_split)
if self.use_cache:
self.save_cache(self.annot_path_cache, self.datalist)
data/SHAPY/.gitignore
-3
View File
@@ -1,3 +0,0 @@
__pycache__/
build/
*.so
data/SHAPY/SHAPY.py
-212
View File
@@ -1,212 +0,0 @@
import os
import os.path as osp
import numpy as np
import torch
import cv2
import json
import copy
from pycocotools.coco import COCO
from config import cfg
from utils.human_models import smpl_x
from utils.preprocessing import load_img, process_bbox, augmentation, process_db_coord, process_human_model_output, \
get_fitting_error_3D
from utils.transforms import world2cam, cam2pixel, rigid_align
from humandata import HumanDataset
import pickle
from body_measurements import BodyMeasurements
import smplx
from test_submission_format import test_submission_file_format
def point_error(x, y, align=True):
""" Ref: https://github.com/muelea/shapy/blob/master/regressor/hbw_evaluation/evaluate_hbw.py#LL44C1-L58C31 """
t = 0.0
if align:
t = x.mean(0, keepdims=True) - y.mean(0, keepdims=True)
x_hat = x - t
error = np.sqrt(np.power(x_hat - y, 2).sum(axis=-1))
return error.mean().item()
class SHAPY(HumanDataset):
def __init__(self, transform, data_split):
super(SHAPY, self).__init__(transform, data_split)
self.eval_split = getattr(cfg, 'shapy_eval_split')
if self.data_split == 'train':
raise NotImplementedError('Shapy train not implemented yet. Need to consider invalid parameters')
if self.data_split == 'test' and self.eval_split == 'test':
filename = getattr(cfg, 'filename', 'shapy_test_230512_1631.npz')
elif self.data_split == 'test' and self.eval_split == 'val':
filename = getattr(cfg, 'filename', 'shapy_val_230512_705.npz')
else:
raise ValueError(f'Undefined. data split: {self.data_split}; eval_split: {self.test_set}')
self.img_dir = osp.join(cfg.data_dir, 'SHAPY')
self.annot_path = osp.join(cfg.data_dir, 'preprocessed_datasets', filename)
self.v_shape_load_dir = osp.join(cfg.data_dir, 'SHAPY', 'HBW', 'smplx', 'val')
self.img_shape = None # variable img_shape
self.cam_param = {}
# load data
self.datalist = self.load_data(
train_sample_interval=getattr(cfg, f'{self.__class__.__name__}_train_sample_interval', 1))
### SHAPY utils
### ref: https://github.com/muelea/shapy/blob/master/regressor/hbw_evaluation/evaluate_hbw.py#L28
# load body model
# ref: common/utils/human_models.py
self.layer_arg = {'create_global_orient': False, 'create_body_pose': False, 'create_left_hand_pose': False,
'create_right_hand_pose': False, 'create_jaw_pose': False, 'create_leye_pose': False,
'create_reye_pose': False, 'create_betas': False, 'create_expression': False,
'create_transl': False}
self.smplx_layer = smplx.create(cfg.human_model_path,
'smplx',
gender='NEUTRAL',
use_pca=False,
use_face_contour=True,
flat_hand_mean=True, # critical!
**self.layer_arg).cuda()
# self.smplx_layer = copy.deepcopy(smpl_x.layer['neutral']).cuda()
self.faces_tensor_smplx = self.smplx_layer.faces_tensor.detach().cpu().numpy()
# load files to compute P2P-20K Error
point_reg = osp.join(cfg.data_dir, 'SHAPY', 'utility_files', 'evaluation', 'eval_point_set', 'HD_SMPLX_from_SMPL.pkl')
with open(point_reg, 'rb') as f:
self.point_regressor = pickle.load(f)
# load files to compute Measurements Error
body_measurement_folder = osp.join(cfg.data_dir, 'SHAPY', 'utility_files', 'measurements')
meas_def_path = osp.join(body_measurement_folder, 'measurement_defitions.yaml')
meas_verts_path_gt = osp.join(body_measurement_folder, 'smplx_measurements.yaml')
self.body_measurements = BodyMeasurements(
{'meas_definition_path': meas_def_path,
'meas_vertices_path': meas_verts_path_gt},
).to('cuda')
self.v_shaped_gt = {}
# to save preditions
self.images_names = []
self.v_shaped = []
def evaluate(self, outs, cur_sample_idx):
annots = self.datalist
sample_num = len(outs)
eval_result = {'v2v_t_errors': [], 'point_t_errors': [], 'height': [], 'chest': [], 'waist': [], 'hips': [], 'mass': []}
for n in range(sample_num):
annot = annots[cur_sample_idx + n]
out = outs[n]
betas_fit = out['smplx_shape']
img_path = out['img_path']
# compute v_shaped
betas_fit = torch.tensor(betas_fit.reshape(-1, 10)).cuda()
output = self.smplx_layer(
betas=betas_fit,
body_pose=torch.zeros((1, 63)).to(betas_fit.device),
global_orient=torch.zeros((1, 3)).to(betas_fit.device),
right_hand_pose=torch.zeros((1, 45)).to(betas_fit.device),
left_hand_pose=torch.zeros((1, 45)).to(betas_fit.device),
jaw_pose=torch.zeros((1, 3)).to(betas_fit.device),
leye_pose=torch.zeros((1, 3)).to(betas_fit.device),
reye_pose=torch.zeros((1, 3)).to(betas_fit.device),
expression=torch.zeros((1, 10)).to(betas_fit.device),
return_verts=True
)
v_shaped_fit = output.vertices.detach().cpu().numpy().squeeze()
image_name = '/'.join(img_path.split('/')[-4:])
self.images_names.append(image_name)
self.v_shaped.append(v_shaped_fit)
if self.eval_split == 'val':
# load gt vertices
subject = img_path.split('/')[-3]
subject_id_npy = subject.split('_')[0] + '.npy'
v_shaped_gt_path = osp.join(self.v_shape_load_dir, subject_id_npy)
if v_shaped_gt_path not in self.v_shaped_gt:
v_shaped_gt = np.load(v_shaped_gt_path)
self.v_shaped_gt[v_shaped_gt_path] = v_shaped_gt
else:
v_shaped_gt = self.v_shaped_gt[v_shaped_gt_path]
# compute vertex-to-vertex error (SMPL-X only)
# ref: https://github.com/muelea/shapy/blob/master/regressor/hbw_evaluation/evaluate_hbw.py#LL142C1-L171C48
v2v_error = point_error(v_shaped_fit, v_shaped_gt, align=True)
eval_result['v2v_t_errors'].append(v2v_error)
# compute P2P-20k error
points_gt = self.point_regressor.dot(v_shaped_gt)
points_fit = self.point_regressor.dot(v_shaped_fit)
p2p_error = point_error(points_gt, points_fit, align=True)
eval_result['point_t_errors'].append(p2p_error)
# compute height/chest/waist/hip error
shaped_triangles_gt = v_shaped_gt[self.faces_tensor_smplx]
shaped_triangles_gt = torch.from_numpy(shaped_triangles_gt).unsqueeze(0).to('cuda')
measurements_gt = self.body_measurements(shaped_triangles_gt)['measurements']
shaped_triangles_fit = v_shaped_fit[self.faces_tensor_smplx]
shaped_triangles_fit = torch.from_numpy(shaped_triangles_fit).unsqueeze(0).to('cuda')
measurements_fit = self.body_measurements(shaped_triangles_fit)['measurements']
for k in ['height', 'chest', 'waist', 'hips', 'mass']:
error = abs(measurements_gt[k]['tensor'].item() - measurements_fit[k]['tensor'].item())
eval_result[k].append(error)
return eval_result
def print_eval_result(self, eval_result):
# print('SHAPY results are dumped at: ' + osp.join(cfg.result_dir, 'predictions'))
if self.data_split == 'test' and self.eval_split == 'test': # do not print. just submit the results to the official evaluation server
# save predictions in the format of HBW challenge
# ref: https://github.com/muelea/shapy/blob/master/regressor/hbw_evaluation/README_HBW_EVAL.md#hbw-challenge
save_dir = osp.join(cfg.result_dir, 'predictions')
os.makedirs(save_dir, exist_ok=True)
save_name = osp.join(save_dir, 'hbw_prediction')
images_names = np.array(self.images_names).reshape(1631, )
v_shaped = np.array(self.v_shaped).reshape(1631, 10475, 3)
np.savez(save_name,
image_name=images_names,
v_shaped=v_shaped)
print('predictions saved at: ' + save_name + '.npz')
# run format test
test_submission_file_format(save_name + '.npz')
return
v2v_t_errors = np.mean(eval_result['v2v_t_errors']) * 1000
point_t_errors = np.mean(eval_result['point_t_errors']) * 1000
chest = np.mean(eval_result['chest']) * 1000
waist = np.mean(eval_result['waist']) * 1000
hips = np.mean(eval_result['hips']) * 1000
height = np.mean(eval_result['height']) * 1000
mass = np.mean(eval_result['mass'])
print('======SHAPY-val======')
print('Height Error: %.2f mm' % height)
print('Chest Error: %.2f mm' % chest)
print('Waist Error: %.2f mm' % waist)
print('Hips Error: %.2f mm' % hips)
print('P2P-20k Error: %.2f mm' % point_t_errors)
print('V2V Error: %.2f mm' % v2v_t_errors)
print('Mass Error: %.2f kg' % mass)
f = open(os.path.join(cfg.result_dir, 'result.txt'), 'w')
f.write(f'SHAPY-val dataset: \n')
f.write('Height Error: %.2f mm\n' % height)
f.write('Chest Error: %.2f mm' % chest)
f.write('Waist Error: %.2f mm\n' % waist)
f.write('Hips Error: %.2f mm\n' % hips)
f.write('P2P-20k Error: %.2f mm' % point_t_errors)
f.write('V2V Error: %.2f mm\n' % v2v_t_errors)
f.write('Mass Error: %.2f kg\n' % mass)
f.close()
data/SHAPY/mesh-mesh-intersection/LICENSE
-58
View File
@@ -1,58 +0,0 @@
License
Software Copyright License for non-commercial scientific research purposes
Please read carefully the following terms and conditions and any accompanying documentation before you download and/or use the SMPL-X/SMPLify-X model, data and software, (the "Model & Software"), including 3D meshes, blend weights, blend shapes, textures, software, scripts, and animations. By downloading and/or using the Model & Software (including downloading, cloning, installing, and any other use of this github repository), you acknowledge that you have read these terms and conditions, understand them, and agree to be bound by them. If you do not agree with these terms and conditions, you must not download and/or use the Model & Software. Any infringement of the terms of this agreement will automatically terminate your rights under this License
Ownership / Licensees
The Software and the associated materials has been developed at the
Max Planck Institute for Intelligent Systems (hereinafter "MPI").
Any copyright or patent right is owned by and proprietary material of the
Max-Planck-Gesellschaft zur Förderung der Wissenschaften e.V. (hereinafter “MPG”; MPI and MPG hereinafter collectively “Max-Planck”)
hereinafter the “Licensor”.
License Grant
Licensor grants you (Licensee) personally a single-user, non-exclusive, non-transferable, free of charge right:
To install the Model & Software on computers owned, leased or otherwise controlled by you and/or your organization;
To use the Model & Software for the sole purpose of performing non-commercial scientific research, non-commercial education, or non-commercial artistic projects;
Any other use, in particular any use for commercial purposes, is prohibited. This includes, without limitation, incorporation in a commercial product, use in a commercial service, or production of other artifacts for commercial purposes. The Model & Software may not be reproduced, modified and/or made available in any form to any third party without Max-Planck’s prior written permission.
The Model & Software may not be used for pornographic purposes or to generate pornographic material whether commercial or not. This license also prohibits the use of the Model & Software to train methods/algorithms/neural networks/etc. for commercial use of any kind. By downloading the Model & Software, you agree not to reverse engineer it.
No Distribution
The Model & Software and the license herein granted shall not be copied, shared, distributed, re-sold, offered for re-sale, transferred or sub-licensed in whole or in part except that you may make one copy for archive purposes only.
Disclaimer of Representations and Warranties
You expressly acknowledge and agree that the Model & Software results from basic research, is provided “AS IS”, may contain errors, and that any use of the Model & Software is at your sole risk. LICENSOR MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND CONCERNING THE MODEL & SOFTWARE, NEITHER EXPRESS NOR IMPLIED, AND THE ABSENCE OF ANY LEGAL OR ACTUAL DEFECTS, WHETHER DISCOVERABLE OR NOT. Specifically, and not to limit the foregoing, licensor makes no representations or warranties (i) regarding the merchantability or fitness for a particular purpose of the Model & Software, (ii) that the use of the Model & Software will not infringe any patents, copyrights or other intellectual property rights of a third party, and (iii) that the use of the Model & Software will not cause any damage of any kind to you or a third party.
Limitation of Liability
Because this Model & Software License Agreement qualifies as a donation, according to Section 521 of the German Civil Code (Bürgerliches Gesetzbuch – BGB) Licensor as a donor is liable for intent and gross negligence only. If the Licensor fraudulently conceals a legal or material defect, they are obliged to compensate the Licensee for the resulting damage.
Licensor shall be liable for loss of data only up to the amount of typical recovery costs which would have arisen had proper and regular data backup measures been taken. For the avoidance of doubt Licensor shall be liable in accordance with the German Product Liability Act in the event of product liability. The foregoing applies also to Licensor’s legal representatives or assistants in performance. Any further liability shall be excluded.
Patent claims generated through the usage of the Model & Software cannot be directed towards the copyright holders.
The Model & Software is provided in the state of development the licensor defines. If modified or extended by Licensee, the Licensor makes no claims about the fitness of the Model & Software and is not responsible for any problems such modifications cause.
No Maintenance Services
You understand and agree that Licensor is under no obligation to provide either maintenance services, update services, notices of latent defects, or corrections of defects with regard to the Model & Software. Licensor nevertheless reserves the right to update, modify, or discontinue the Model & Software at any time.
Defects of the Model & Software must be notified in writing to the Licensor with a comprehensible description of the error symptoms. The notification of the defect should enable the reproduction of the error. The Licensee is encouraged to communicate any use, results, modification or publication.
Publications using the Model & Software
You acknowledge that the Model & Software is a valuable scientific resource and agree to appropriately reference the following paper in any publication making use of the Model & Software.
Citation:
@inproceedings{SMPL-X:2019,
title = {Expressive Body Capture: 3D Hands, Face, and Body from a Single Image},
author = {Pavlakos, Georgios and Choutas, Vasileios and Ghorbani, Nima and Bolkart, Timo and Osman, Ahmed A. A. and Tzionas, Dimitrios and Black, Michael J.},
booktitle = {Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)},
year = {2019}
}
Commercial licensing opportunities
For commercial uses of the Software, please send email to ps-license@tue.mpg.de
This Agreement shall be governed by the laws of the Federal Republic of Germany except for the UN Sales Convention.
data/SHAPY/mesh-mesh-intersection/README.md
-82
View File
@@ -1,82 +0,0 @@
# Computing mesh-mesh intersection
This package provides a PyTorch module that can efficiently compute mesh-mesh
intersections using a BVH.
## Table of Contents
* [Description](#description)
* [Installation](#installation)
* [Examples](#examples)
* [Citation](#citation)
* [License](#license)
* [Contact](#contact)
## Description
This repository provides a PyTorch wrapper around a CUDA kernel that implements
the method described in [Maximizing parallelism in the construction of BVHs,
octrees, and k-d trees](https://dl.acm.org/citation.cfm?id=2383801). More
specifically, given an input mesh it builds a
BVH tree for each one and queries it for self-intersections.
## Installation
See the instructions [here](docs/install.md) on how to install the package.
## Examples
### Fitting to measurements
To fit a 3D human body model to height, weight and circumenference measurements
use the following command:
```python
python examples/fit_measurements.py --model-folder PATH_TO_BODY_MODELS \
--model-type [smpl/smplh/star/smplx] --gender neutral/female/male --num-betas 30 \
--meas-vertices-path data/smpl_measurement_vertices.yaml
```
If you are using SMPL-X then set `--meas-vertices-path data/smplx_measurements.yaml`.
## Citation
If you find this code useful in your research please cite the relevant work(s) of the following list, for detecting and penalizing mesh intersections accordingly:
```
@inproceedings{Karras:2012:MPC:2383795.2383801,
author = {Karras, Tero},
title = {Maximizing Parallelism in the Construction of BVHs, Octrees, and K-d Trees},
booktitle = {Proceedings of the Fourth ACM SIGGRAPH / Eurographics Conference on High-Performance Graphics},
year = {2012},
pages = {33--37},
numpages = {5},
url = {https://doi.org/10.2312/EGGH/HPG12/033-037},
doi = {10.2312/EGGH/HPG12/033-037},
publisher = {Eurographics Association}
}
```
## License
Software Copyright License for **non-commercial scientific research purposes**.
Please read carefully the [terms and
conditions](https://github.com/vchoutas/mesh-mesh-intersection/blob/master/LICENSE) and any
accompanying documentation before you download and/or use the SMPL-X/SMPLify-X
model, data and software, (the "Model & Software"), including 3D meshes, blend
weights, blend shapes, textures, software, scripts, and animations. By
downloading and/or using the Model & Software (including downloading, cloning,
installing, and any other use of this github repository), you acknowledge that
you have read these terms and conditions, understand them, and agree to be bound
by them. If you do not agree with these terms and conditions, you must not
download and/or use the Model & Software. Any infringement of the terms of this
agreement will automatically terminate your rights under this
[License](./LICENSE).
## Contact
The code of this repository was implemented by [Vassilis Choutas](vassilis.choutas@tuebingen.mpg.de).
For questions, please contact [smplx@tue.mpg.de](smplx@tue.mpg.de).
For commercial licensing (and all related questions for business applications), please contact [ps-licensing@tue.mpg.de](ps-licensing@tue.mpg.de). Please note that the method for this component has been [patented by NVidia](https://patents.google.com/patent/US9396512B2/en) and a license needs to be obtained also by them.
data/SHAPY/mesh-mesh-intersection/body_measurements/__init__.py
-2
View File
@@ -1,2 +0,0 @@
from .body_measurements import BodyMeasurements
from .cwh_measurements import ChestWaistHipsMeasurements
data/SHAPY/mesh-mesh-intersection/body_measurements/body_measurements.py
-246
View File
@@ -1,246 +0,0 @@
from typing import NewType, Dict, Tuple
import os.path as osp
import yaml
import numpy as np
from mesh_mesh_intersection import MeshMeshIntersection
import torch
import torch.autograd as autograd
import torch.nn as nn
import torch.nn.functional as F
from scipy.spatial import ConvexHull
# from loguru import logger
Tensor = NewType('Tensor', torch.Tensor)
class BodyMeasurements(nn.Module):
# The density of the human body is 985 kg / m^3
DENSITY = 985
def __init__(self, cfg, **kwargs):
''' Loss that penalizes deviations in weight and height
'''
super(BodyMeasurements, self).__init__()
meas_definition_path = cfg.get('meas_definition_path', '')
meas_definition_path = osp.expanduser(
osp.expandvars(meas_definition_path))
meas_vertices_path = cfg.get('meas_vertices_path', '')
meas_vertices_path = osp.expanduser(
osp.expandvars(meas_vertices_path))
with open(meas_definition_path, 'r') as f:
measurements_definitions = yaml.safe_load(f, )
with open(meas_vertices_path, 'r') as f:
meas_vertices = yaml.safe_load(f)
head_top = meas_vertices['HeadTop']
left_heel = meas_vertices['HeelLeft']
left_heel_bc = left_heel['bc']
self.left_heel_face_idx = left_heel['face_idx']
left_heel_bc = torch.tensor(left_heel['bc'], dtype=torch.float32)
self.register_buffer('left_heel_bc', left_heel_bc)
head_top_bc = torch.tensor(head_top['bc'], dtype=torch.float32)
self.register_buffer('head_top_bc', head_top_bc)
self.head_top_face_idx = head_top['face_idx']
action = measurements_definitions['CW_p']
chest_periphery_data = meas_vertices[action[0]]
self.chest_face_index = chest_periphery_data['face_idx']
chest_bcs = torch.tensor(
chest_periphery_data['bc'], dtype=torch.float32)
self.register_buffer('chest_bcs', chest_bcs)
action = measurements_definitions['BW_p']
belly_periphery_data = meas_vertices[action[0]]
self.belly_face_index = belly_periphery_data['face_idx']
belly_bcs = torch.tensor(
belly_periphery_data['bc'], dtype=torch.float32)
self.register_buffer('belly_bcs', belly_bcs)
action = measurements_definitions['IW_p']
hips_periphery_data = meas_vertices[action[0]]
self.hips_face_index = hips_periphery_data['face_idx']
hips_bcs = torch.tensor(
hips_periphery_data['bc'], dtype=torch.float32)
self.register_buffer('hips_bcs', hips_bcs)
max_collisions = cfg.get('max_collisions', 256)
self.isect_module = MeshMeshIntersection(max_collisions=max_collisions)
def extra_repr(self) -> str:
msg = []
msg.append(f'Human Body Density: {self.DENSITY}')
return '\n'.join(msg)
def _get_plane_at_heights(self, height: Tensor) -> Tuple[Tensor, Tensor, Tensor]:
device = height.device
batch_size = height.shape[0]
verts = torch.tensor(
[[-1., 0, -1], [1, 0, -1], [1, 0, 1], [-1, 0, 1]],
device=device).unsqueeze(dim=0).expand(batch_size, -1, -1).clone()
verts[:, :, 1] = height.reshape(batch_size, -1)
faces = torch.tensor([[0, 1, 2], [0, 2, 3]], device=device,
dtype=torch.long)
return verts, faces, verts[:, faces]
def compute_peripheries(
self,
triangles: Tensor,
compute_chest: bool = True,
compute_waist: bool = True,
compute_hips: bool = True,
) -> Dict[str, Tensor]:
'''
Parameters
----------
triangles: BxFx3x3 torch.Tensor
Contains the triangle coordinates for a batch of meshes with
the same topology
'''
batch_size, num_triangles = triangles.shape[:2]
device = triangles.device
batch_indices = torch.arange(
batch_size, dtype=torch.long,
device=device).reshape(-1, 1) * num_triangles
meas_data = {}
if compute_chest:
meas_data['chest'] = (self.chest_face_index, self.chest_bcs)
if compute_waist:
meas_data['waist'] = (self.belly_face_index, self.belly_bcs)
if compute_hips:
meas_data['hips'] = (self.hips_face_index, self.hips_bcs)
output = {}
for name, (face_index, bcs) in meas_data.items():
vertex = (
triangles[:, face_index] * bcs.reshape(1, 3, 1)).sum(axis=1)
_, _, plane_tris = self._get_plane_at_heights(vertex[:, 1])
with torch.no_grad():
collision_faces, collision_bcs = self.isect_module(
plane_tris, triangles)
selected_triangles = triangles.view(-1, 3, 3)[
(collision_faces + batch_indices).view(-1)].reshape(
batch_size, -1, 3, 3)
points = (
selected_triangles[:, :, None] *
collision_bcs[:, :, :, :, None]).sum(
axis=-2).reshape(batch_size, -1, 2, 3)
np_points = points.detach().cpu().numpy()
collision_faces = collision_faces.detach().cpu().numpy()
collision_bcs = collision_bcs.detach().cpu().numpy()
output[name] = {
'points': [],
'valid_points': [],
'value': [],
'plane_height': vertex[:, 1],
}
for ii in range(batch_size):
valid_face_idxs = np.where(collision_faces[ii] > 0)[0]
points_in_plane = np_points[
ii, valid_face_idxs, :, ][:, :, [0, 2]].reshape(
-1, 2)
hull = ConvexHull(points_in_plane)
point_indices = hull.simplices.reshape(-1)
hull_points = points[ii][valid_face_idxs].view(
-1, 3)[point_indices].reshape(-1, 2, 3)
meas_value = (
hull_points[:, 1] - hull_points[:, 0]).pow(2).sum(
dim=-1).sqrt().sum()
output[name]['valid_points'].append(
np_points[ii, valid_face_idxs])
output[name]['points'].append(hull_points)
output[name]['value'].append(meas_value)
output[name]['tensor'] = torch.stack(output[name]['value'])
return output
def compute_height(self, shaped_triangles: Tensor) -> Tuple[Tensor, Tensor]:
''' Compute the height using the heel and the top of the head
'''
head_top_tri = shaped_triangles[:, self.head_top_face_idx]
head_top = (head_top_tri[:, 0, :] * self.head_top_bc[0] +
head_top_tri[:, 1, :] * self.head_top_bc[1] +
head_top_tri[:, 2, :] * self.head_top_bc[2])
head_top = (
head_top_tri * self.head_top_bc.reshape(1, 3, 1)
).sum(dim=1)
left_heel_tri = shaped_triangles[:, self.left_heel_face_idx]
left_heel = (
left_heel_tri * self.left_heel_bc.reshape(1, 3, 1)
).sum(dim=1)
return (torch.abs(head_top[:, 1] - left_heel[:, 1]),
torch.stack([head_top, left_heel], axis=0)
)
def compute_mass(self, tris: Tensor) -> Tensor:
''' Computes the mass from volume and average body density
'''
x = tris[:, :, :, 0]
y = tris[:, :, :, 1]
z = tris[:, :, :, 2]
volume = (
-x[:, :, 2] * y[:, :, 1] * z[:, :, 0] +
x[:, :, 1] * y[:, :, 2] * z[:, :, 0] +
x[:, :, 2] * y[:, :, 0] * z[:, :, 1] -
x[:, :, 0] * y[:, :, 2] * z[:, :, 1] -
x[:, :, 1] * y[:, :, 0] * z[:, :, 2] +
x[:, :, 0] * y[:, :, 1] * z[:, :, 2]
).sum(dim=1).abs() / 6.0
return volume * self.DENSITY
def forward(
self,
triangles: Tensor,
compute_mass: bool = True,
compute_height: bool = True,
compute_chest: bool = True,
compute_waist: bool = True,
compute_hips: bool = True,
**kwargs
):
measurements = {}
if compute_mass:
measurements['mass'] = {}
mesh_mass = self.compute_mass(triangles)
measurements['mass']['tensor'] = mesh_mass
if compute_height:
measurements['height'] = {}
mesh_height, points = self.compute_height(triangles)
measurements['height']['tensor'] = mesh_height
measurements['height']['points'] = points
output = self.compute_peripheries(triangles,
compute_chest=compute_chest,
compute_waist=compute_waist,
compute_hips=compute_hips,
)
measurements.update(output)
return {'measurements': measurements}
data/SHAPY/mesh-mesh-intersection/body_measurements/cwh_measurements.py
-182
View File
@@ -1,182 +0,0 @@
# -*- coding: utf-8 -*-
# Max-Planck-Gesellschaft zur Förderung der Wissenschaften e.V. (MPG) is
# holder of all proprietary rights on this computer program.
# You can only use this computer program if you have closed
# a license agreement with MPG or you get the right to use the computer
# program from someone who is authorized to grant you that right.
# Any use of the computer program without a valid license is prohibited and
# liable to prosecution.
#
# Copyright©2019 Max-Planck-Gesellschaft zur Förderung
# der Wissenschaften e.V. (MPG). acting on behalf of its Max Planck Institute
# for Intelligent Systems. All rights reserved.
#
# Contact: ps-license@tuebingen.mpg.de
import sys
import os
import os.path as osp
from typing import NewType, Dict
import time
import yaml
import numpy as np
import torch
import torch.nn as nn
import torch.autograd as autograd
# from loguru import logger
from mesh_mesh_intersection import MeshMeshIntersection
from scipy.spatial import ConvexHull
Tensor = NewType('Tensor', torch.Tensor)
class ChestWaistHipsMeasurements(nn.Module):
def __init__(
self, meas_definition_path: str, meas_vertices_path: str,
max_collisions=256,
*args, **kwargs
) -> None:
super(ChestWaistHipsMeasurements, self).__init__()
meas_definition_path = osp.expanduser(
osp.expandvars(meas_definition_path))
meas_vertices_path = osp.expanduser(
osp.expandvars(meas_vertices_path))
assert osp.exists(meas_definition_path), (
'Measurement definition path does not exist:'
f' {meas_definition_path}'
)
assert osp.exists(meas_definition_path), (
'Measurement vertex path does not exist:'
f' {meas_vertices_path}'
)
with open(meas_definition_path, 'r') as f:
measurements_definitions = yaml.load(f)
with open(meas_vertices_path, 'r') as f:
meas_vertices = yaml.load(f)
action = measurements_definitions['CW_p']
chest_periphery_data = meas_vertices[action[0]]
self.chest_face_index = chest_periphery_data['face_idx']
chest_bcs = torch.tensor(
chest_periphery_data['bc'], dtype=torch.float32)
self.register_buffer('chest_bcs', chest_bcs)
action = measurements_definitions['BW_p']
belly_periphery_data = meas_vertices[action[0]]
self.belly_face_index = belly_periphery_data['face_idx']
belly_bcs = torch.tensor(
belly_periphery_data['bc'], dtype=torch.float32)
self.register_buffer('belly_bcs', belly_bcs)
action = measurements_definitions['IW_p']
hips_periphery_data = meas_vertices[action[0]]
self.hips_face_index = hips_periphery_data['face_idx']
hips_bcs = torch.tensor(
hips_periphery_data['bc'], dtype=torch.float32)
self.register_buffer('hips_bcs', hips_bcs)
self.isect_module = MeshMeshIntersection(max_collisions=max_collisions)
def _get_plane_at_heights(self, height: Tensor):
device = height.device
batch_size = height.shape[0]
verts = torch.tensor(
[[-1., 0, -1], [1, 0, -1], [1, 0, 1], [-1, 0, 1]],
device=device).unsqueeze(dim=0).expand(batch_size, -1, -1).clone()
verts[:, :, 1] = height.reshape(batch_size, -1)
faces = torch.tensor([[0, 1, 2], [0, 2, 3]], device=device,
dtype=torch.long)
return verts, faces, verts[:, faces]
def forward(
self,
triangles: Tensor
) -> Dict[str, Tensor]:
'''
Parameters
----------
triangles: BxFx3x3 torch.Tensor
Contains the triangle coordinates for a batch of meshes with
the same topology
'''
batch_size, num_triangles = triangles.shape[:2]
device = triangles.device
batch_indices = torch.arange(
batch_size, dtype=torch.long,
device=device).reshape(-1, 1) * num_triangles
meas_data = {
'chest': (self.chest_face_index, self.chest_bcs),
'belly': (self.belly_face_index, self.belly_bcs),
'hips': (self.hips_face_index, self.hips_bcs),
}
output = {}
for name, (face_index, bcs) in meas_data.items():
vertex = (
triangles[:, face_index] * bcs.reshape(1, 3, 1)).sum(axis=1)
_, _, plane_tris = self._get_plane_at_heights(vertex[:, 1])
with torch.no_grad():
collision_faces, collision_bcs = self.isect_module(
plane_tris, triangles)
selected_triangles = triangles.view(-1, 3, 3)[
(collision_faces + batch_indices).view(-1)].reshape(
batch_size, -1, 3, 3)
points = (
selected_triangles[:, :, None] *
collision_bcs[:, :, :, :, None]).sum(
axis=-2).reshape(batch_size, -1, 2, 3)
np_points = points.detach().cpu().numpy()
collision_faces = collision_faces.detach().cpu().numpy()
collision_bcs = collision_bcs.detach().cpu().numpy()
output[name] = {
'points': [],
'valid_points': [],
'value': [],
'plane_height': vertex[:, 1],
}
for ii in range(batch_size):
valid_face_idxs = np.where(collision_faces[ii] > 0)[0]
points_in_plane = np_points[
ii, valid_face_idxs, :, ][:, :, [0, 2]].reshape(
-1, 2)
hull = ConvexHull(points_in_plane)
point_indices = hull.simplices.reshape(-1)
hull_points = points[ii][valid_face_idxs].view(
-1, 3)[point_indices]
meas_value = (
hull_points[1::2] - hull_points[:-1:2]).pow(2).sum(
dim=-1).sqrt().sum()
# logger.info(f'{ii}: {name}, {meas_value}')
output[name]['valid_points'].append(
np_points[ii, valid_face_idxs])
output[name]['points'].append(hull_points)
output[name]['value'].append(meas_value)
# values.append(
# )
return output
data/SHAPY/mesh-mesh-intersection/data/measurement_defitions.yaml
-111
View File
@@ -1,111 +0,0 @@
A:
- FingerTipRight
- FingerTipLeft
- 0
- - 0
- 0
- 0.2
BW:
- BellyButton
- 0
- - 0.0
- 0
- 0.15
BW_p:
- BellyButton
- 0
- - 0.0
- 0
- 0
CW:
- NippleRight
- 0
- - 0.0
- 0
- 0.2
CW_p:
- NippleRight
- 0
- - 0.0
- 0
- 0
DBB:
- BackBellyButton
- BellyButton
- 2
- - 0.15
- 0
- 0
H:
- HeelLeft
- HeadTop
- 1
- - -0.15
- 0
- 0.0
HB:
- HeelLeft
- NippleRight
- NippleLeft
- 1
- - -0.1
- 0
- 0.0
HBB:
- HeelLeft
- BellyButton
- 1
- - -0.05
- 0
- 0.0
HI:
- HeelLeft
- Crotch
- 1
- - 0
- 0
- 0
IW:
- Crotch
- 0
- - 0.0
- 0
- 0.15
IW_p:
- Crotch
- 0
- - 0.0
- 0
- 0
SW:
- ShoulderApose
- 0
- - 0
- 0
- 0.1
V: []
W2E:
- ElbowRight
- WristRight
- ElbowLeft
- WristLeft
- 0
- - 0
- 0.0
- 0.1
W2S:
- ShoulderRight
- WristRight
- ShoulderLeft
- WristLeft
- 0
- - 0
- 0.0
- 0.1
W2W:
- WristRight
- WristLeft
- 0
- - 0
- -0.0
- 0.15
data/SHAPY/mesh-mesh-intersection/data/smpl_measurement_vertices.yaml
-112
View File
@@ -1,112 +0,0 @@
BackBellyButton:
bc:
- 1.0
- 0.0
- 0.0
face_idx: 4971
vertex_id: 3022
BellyButton:
bc:
- 0.0
- 0.0
- 1.0
face_idx: 6833
vertex_id: 3501
Crotch:
bc:
- 0.0
- 1.0
- 0.0
face_idx: 1341
vertex_id: 1210
ElbowLeft:
bc:
- 1.0
- 0.0
- 0.0
face_idx: 1867
vertex_id: 1658
ElbowRight:
bc:
- 0.0
- 0.0
- 1.0
face_idx: 8756
vertex_id: 5129
FingerTipLeft:
bc:
- 0.0
- 0.0
- 1.0
face_idx: 3259
vertex_id: 2445
FingerTipRight:
bc:
- 0.0
- 0.0
- 1.0
face_idx: 10147
vertex_id: 5905
HeadTop:
bc:
- 0.0
- 1.0
- 0.0
face_idx: 435
vertex_id: 411
HeelLeft:
bc:
- 0.0
- 0.0
- 1.0
face_idx: 5975
vertex_id: 3466
NippleLeft:
bc:
- 0.0
- 0.0
- 1.0
face_idx: 4997
vertex_id: 3042
NippleRight:
bc:
- 0.0
- 0.0
- 1.0
face_idx: 11885
vertex_id: 6489
ShoulderApose:
bc:
- 1.0
- 0.0
- 0.0
face_idx: 11937
vertex_id: 6496
ShoulderLeft:
bc:
- 0.0
- 0.0
- 1.0
face_idx: 4572
vertex_id: 2893
ShoulderRight:
bc:
- 1.0
- 0.0
- 0.0
face_idx: 9117
vertex_id: 5291
WristLeft:
bc:
- 0.0
- 0.0
- 1.0
face_idx: 2603
vertex_id: 2099
WristRight:
bc:
- 0.0
- 1.0
- 0.0
face_idx: 9491
vertex_id: 5559
data/SHAPY/mesh-mesh-intersection/data/smplx_measurements.yaml
-160
View File
@@ -1,160 +0,0 @@
BackBellyButton:
bc:
- 0.0
- 0.0
- 1.0
closest_points:
- 0.0
- -0.261382
- -0.102003
face_idx: 7861
BellyButton:
bc:
- 0.0
- 1.0
- 0.0
closest_points:
- -0.0
- -0.271119
- 0.144329
face_idx: 19229
Crotch:
bc:
- 0.0
- 0.0
- 1.0
closest_points:
- 0.0
- -0.53204
- 0.036471
face_idx: 6194
ElbowLeft:
bc:
- 0.0
- 1.0
- 0.0
closest_points:
- 0.447234
- 0.075995
- -0.099974
face_idx: 3959
ElbowRight:
bc:
- 0.0
- 1.0
- 0.0
closest_points:
- -0.447233
- 0.075995
- -0.099973
face_idx: 7846
FingerTipLeft:
bc:
- 0.0
- 0.0
- 1.0
closest_points:
- 0.918755
- 0.085413
- -0.084745
face_idx: 3469
FingerTipRight:
bc:
- 1.3918288479186636e-05
- 0.9998084353210817
- 0.00017764639043920215
closest_points:
- -0.9187545563631815
- 0.08541207136535614
- -0.08474471930198169
face_idx: 17602
HeadTop:
bc:
- 0.8277337276382795
- 0.1422200962169292
- 0.030046176144791284
closest_points:
- -0.0017716260945737938
- 0.4363661265424736
- -0.015488867245138805
face_idx: 2581
HeelLeft:
bc:
- 0.0
- 1.0
- 0.0
closest_points:
- 0.103005
- -1.346656
- -0.082664
face_idx: 15605
NippleLeft:
bc:
- 1.0
- 0.0
- 0.0
closest_points:
- 0.09747
- -0.032798
- 0.103687
face_idx: 16306
NippleRight:
bc:
- 0.0
- 0.0
- 1.0
closest_points:
- -0.09747
- -0.032798
- 0.103687
face_idx: 18402
ShoulderApose:
bc:
- 1.0
- 0.0
- 0.0
closest_points:
- -0.013092
- 0.117681
- 0.03777
face_idx: 18412
ShoulderLeft:
bc:
- 0.0
- 0.0
- 1.0
closest_points:
- 0.183378
- 0.139588
- -0.066495
face_idx: 3865
ShoulderRight:
bc:
- 0.0
- 0.0
- 1.0
closest_points:
- -0.196498
- 0.140399
- -0.063292
face_idx: 18186
WristLeft:
bc:
- 1.0
- 0.0
- 0.0
closest_points:
- 0.72226
- 0.065775
- -0.070361
face_idx: 3363
WristRight:
bc:
- 0.0
- 0.0
- 1.0
closest_points:
- -0.72226
- 0.065775
- -0.070361
face_idx: 6722
data/SHAPY/mesh-mesh-intersection/docs/install.md
-31
View File
@@ -1,31 +0,0 @@
# Installation
Before installing anything please make sure to set the environment variable
*$CUDA_SAMPLES_INC* to the path that contains the header `helper_math.h`, which
can be found in the repo [CUDA Samples repository](https://github.com/NVIDIA/cuda-samples).
To install the module run the following commands:
**1. Clone this repository**
```Shell
git clone https://github.com/vchoutas/torch-mesh-isect
cd torch-mesh-isect
```
**2. Install the dependencies**
```Shell
pip install -r requirements.txt
```
**3. Run the *setup.py* script**
```Shell
python setup.py install
```
## Dependencies
1. [PyTorch](https://pytorch.org)
### Optional Dependencies
1. [Trimesh](https://trimsh.org) for loading triangular meshes
2. [open3d](http://www.open3d.org/) for visualization
The code has been tested with Python 3.6, CUDA 10.0, CuDNN 7.3 and PyTorch 1.0.
data/SHAPY/mesh-mesh-intersection/examples/fit_measurements.py
-304
View File
@@ -1,304 +0,0 @@
# -*- coding: utf-8 -*-
# Max-Planck-Gesellschaft zur Förderung der Wissenschaften e.V. (MPG) is
# holder of all proprietary rights on this computer program.
# You can only use this computer program if you have closed
# a license agreement with MPG or you get the right to use the computer
# program from someone who is authorized to grant you that right.
# Any use of the computer program without a valid license is prohibited and
# liable to prosecution.
#
# Copyright©2019 Max-Planck-Gesellschaft zur Förderung
# der Wissenschaften e.V. (MPG). acting on behalf of its Max Planck Institute
# for Intelligent Systems. All rights reserved.
#
# Contact: ps-license@tuebingen.mpg.de
import sys
import os.path as osp
import argparse
import numpy as np
import torch
import torch.nn.functional as F
import torch.optim as optim
import smplx
import open3d as o3d
import time
import cv2
from tqdm import tqdm
import trimesh
from loguru import logger
from star.pytorch.star import STAR
from star.config import cfg as star_cfg
from body_measurements import BodyMeasurements
from torchtrustncg import TrustRegion
def get_plane_at_height(h):
verts = np.array([[-1., h, -1], [1, h, -1], [1, h, 1], [-1, h, 1]])
faces = np.array([[0, 1, 2], [0, 2, 3]])
normal = np.array([0.0, 1.0, 0.0])
return verts, faces, (verts[0], normal)
def main(
model_folder,
height: float = 1.76,
mass: float = -1,
chest: float = 1.12,
waist: float = 0.93,
hips: float = 1.14,
model_type='smplx',
ext='npz',
gender='neutral',
num_betas=10,
meas_definition_path: str = 'data/measurement_defitions.yaml',
meas_vertices_path: str = 'data/smpl_measurement_vertices.yaml',
summary_steps: int = 50,
num_iterations: int = 500,
betas_weight: float = 0.0,
):
device = torch.device('cuda')
dtype = torch.float32
cfg = {
'meas_definition_path': meas_definition_path,
'meas_vertices_path': meas_vertices_path,
}
meas_module = BodyMeasurements(cfg)
meas_module = meas_module.to(device=device)
num_samples = 1
trans, pose = None, None
logger.info(f'Model type: {model_type}')
if 'star' in model_type:
star_cfg.path_male_star = osp.expandvars(
osp.join(model_folder, 'star', 'STAR_MALE.npz'))
star_cfg.path_female_star = osp.expandvars(
osp.join(model_folder, 'star', 'STAR_FEMALE.npz'))
model = STAR(gender=gender, num_betas=num_betas)
trans = torch.zeros([num_samples, 3], dtype=dtype, device=device)
pose = torch.zeros([num_samples, 72], dtype=dtype, device=device)
else:
model = smplx.build_layer(
model_folder, model_type=model_type,
gender=gender,
num_betas=num_betas,
ext=ext)
logger.info(model)
model = model.to(device=device)
betas = torch.zeros(
[num_samples, model.num_betas],
requires_grad=True, dtype=torch.float32, device=device)
dtype = torch.float32
gt = {
'height': torch.tensor(height, dtype=dtype, device=device),
'mass': torch.tensor(mass, dtype=dtype, device=device),
'chest': torch.tensor(chest, dtype=dtype, device=device),
'waist': torch.tensor(waist, dtype=dtype, device=device),
'hips': torch.tensor(hips, dtype=dtype, device=device),
}
weights = {
'height': 100.0 if height > 0 else 0.0,
'mass': 1.0 if mass > 0 else 0.0,
'chest': 2000.0 if chest > 0 else 0.0,
'waist': 1000.0 if waist > 0 else 0.0,
'hips': 1000.0 if hips > 0 else 0.0,
}
optimizer = TrustRegion([betas])
def compute_loss(gt, output, weights):
losses = {}
for key, gt_val in gt.items():
if weights[key] <= 1e-3 or gt_val.item() < 0:
continue
est_val = output[key]['tensor']
if isinstance(est_val, (tuple, list)):
est_val = torch.stack(output[key]['value'])
curr_loss = (gt_val - est_val).pow(2).sum() * weights[key]
losses[key] = curr_loss
losses['betas'] = betas_weight * betas.pow(2).sum()
return losses
def closure(backward=True):
if backward:
optimizer.zero_grad()
if model_type == 'star':
vertices = model(pose=pose, trans=trans, betas=betas)
model_tris = vertices[:, model.faces]
else:
output = model(betas=betas, return_verts=True)
model_tris = output.vertices[:, model.faces_tensor]
output = meas_module(model_tris)['measurements']
losses = compute_loss(gt, output, weights)
loss = sum(losses.values())
if backward:
loss.backward(create_graph=True)
return loss
Y_OFFSET = -1.10
for n in tqdm(range(num_iterations)):
loss = optimizer.step(closure)
if n % summary_steps == 0:
if model_type == 'star':
vertices = model(pose=pose, trans=trans, betas=betas)
model_tris = vertices[:, model.faces]
vertices = vertices.detach().cpu().numpy().squeeze()
faces = model.faces.detach().cpu().numpy()
else:
output = model(betas=betas, return_verts=True)
vertices = output.vertices.detach().cpu().numpy().squeeze()
faces = model.faces
model_tris = output.vertices[:, model.faces_tensor]
y_offset = - vertices[:, 1].min() + Y_OFFSET
vertices[:, 1] = vertices[:, 1] + y_offset
# for key, val in losses.items():
mesh = o3d.geometry.TriangleMesh()
mesh.vertices = o3d.utility.Vector3dVector(vertices)
mesh.triangles = o3d.utility.Vector3iVector(faces)
mesh.compute_vertex_normals()
colors = np.ones_like(vertices) * [0.3, 0.3, 0.3]
mesh.vertex_colors = o3d.utility.Vector3dVector(colors)
geometry = []
geometry.append(mesh)
output = meas_module(model_tris)['measurements']
for key, val in gt.items():
est_val = output[key]["tensor"][0].item()
logger.info(
f'[{n:04d}]: {key}: est = {est_val}, gt = {val}')
losses = compute_loss(gt, output, weights)
for key, val in losses.items():
logger.info(f'[{n:04d}]: {key} loss = {val:.3f}')
for meas_name in output:
pcl = o3d.geometry.PointCloud()
if 'points' not in output[meas_name]:
continue
points = output[meas_name]['points']
if isinstance(points, (tuple, list)):
points = torch.stack(points)
if torch.is_tensor(points):
points = points.detach().cpu().numpy()
points = points.reshape(-1, 3)
points[:, 1] = points[:, 1] + y_offset
pcl.points = o3d.utility.Vector3dVector(points)
pcl.paint_uniform_color([1.0, 0.0, 0.0])
geometry.append(pcl)
lineset = o3d.geometry.LineSet()
line_ids = np.arange(len(points)).reshape(-1, 2)
lineset.points = o3d.utility.Vector3dVector(points)
lineset.lines = o3d.utility.Vector2iVector(line_ids)
lineset.paint_uniform_color([0.0, 0.0, 0.0])
geometry.append(lineset)
o3d.visualization.draw_geometries(
geometry,
lookat=np.array([0.0, 0.0, 0.0]).reshape(3, 1),
up=np.array([0.0, 1.0, 0.0]).reshape(3, 1),
front=np.array([0.0, 0.0, 1.0]).reshape(3, 1),
zoom=1.0,
)
if __name__ == '__main__':
logger.remove()
logger.add(lambda x: tqdm.write(x, end=''), colorize=True)
parser = argparse.ArgumentParser(description='SMPL-X Demo')
parser.add_argument('--model-folder', required=True, type=str,
help='The path to the model folder')
parser.add_argument('--model-type', default='smpl', type=str,
choices=['smpl', 'smplh', 'smplx', 'mano', 'flame',
'star', ],
help='The type of model to load')
parser.add_argument('--gender', type=str, default='neutral',
help='The gender of the model')
parser.add_argument('--num-betas', default=10, type=int,
dest='num_betas',
help='Number of shape coefficients.')
parser.add_argument('--ext', type=str, default='npz',
help='Which extension to use for loading')
parser.add_argument('--height', type=float, default=1.80,
help='Height of the subject in meters')
parser.add_argument('--mass', type=float, default=-1,
help='Mass of the subject in kilograms')
parser.add_argument('--chest', type=float, default=-1,
help='Chest circumference in meters')
parser.add_argument('--waist', type=float, default=-1,
help='Waist circumference in meters')
parser.add_argument('--hips', type=float, default=-1,
help='Hips circumference in meters')
parser.add_argument('--meas-definition-path',
dest='meas_definition_path',
default='data/measurement_defitions.yaml',
type=str,
help='The definitions of the measurements')
parser.add_argument('--meas-vertices-path', dest='meas_vertices_path',
type=str,
default='data/smpl_measurement_vertices.yaml',
help='The indices of the vertices used for the'
' the measurements')
parser.add_argument('--betas-weight', dest='betas_weight', default=0.0,
type=float,
help='The weight of the shape prior term.')
args = parser.parse_args()
model_folder = osp.expanduser(osp.expandvars(args.model_folder))
model_type = args.model_type
gender = args.gender
ext = args.ext
num_betas = args.num_betas
height = args.height
mass = args.mass
chest = args.chest
waist = args.waist
hips = args.hips
meas_definition_path = args.meas_definition_path
meas_vertices_path = args.meas_vertices_path
betas_weight = args.betas_weight
main(model_folder,
height=height,
mass=mass,
chest=chest,
waist=waist,
hips=hips,
model_type=model_type,
ext=ext,
gender=gender,
num_betas=num_betas,
meas_definition_path=meas_definition_path,
meas_vertices_path=meas_vertices_path,
betas_weight=betas_weight,
)
data/SHAPY/mesh-mesh-intersection/examples/vis_measurements.py
-245
View File
@@ -1,245 +0,0 @@
# -*- coding: utf-8 -*-
# Max-Planck-Gesellschaft zur Förderung der Wissenschaften e.V. (MPG) is
# holder of all proprietary rights on this computer program.
# You can only use this computer program if you have closed
# a license agreement with MPG or you get the right to use the computer
# program from someone who is authorized to grant you that right.
# Any use of the computer program without a valid license is prohibited and
# liable to prosecution.
#
# Copyright©2019 Max-Planck-Gesellschaft zur Förderung
# der Wissenschaften e.V. (MPG). acting on behalf of its Max Planck Institute
# for Intelligent Systems. All rights reserved.
#
# Contact: ps-license@tuebingen.mpg.de
import sys
import os.path as osp
import argparse
import numpy as np
import torch
import smplx
import open3d as o3d
import time
import cv2
from scipy.spatial import ConvexHull
import trimesh
# from meas_definitions import measurements_definitions, measures_vertex
from loguru import logger
from star.pytorch.star import STAR
from star.config import cfg as star_cfg
from mesh_mesh_intersection import MeshMeshIntersection
from body_measurements import ChestWaistHipsMeasurements
def get_plane_at_height(h):
verts = np.array([[-1., h, -1], [1, h, -1], [1, h, 1], [-1, h, 1]])
faces = np.array([[0, 1, 2], [0, 2, 3]])
normal = np.array([0.0, 1.0, 0.0])
return verts, faces, (verts[0], normal)
def main(
model_folder,
model_type='smplx',
ext='npz',
gender='neutral',
plot_joints=False,
num_betas=10,
sample_shape=False,
num_expression_coeffs=10,
plotting_module='pyrender',
num_samples=1,
use_face_contour=False,
meas_definition_path: str = 'data/measurement_defitions.yaml',
meas_vertices_path: str = 'data/smpl_measurement_vertices.yaml',
):
device = torch.device('cuda')
meas_module = ChestWaistHipsMeasurements(
meas_definition_path=meas_definition_path,
meas_vertices_path=meas_vertices_path,
# '$HOME/workspace/caesar_betas_exps/measurement_defitions.yaml',
# '$HOME/workspace/caesar_betas_exps/smpl_measurement_vertices.yaml',
)
# meas_module = ChestWaistHipsMeasurements(
# '$HOME/workspace/caesar_betas_exps/measurement_defitions.yaml',
# '$HOME/workspace/caesar_betas_exps/smplx_measurements.yaml',
# )
meas_module = meas_module.to(device=device)
dtype = torch.float32
trans, pose = None, None
if model_type == 'star':
star_cfg.path_male_star = osp.expandvars(
'$HOME/workspace/body_models/star/STAR_MALE.npz')
star_cfg.path_female_star = osp.expandvars(
'$HOME/workspace/body_models/star/STAR_FEMALE.npz')
model = STAR(gender=gender, num_betas=num_betas)
trans = torch.zeros([num_samples, 3], dtype=dtype, device=device)
pose = torch.zeros([num_samples, 72], dtype=dtype, device=device)
else:
model = smplx.build_layer(
model_folder, model_type=model_type,
gender=gender, use_face_contour=use_face_contour,
num_betas=num_betas,
num_expression_coeffs=num_expression_coeffs,
ext=ext)
model = model.to(device=device)
# meas_to_vis = ['CW_p', 'BW_p', 'IW_p']
# meas_to_vis = ['CW_p']
if sample_shape:
# betas = shape_dist.sample().reshape(1, -1)
betas = torch.randn(
[num_samples, model.num_betas],
requires_grad=True,
dtype=torch.float32, device=device)
else:
betas = torch.zeros(
[num_samples, model.num_betas],
requires_grad=True, dtype=torch.float32, device=device)
model.zero_grad()
if model_type == 'star':
vertices = model(pose=pose, trans=trans, betas=betas)
model_tris = vertices[:, model.faces]
vertices = vertices.detach().cpu().numpy()
faces = model.faces.detach().cpu().numpy()
else:
output = model(betas=betas, return_verts=True)
vertices = output.vertices.detach().cpu().numpy()
model_tris = output.vertices[:, model.faces_tensor]
faces = model.faces
output = meas_module(model_tris)
# loss = sum(v.pow(2) for v in output['chest']['value'])
for n in range(num_samples):
mesh = o3d.geometry.TriangleMesh()
mesh.vertices = o3d.utility.Vector3dVector(vertices[n])
mesh.triangles = o3d.utility.Vector3iVector(faces)
mesh.compute_vertex_normals()
colors = np.ones_like(vertices[n]) * [0.3, 0.3, 0.3]
mesh.vertex_colors = o3d.utility.Vector3dVector(colors)
geometry = []
geometry.append(mesh)
for meas_name in output:
pcl = o3d.geometry.PointCloud()
if 'points' not in output[meas_name]:
continue
points = output[meas_name]['points']
if isinstance(points, (tuple, list)):
points = torch.stack(points)
if torch.is_tensor(points):
points = points.detach().cpu().numpy()
points = points.reshape(-1, 3)
pcl.points = o3d.utility.Vector3dVector(points)
pcl.paint_uniform_color([1.0, 0.0, 0.0])
geometry.append(pcl)
lineset = o3d.geometry.LineSet()
line_ids = np.arange(len(points)).reshape(-1, 2)
lineset.points = o3d.utility.Vector3dVector(points)
lineset.lines = o3d.utility.Vector2iVector(line_ids)
lineset.paint_uniform_color([0.0, 0.0, 0.0])
geometry.append(lineset)
o3d.visualization.draw_geometries(geometry)
if __name__ == '__main__':
parser = argparse.ArgumentParser(description='SMPL-X Demo')
parser.add_argument('--model-folder', required=True, type=str,
help='The path to the model folder')
parser.add_argument('--model-type', default='smplx', type=str,
choices=['smpl', 'smplh', 'smplx', 'mano', 'flame',
'star', ],
help='The type of model to load')
parser.add_argument('--gender', type=str, default='neutral',
help='The gender of the model')
parser.add_argument('--num-betas', default=10, type=int,
dest='num_betas',
help='Number of shape coefficients.')
parser.add_argument('--num-expression-coeffs', default=10, type=int,
dest='num_expression_coeffs',
help='Number of expression coefficients.')
parser.add_argument('--plotting-module', type=str, default='pyrender',
dest='plotting_module',
choices=['pyrender', 'matplotlib', 'open3d'],
help='The module to use for plotting the result')
parser.add_argument('--ext', type=str, default='npz',
help='Which extension to use for loading')
parser.add_argument('--plot-joints', default=False,
type=lambda arg: arg.lower() in ['true', '1'],
help='The path to the model folder')
parser.add_argument('--sample-shape', default=False,
dest='sample_shape',
type=lambda arg: arg.lower() in ['true', '1'],
help='Sample a random shape')
parser.add_argument('--sample-expression', default=True,
dest='sample_expression',
type=lambda arg: arg.lower() in ['true', '1'],
help='Sample a random expression')
parser.add_argument('--use-face-contour', default=False,
type=lambda arg: arg.lower() in ['true', '1'],
help='Compute the contour of the face')
parser.add_argument('--num-samples', default=1, type=int,
dest='num_samples',
help='Number of samples to draw.')
parser.add_argument('--meas-definition-path',
dest='meas_definition_path',
default='data/measurement_defitions.yaml',
type=str,
help='The definitions of the measurements')
parser.add_argument('--meas-vertices-path', dest='meas_vertices_path',
type=str,
default='data/smpl_measurement_vertices.yaml',
help='The indices of the vertices used for the'
' the measurements')
args = parser.parse_args()
model_folder = osp.expanduser(osp.expandvars(args.model_folder))
model_type = args.model_type
plot_joints = args.plot_joints
use_face_contour = args.use_face_contour
gender = args.gender
ext = args.ext
plotting_module = args.plotting_module
num_betas = args.num_betas
num_expression_coeffs = args.num_expression_coeffs
sample_shape = args.sample_shape
sample_expression = args.sample_expression
num_samples = args.num_samples
meas_definition_path = args.meas_definition_path
meas_vertices_path = args.meas_vertices_path
main(model_folder, model_type, ext=ext,
gender=gender, plot_joints=plot_joints,
num_betas=num_betas,
num_samples=num_samples,
num_expression_coeffs=num_expression_coeffs,
sample_shape=sample_shape,
plotting_module=plotting_module,
meas_definition_path=meas_definition_path,
meas_vertices_path=meas_vertices_path,
use_face_contour=use_face_contour,
)
data/SHAPY/mesh-mesh-intersection/include/aabb.hpp
-117
View File
@@ -1,117 +0,0 @@
/*
* Max-Planck-Gesellschaft zur Förderung der Wissenschaften e.V. (MPG) is
* holder of all proprietary rights on this computer program.
* You can only use this computer program if you have closed
* a license agreement with MPG or you get the right to use the computer
* program from someone who is authorized to grant you that right.
* Any use of the computer program without a valid license is prohibited and
* liable to prosecution.
*
* Copyright©2019 Max-Planck-Gesellschaft zur Förderung
* der Wissenschaften e.V. (MPG). acting on behalf of its Max Planck Institute
* for Intelligent Systems. All rights reserved.
*
* @author Vasileios Choutas
* Contact: vassilis.choutas@tuebingen.mpg.de
* Contact: ps-license@tuebingen.mpg.de
*
*/
#ifndef AABB_H
#define AABB_H
#include <cuda.h>
#include "device_launch_parameters.h"
#include <cuda_runtime.h>
#include "defs.hpp"
#include "math_utils.hpp"
#include "double_vec_ops.hpp"
#include "helper_math.h"
template <typename T>
__align__(32)
struct AABB {
public:
__host__ __device__ AABB() {
min_t.x = std::is_same<T, float>::value ? FLT_MAX : DBL_MAX;
min_t.y = std::is_same<T, float>::value ? FLT_MAX : DBL_MAX;
min_t.z = std::is_same<T, float>::value ? FLT_MAX : DBL_MAX;
max_t.x = std::is_same<T, float>::value ? -FLT_MAX : -DBL_MAX;
max_t.y = std::is_same<T, float>::value ? -FLT_MAX : -DBL_MAX;
max_t.z = std::is_same<T, float>::value ? -FLT_MAX : -DBL_MAX;
};
__host__ __device__ AABB(const vec3<T> &min_t, const vec3<T> &max_t)
: min_t(min_t), max_t(max_t){};
__host__ __device__ ~AABB(){};
__host__ __device__ AABB(T min_t_x, T min_t_y, T min_t_z, T max_t_x,
T max_t_y, T max_t_z) {
min_t.x = min_t_x;
min_t.y = min_t_y;
min_t.z = min_t_z;
max_t.x = max_t_x;
max_t.y = max_t_y;
max_t.z = max_t_z;
}
__host__ __device__ AABB<T> operator+(const AABB<T> &bbox2) const {
return AABB<T>(
min(this->min_t.x, bbox2.min_t.x), min(this->min_t.y, bbox2.min_t.y),
min(this->min_t.z, bbox2.min_t.z), max(this->max_t.x, bbox2.max_t.x),
max(this->max_t.y, bbox2.max_t.y), max(this->max_t.z, bbox2.max_t.z));
};
__host__ __device__ T distance(const vec3<T> point) const {
};
__host__ __device__ T operator*(const AABB<T> &bbox2) const {
return (min(this->max_t.x, bbox2.max_t.x) -
max(this->min_t.x, bbox2.min_t.x)) *
(min(this->max_t.y, bbox2.max_t.y) -
max(this->min_t.y, bbox2.min_t.y)) *
(min(this->max_t.z, bbox2.max_t.z) -
max(this->min_t.z, bbox2.min_t.z));
};
vec3<T> min_t;
vec3<T> max_t;
};
template <typename T>
std::ostream &operator<<(std::ostream &os, const AABB<T> &x) {
os << x.min_t << std::endl;
os << x.max_t << std::endl;
return os;
}
template <typename T> struct MergeAABB {
public:
__host__ __device__ MergeAABB(){};
// Create an operator Struct that will be used by thrust::reduce
// to calculate the bounding box of the scene.
__host__ __device__ AABB<T> operator()(const AABB<T> &bbox1,
const AABB<T> &bbox2) {
return bbox1 + bbox2;
};
};
template <typename T>
__forceinline__
__host__ __device__ T pointToAABBDistance(vec3<T> point, const AABB<T>& bbox ) {
T diff_x = point.x - clamp<T>(point.x, bbox.min_t.x, bbox.max_t.x);
T diff_y = point.y - clamp<T>(point.y, bbox.min_t.y, bbox.max_t.y);
T diff_z = point.z - clamp<T>(point.z, bbox.min_t.z, bbox.max_t.z);
return diff_x * diff_x + diff_y * diff_y + diff_z * diff_z;
}
#endif // ifndef AABB_H
data/SHAPY/mesh-mesh-intersection/include/defs.hpp
-68
View File
@@ -1,68 +0,0 @@
/*
* Max-Planck-Gesellschaft zur Förderung der Wissenschaften e.V. (MPG) is
* holder of all proprietary rights on this computer program.
* You can only use this computer program if you have closed
* a license agreement with MPG or you get the right to use the computer
* program from someone who is authorized to grant you that right.
* Any use of the computer program without a valid license is prohibited and
* liable to prosecution.
*
* Copyright©2019 Max-Planck-Gesellschaft zur Förderung
* der Wissenschaften e.V. (MPG). acting on behalf of its Max Planck Institute
* for Intelligent Systems. All rights reserved.
*
* @author Vasileios Choutas
* Contact: vassilis.choutas@tuebingen.mpg.de
* Contact: ps-license@tuebingen.mpg.de
*
*/
#ifndef DEFINITIONS_H
#define DEFINITIONS_H
#include <cuda.h>
#include "device_launch_parameters.h"
#include <cuda_runtime.h>
template <typename T>
using vec3 = typename std::conditional<std::is_same<T, float>::value, float3,
double3>::type;
template <typename T>
using vec2 = typename std::conditional<std::is_same<T, float>::value, float2,
double2>::type;
float3 make_float3(double3 vec) {
return make_float3(vec.x, vec.y, vec.z);
}
float3 make_float3(double x, double y, double z) {
return make_float3(x, y, z);
}
double3 make_double3(float3 vec) {
return make_double3(vec.x, vec.y, vec.z);
}
double3 make_double3(float x, float y, float z) {
return make_double3(x, y, z);
}
template <typename T>
__host__ __device__
vec3<T> make_vec3(T x, T y, T z) {
}
template <>
__host__ __device__
vec3<float> make_vec3(float x, float y, float z) {
return make_float3(static_cast<float>(x), static_cast<float>(y), static_cast<float>(z));
}
template <>
__host__ __device__
vec3<double> make_vec3(double x, double y, double z) {
return make_double3(static_cast<double>(x), static_cast<double>(y), static_cast<double>(z));
}
#endif // ifndef DEFINITIONS_H
data/SHAPY/mesh-mesh-intersection/include/double_vec_ops.h
-117
View File
@@ -1,117 +0,0 @@
/*
* Max-Planck-Gesellschaft zur Förderung der Wissenschaften e.V. (MPG) is
* holder of all proprietary rights on this computer program.
* You can only use this computer program if you have closed
* a license agreement with MPG or you get the right to use the computer
* program from someone who is authorized to grant you that right.
* Any use of the computer program without a valid license is prohibited and
* liable to prosecution.
*
* Copyright©2019 Max-Planck-Gesellschaft zur Förderung
* der Wissenschaften e.V. (MPG). acting on behalf of its Max Planck Institute
* for Intelligent Systems. All rights reserved.
*
* @author Vasileios Choutas
* Contact: vassilis.choutas@tuebingen.mpg.de
* Contact: ps-license@tuebingen.mpg.de
*
*/
#ifndef DOUBLE_VEC_OPS_H
#define DOUBLE_VEC_OPS_H
#include "cuda_runtime.h"
inline __host__ __device__ double2 operator+(double2 a, double2 b) {
return make_double2(a.x + b.x, a.y + b.y);
}
inline __host__ __device__ double3 operator+(double3 a, double3 b) {
return make_double3(a.x + b.x, a.y + b.y, a.z + b.z);
}
inline __host__ __device__ void operator/=(double2 &a, double2 b) {
a.x /= b.x;
a.y /= b.y;
}
inline __host__ __device__ double2 operator/(double2 a, double b) {
return make_double2(a.x / b, a.y / b);
}
inline __host__ __device__ double3 operator/(double3 a, double3 b) {
return make_double3(a.x / b.x, a.y / b.y, a.z / b.z);
}
inline __host__ __device__ double3 operator*(double a, double3 b) {
return make_double3(a * b.x, a * b.y, a * b.z);
}
inline __host__ __device__ double3 operator*(double3 a, double3 b) {
return make_double3(a.x * b.x, a.y * b.y, a.z * b.z);
}
inline __host__ __device__ void operator/=(double3 &a, double3 b) {
a.x /= b.x;
a.y /= b.y;
a.z /= b.z;
}
inline __host__ __device__ double3 operator/(double3 a, double b) {
return make_double3(a.x / b, a.y / b, a.z / b);
}
inline __host__ __device__ double dot(double2 a, double2 b) {
return a.x * b.x + a.y * b.y;
}
inline __host__ __device__ double dot(double3 a, double3 b) {
return a.x * b.x + a.y * b.y + a.z * b.z;
}
inline __host__ __device__ double3 cross(double3 a, double3 b)
{
return make_double3(a.y*b.z - a.z*b.y, a.z*b.x - a.x*b.z, a.x*b.y - a.y*b.x);
}
inline __host__ __device__ double2 operator-(double2 a, double2 b)
{
return make_double2(a.x - b.x, a.y - b.y);
}
inline __host__ __device__ void operator-=(double2 &a, double2 b)
{
a.x -= b.x;
a.y -= b.y;
}
inline __host__ __device__ double2 operator-(double2 a, double b)
{
return make_double2(a.x - b, a.y - b);
}
inline __host__ __device__ double2 operator-(double b, double2 a)
{
return make_double2(b - a.x, b - a.y);
}
inline __host__ __device__ double3 operator-(double3 a, double3 b)
{
return make_double3(a.x - b.x, a.y - b.y, a.z - b.z);
}
inline __host__ __device__ void operator-=(double3 &a, double3 b)
{
a.x -= b.x;
a.y -= b.y;
a.z -= b.z;
}
inline __host__ __device__ double3 operator-(double3 a, double b)
{
return make_double3(a.x - b, a.y - b, a.z - b);
}
inline __host__ __device__ double3 operator-(double b, double3 a)
{
return make_double3(b - a.x, b - a.y, b - a.z);
}
#endif // ifndef DOUBLE_VEC_OPS_H
data/SHAPY/mesh-mesh-intersection/include/double_vec_ops.hpp
-117
View File
@@ -1,117 +0,0 @@
/*
* Max-Planck-Gesellschaft zur Förderung der Wissenschaften e.V. (MPG) is
* holder of all proprietary rights on this computer program.
* You can only use this computer program if you have closed
* a license agreement with MPG or you get the right to use the computer
* program from someone who is authorized to grant you that right.
* Any use of the computer program without a valid license is prohibited and
* liable to prosecution.
*
* Copyright©2019 Max-Planck-Gesellschaft zur Förderung
* der Wissenschaften e.V. (MPG). acting on behalf of its Max Planck Institute
* for Intelligent Systems. All rights reserved.
*
* @author Vasileios Choutas
* Contact: vassilis.choutas@tuebingen.mpg.de
* Contact: ps-license@tuebingen.mpg.de
*
*/
#ifndef DOUBLE_VEC_OPS_H
#define DOUBLE_VEC_OPS_H
#include "cuda_runtime.h"
inline __host__ __device__ double2 operator+(double2 a, double2 b) {
return make_double2(a.x + b.x, a.y + b.y);
}
inline __host__ __device__ double3 operator+(double3 a, double3 b) {
return make_double3(a.x + b.x, a.y + b.y, a.z + b.z);
}
inline __host__ __device__ void operator/=(double2 &a, double2 b) {
a.x /= b.x;
a.y /= b.y;
}
inline __host__ __device__ double2 operator/(double2 a, double b) {
return make_double2(a.x / b, a.y / b);
}
inline __host__ __device__ double3 operator/(double3 a, double3 b) {
return make_double3(a.x / b.x, a.y / b.y, a.z / b.z);
}
inline __host__ __device__ double3 operator*(double a, double3 b) {
return make_double3(a * b.x, a * b.y, a * b.z);
}
inline __host__ __device__ double3 operator*(double3 a, double3 b) {
return make_double3(a.x * b.x, a.y * b.y, a.z * b.z);
}
inline __host__ __device__ void operator/=(double3 &a, double3 b) {
a.x /= b.x;
a.y /= b.y;
a.z /= b.z;
}
inline __host__ __device__ double3 operator/(double3 a, double b) {
return make_double3(a.x / b, a.y / b, a.z / b);
}
inline __host__ __device__ double dot(double2 a, double2 b) {
return a.x * b.x + a.y * b.y;
}
inline __host__ __device__ double dot(double3 a, double3 b) {
return a.x * b.x + a.y * b.y + a.z * b.z;
}
inline __host__ __device__ double3 cross(double3 a, double3 b)
{
return make_double3(a.y*b.z - a.z*b.y, a.z*b.x - a.x*b.z, a.x*b.y - a.y*b.x);
}
inline __host__ __device__ double2 operator-(double2 a, double2 b)
{
return make_double2(a.x - b.x, a.y - b.y);
}
inline __host__ __device__ void operator-=(double2 &a, double2 b)
{
a.x -= b.x;
a.y -= b.y;
}
inline __host__ __device__ double2 operator-(double2 a, double b)
{
return make_double2(a.x - b, a.y - b);
}
inline __host__ __device__ double2 operator-(double b, double2 a)
{
return make_double2(b - a.x, b - a.y);
}
inline __host__ __device__ double3 operator-(double3 a, double3 b)
{
return make_double3(a.x - b.x, a.y - b.y, a.z - b.z);
}
inline __host__ __device__ void operator-=(double3 &a, double3 b)
{
a.x -= b.x;
a.y -= b.y;
a.z -= b.z;
}
inline __host__ __device__ double3 operator-(double3 a, double b)
{
return make_double3(a.x - b, a.y - b, a.z - b);
}
inline __host__ __device__ double3 operator-(double b, double3 a)
{
return make_double3(b - a.x, b - a.y, b - a.z);
}
#endif // ifndef DOUBLE_VEC_OPS_H
data/SHAPY/mesh-mesh-intersection/include/helper_math.h
-1453
View File
File diff suppressed because it is too large Load Diff
data/SHAPY/mesh-mesh-intersection/include/math_utils.hpp
-58
View File
@@ -1,58 +0,0 @@
/*
* Max-Planck-Gesellschaft zur Förderung der Wissenschaften e.V. (MPG) is
* holder of all proprietary rights on this computer program.
* You can only use this computer program if you have closed
* a license agreement with MPG or you get the right to use the computer
* program from someone who is authorized to grant you that right.
* Any use of the computer program without a valid license is prohibited and
* liable to prosecution.
*
* Copyright©2019 Max-Planck-Gesellschaft zur Förderung
* der Wissenschaften e.V. (MPG). acting on behalf of its Max Planck Institute
* for Intelligent Systems. All rights reserved.
*
* @author Vasileios Choutas
* Contact: vassilis.choutas@tuebingen.mpg.de
* Contact: ps-license@tuebingen.mpg.de
*
*/
#ifndef MATH_UTILS_H
#define MATH_UTILS_H
#include <cuda.h>
#include "device_launch_parameters.h"
#include <cuda_runtime.h>
#include "defs.hpp"
#include "double_vec_ops.hpp"
#include "helper_math.h"
template <typename T>
__host__ __device__ T sign(T x) {
return x > 0 ? 1 : (x < 0 ? -1 : 0);
}
template <typename T>
__host__ __device__ __forceinline__ float vec_abs_diff(const vec3<T> &vec1,
const vec3<T> &vec2) {
return fabs(vec1.x - vec2.x) + fabs(vec1.y - vec2.y) + fabs(vec1.z - vec2.z);
}
template <typename T>
__host__ __device__ __forceinline__ float vec_sq_diff(const vec3<T> &vec1,
const vec3<T> &vec2) {
return dot(vec1 - vec2, vec1 - vec2);
}
template <typename T>
__host__ __device__ __forceinline__ T clamp(T value, T min_value, T max_value) {
return min(max(value, min_value), max_value);
}
template <typename T> __host__ __device__ T dot2(vec3<T> v) {
return dot(v, v);
}
#endif // ifndef MATH_UTILS_H
data/SHAPY/mesh-mesh-intersection/include/moller.h
-704
View File
@@ -1,704 +0,0 @@
/* Triangle/triangle intersection test routine,
* by Tomas Moller, 1997.
* See article "A Fast Triangle-Triangle Intersection Test",
* Journal of Graphics Tools, 2(2), 1997
* updated: 2001-06-20 (added line of intersection)
*
* int tri_tri_intersect(float V0[3],float V1[3],float V2[3],
* float U0[3],float U1[3],float U2[3])
*
* parameters: vertices of triangle 1: V0,V1,V2
* vertices of triangle 2: U0,U1,U2
* result : returns 1 if the triangles intersect, otherwise 0
*
* Here is a version withouts divisions (a little faster)
* int NoDivTriTriIsect(float V0[3],float V1[3],float V2[3],
* float U0[3],float U1[3],float U2[3]);
*
* This version computes the line of intersection as well (if they are not
*coplanar): int tri_tri_intersect_with_isectline(float V0[3],float V1[3],float
*V2[3], float U0[3],float U1[3],float U2[3],int *coplanar, float
*isect_point1[3],float isect_point2[3]); coplanar returns whether the tris are
*coplanar isect_point1, isect_point2 are the endpoints of the line of
*intersection
*/
#include <math.h>
#include "defs.h"
#include "double_vec_ops.hpp"
#include "helper_math.h"
#define FABS(x) ((float)fabs(x)) /* implement as is fastest on your machine */
/* if USE_EPSILON_TEST is true then we do a check:
if |dv|<EPSILON then dv=0.0;
else no check is done (which is less robust)
*/
#define USE_EPSILON_TEST TRUE
#define EPSILON 0.000001
template <typename T> __host__ __device__ inline void sort(T *a, T *b) {
if (a > b) {
T c;
c = *a;
*a = *b;
b = *c;
}
return;
}
template <typename T> __host__ __device__ inline int sort(T *a, T *b) {
if (a > b) {
T c;
c = *a;
*a = *b;
b = *c;
return 1;
} else
return 0;
}
#define ISECT(VV0, VV1, VV2, D0, D1, D2, isect0, isect1) \
isect0 = VV0 + (VV1 - VV0) * D0 / (D0 - D1); \
isect1 = VV0 + (VV2 - VV0) * D0 / (D0 - D2);
#define COMPUTE_INTERVALS(VV0, VV1, VV2, D0, D1, D2, D0D1, D0D2, isect0, \
isect1) \
if (D0D1 > 0.0f) { \
/* here we know that D0D2<=0.0 */ \
/* that is D0, D1 are on the same side, D2 on the other or on the plane */ \
ISECT(VV2, VV0, VV1, D2, D0, D1, isect0, isect1); \
} else if (D0D2 > 0.0f) { \
/* here we know that d0d1<=0.0 */ \
ISECT(VV1, VV0, VV2, D1, D0, D2, isect0, isect1); \
} else if (D1 * D2 > 0.0f || D0 != 0.0f) { \
/* here we know that d0d1<=0.0 or that D0!=0.0 */ \
ISECT(VV0, VV1, VV2, D0, D1, D2, isect0, isect1); \
} else if (D1 != 0.0f) { \
ISECT(VV1, VV0, VV2, D1, D0, D2, isect0, isect1); \
} else if (D2 != 0.0f) { \
ISECT(VV2, VV0, VV1, D2, D0, D1, isect0, isect1); \
} else { \
/* triangles are coplanar */ \
return coplanar_tri_tri(N1, V0, V1, V2, U0, U1, U2); \
}
template <typename T>
inline void compute_intervals(
)
/* this edge to edge test is based on Franlin Antonio's gem:
"Faster Line Segment Intersection", in Graphics Gems III,
pp. 199-202 */
#define EDGE_EDGE_TEST(V0, U0, U1) \
Bx = U0[i0] - U1[i0]; \
By = U0[i1] - U1[i1]; \
Cx = V0[i0] - U0[i0]; \
Cy = V0[i1] - U0[i1]; \
f = Ay * Bx - Ax * By; \
d = By * Cx - Bx * Cy; \
if ((f > 0 && d >= 0 && d <= f) || (f < 0 && d <= 0 && d >= f)) { \
e = Ax * Cy - Ay * Cx; \
if (f > 0) { \
if (e >= 0 && e <= f) \
return 1; \
} else { \
if (e <= 0 && e >= f) \
return 1; \
} \
}
#define EDGE_AGAINST_TRI_EDGES(V0, V1, U0, U1, U2) \
{ \
float Ax, Ay, Bx, By, Cx, Cy, e, d, f; \
Ax = V1[i0] - V0[i0]; \
Ay = V1[i1] - V0[i1]; \
/* test edge U0,U1 against V0,V1 */ \
EDGE_EDGE_TEST(V0, U0, U1); \
/* test edge U1,U2 against V0,V1 */ \
EDGE_EDGE_TEST(V0, U1, U2); \
/* test edge U2,U1 against V0,V1 */ \
EDGE_EDGE_TEST(V0, U2, U0); \
}
#define POINT_IN_TRI(V0, U0, U1, U2) \
{ \
float a, b, c, d0, d1, d2; \
/* is T1 completly inside T2? */ \
/* check if V0 is inside tri(U0,U1,U2) */ \
a = U1[i1] - U0[i1]; \
b = -(U1[i0] - U0[i0]); \
c = -a * U0[i0] - b * U0[i1]; \
d0 = a * V0[i0] + b * V0[i1] + c; \
\
a = U2[i1] - U1[i1]; \
b = -(U2[i0] - U1[i0]); \
c = -a * U1[i0] - b * U1[i1]; \
d1 = a * V0[i0] + b * V0[i1] + c; \
\
a = U0[i1] - U2[i1]; \
b = -(U0[i0] - U2[i0]); \
c = -a * U2[i0] - b * U2[i1]; \
d2 = a * V0[i0] + b * V0[i1] + c; \
if (d0 * d1 > 0.0) { \
if (d0 * d2 > 0.0) \
return 1; \
} \
}
template <typename T>
__host__ __device__
bool point_in_tri(vec3<T> V0, vec3<T>U0, vec3<T>U1, vec3<T>U2)
{
T a, b, c, d0, d1, d2;
/* is T1 completly inside T2? */
/* check if V0 is inside tri(U0,U1,U2) */
a = U1[i1] - U0[i1];
b = -(U1[i0] - U0[i0]);
c = -a * U0[i0] - b * U0[i1];
d0 = a * V0[i0] + b * V0[i1] + c;
a = U2[i1] - U1[i1];
b = -(U2[i0] - U1[i0]);
c = -a * U1[i0] - b * U1[i1];
d1 = a * V0[i0] + b * V0[i1] + c;
a = U0[i1] - U2[i1];
b = -(U0[i0] - U2[i0]);
c = -a * U2[i0] - b * U2[i1];
d2 = a * V0[i0] + b * V0[i1] + c;
if (d0 * d1 > 0.0) {
if (d0 * d2 > 0.0)
return true;
}
return false
}
template <typename T>
__host__ __device__
bool coplanar_tri_tri(
vec3<T> N, vec3<T> V0, vec3<T> V1, vec3<T> V2,
vec3<T> U0, vec3<T> U1, vec3<T> U2) {
vec3<T> A;
short i0, i1;
/* first project onto an axis-aligned plane, that maximizes the area */
/* of the triangles, compute indices: i0,i1. */
A.x = fabs(N.x);
A.y = fabs(N.y);
A.z = fabs(N.z);
if (A.x > A.y) {
if (A.x > A.z) {
i0 = 1; /* A[0] is greatest */
i1 = 2;
} else {
i0 = 0; /* A[2] is greatest */
i1 = 1;
}
} else /* A[0]<=A[1] */
{
if (A.z > A.y) {
i0 = 0; /* A[2] is greatest */
i1 = 1;
} else {
i0 = 0; /* A[1] is greatest */
i1 = 2;
}
}
/* test all edges of triangle 1 against the edges of triangle 2 */
EDGE_AGAINST_TRI_EDGES(V0, V1, U0, U1, U2);
EDGE_AGAINST_TRI_EDGES(V1, V2, U0, U1, U2);
EDGE_AGAINST_TRI_EDGES(V2, V0, U0, U1, U2);
/* finally, test if tri1 is totally contained in tri2 or vice versa */
POINT_IN_TRI(V0, U0, U1, U2);
POINT_IN_TRI(U0, V0, V1, V2);
return false
}
int tri_tri_intersect(float V0[3], float V1[3], float V2[3], float U0[3],
float U1[3], float U2[3]) {
float E1[3], E2[3];
float N1[3], N2[3], d1, d2;
float du0, du1, du2, dv0, dv1, dv2;
float D[3];
float isect1[2], isect2[2];
float du0du1, du0du2, dv0dv1, dv0dv2;
short index;
float vp0, vp1, vp2;
float up0, up1, up2;
float b, c, max;
/* compute plane equation of triangle(V0,V1,V2) */
SUB(E1, V1, V0);
SUB(E2, V2, V0);
CROSS(N1, E1, E2);
d1 = -DOT(N1, V0);
/* plane equation 1: N1.X+d1=0 */
/* put U0,U1,U2 into plane equation 1 to compute signed distances to the
* plane*/
du0 = DOT(N1, U0) + d1;
du1 = DOT(N1, U1) + d1;
du2 = DOT(N1, U2) + d1;
/* coplanarity robustness check */
#if USE_EPSILON_TEST == TRUE
if (fabs(du0) < EPSILON)
du0 = 0.0;
if (fabs(du1) < EPSILON)
du1 = 0.0;
if (fabs(du2) < EPSILON)
du2 = 0.0;
#endif
du0du1 = du0 * du1;
du0du2 = du0 * du2;
if (du0du1 > 0.0f &&
du0du2 > 0.0f) /* same sign on all of them + not equal 0 ? */
return 0; /* no intersection occurs */
/* compute plane of triangle (U0,U1,U2) */
SUB(E1, U1, U0);
SUB(E2, U2, U0);
CROSS(N2, E1, E2);
d2 = -DOT(N2, U0);
/* plane equation 2: N2.X+d2=0 */
/* put V0,V1,V2 into plane equation 2 */
dv0 = DOT(N2, V0) + d2;
dv1 = DOT(N2, V1) + d2;
dv2 = DOT(N2, V2) + d2;
#if USE_EPSILON_TEST == TRUE
if (fabs(dv0) < EPSILON)
dv0 = 0.0;
if (fabs(dv1) < EPSILON)
dv1 = 0.0;
if (fabs(dv2) < EPSILON)
dv2 = 0.0;
#endif
dv0dv1 = dv0 * dv1;
dv0dv2 = dv0 * dv2;
if (dv0dv1 > 0.0f &&
dv0dv2 > 0.0f) /* same sign on all of them + not equal 0 ? */
return 0; /* no intersection occurs */
/* compute direction of intersection line */
CROSS(D, N1, N2);
/* compute and index to the largest component of D */
max = fabs(D[0]);
index = 0;
b = fabs(D[1]);
c = fabs(D[2]);
if (b > max)
max = b, index = 1;
if (c > max)
max = c, index = 2;
/* this is the simplified projection onto L*/
vp0 = V0[index];
vp1 = V1[index];
vp2 = V2[index];
up0 = U0[index];
up1 = U1[index];
up2 = U2[index];
/* compute interval for triangle 1 */
COMPUTE_INTERVALS(vp0, vp1, vp2, dv0, dv1, dv2, dv0dv1, dv0dv2, isect1[0],
isect1[1]);
/* compute interval for triangle 2 */
COMPUTE_INTERVALS(up0, up1, up2, du0, du1, du2, du0du1, du0du2, isect2[0],
isect2[1]);
sort(&isect1[0], &isect1[1]);
sort(&isect2[0], &isect2[1]);
if (isect1[1] < isect2[0] || isect2[1] < isect1[0])
return 0;
return 1;
}
#define NEWCOMPUTE_INTERVALS(VV0, VV1, VV2, D0, D1, D2, D0D1, D0D2, A, B, C, \
X0, X1) \
{ \
if (D0D1 > 0.0f) { \
/* here we know that D0D2<=0.0 */ \
/* that is D0, D1 are on the same side, D2 on the other or on the plane \
*/ \
A = VV2; \
B = (VV0 - VV2) * D2; \
C = (VV1 - VV2) * D2; \
X0 = D2 - D0; \
X1 = D2 - D1; \
} else if (D0D2 > 0.0f) { \
/* here we know that d0d1<=0.0 */ \
A = VV1; \
B = (VV0 - VV1) * D1; \
C = (VV2 - VV1) * D1; \
X0 = D1 - D0; \
X1 = D1 - D2; \
} else if (D1 * D2 > 0.0f || D0 != 0.0f) { \
/* here we know that d0d1<=0.0 or that D0!=0.0 */ \
A = VV0; \
B = (VV1 - VV0) * D0; \
C = (VV2 - VV0) * D0; \
X0 = D0 - D1; \
X1 = D0 - D2; \
} else if (D1 != 0.0f) { \
A = VV1; \
B = (VV0 - VV1) * D1; \
C = (VV2 - VV1) * D1; \
X0 = D1 - D0; \
X1 = D1 - D2; \
} else if (D2 != 0.0f) { \
A = VV2; \
B = (VV0 - VV2) * D2; \
C = (VV1 - VV2) * D2; \
X0 = D2 - D0; \
X1 = D2 - D1; \
} else { \
/* triangles are coplanar */ \
return coplanar_tri_tri(N1, V0, V1, V2, U0, U1, U2); \
} \
}
int NoDivTriTriIsect(float V0[3], float V1[3], float V2[3], float U0[3],
float U1[3], float U2[3]) {
float E1[3], E2[3];
float N1[3], N2[3], d1, d2;
float du0, du1, du2, dv0, dv1, dv2;
float D[3];
float isect1[2], isect2[2];
float du0du1, du0du2, dv0dv1, dv0dv2;
short index;
float vp0, vp1, vp2;
float up0, up1, up2;
float bb, cc, max;
float a, b, c, x0, x1;
float d, e, f, y0, y1;
float xx, yy, xxyy, tmp;
/* compute plane equation of triangle(V0,V1,V2) */
SUB(E1, V1, V0);
SUB(E2, V2, V0);
CROSS(N1, E1, E2);
d1 = -DOT(N1, V0);
/* plane equation 1: N1.X+d1=0 */
/* put U0,U1,U2 into plane equation 1 to compute signed distances to the
* plane*/
du0 = DOT(N1, U0) + d1;
du1 = DOT(N1, U1) + d1;
du2 = DOT(N1, U2) + d1;
/* coplanarity robustness check */
#if USE_EPSILON_TEST == TRUE
if (FABS(du0) < EPSILON)
du0 = 0.0;
if (FABS(du1) < EPSILON)
du1 = 0.0;
if (FABS(du2) < EPSILON)
du2 = 0.0;
#endif
du0du1 = du0 * du1;
du0du2 = du0 * du2;
if (du0du1 > 0.0f &&
du0du2 > 0.0f) /* same sign on all of them + not equal 0 ? */
return 0; /* no intersection occurs */
/* compute plane of triangle (U0,U1,U2) */
SUB(E1, U1, U0);
SUB(E2, U2, U0);
CROSS(N2, E1, E2);
d2 = -DOT(N2, U0);
/* plane equation 2: N2.X+d2=0 */
/* put V0,V1,V2 into plane equation 2 */
dv0 = DOT(N2, V0) + d2;
dv1 = DOT(N2, V1) + d2;
dv2 = DOT(N2, V2) + d2;
#if USE_EPSILON_TEST == TRUE
if (FABS(dv0) < EPSILON)
dv0 = 0.0;
if (FABS(dv1) < EPSILON)
dv1 = 0.0;
if (FABS(dv2) < EPSILON)
dv2 = 0.0;
#endif
dv0dv1 = dv0 * dv1;
dv0dv2 = dv0 * dv2;
if (dv0dv1 > 0.0f &&
dv0dv2 > 0.0f) /* same sign on all of them + not equal 0 ? */
return 0; /* no intersection occurs */
/* compute direction of intersection line */
CROSS(D, N1, N2);
/* compute and index to the largest component of D */
max = (float)FABS(D[0]);
index = 0;
bb = (float)FABS(D[1]);
cc = (float)FABS(D[2]);
if (bb > max)
max = bb, index = 1;
if (cc > max)
max = cc, index = 2;
/* this is the simplified projection onto L*/
vp0 = V0[index];
vp1 = V1[index];
vp2 = V2[index];
up0 = U0[index];
up1 = U1[index];
up2 = U2[index];
/* compute interval for triangle 1 */
NEWCOMPUTE_INTERVALS(vp0, vp1, vp2, dv0, dv1, dv2, dv0dv1, dv0dv2, a, b, c,
x0, x1);
/* compute interval for triangle 2 */
NEWCOMPUTE_INTERVALS(up0, up1, up2, du0, du1, du2, du0du1, du0du2, d, e, f,
y0, y1);
xx = x0 * x1;
yy = y0 * y1;
xxyy = xx * yy;
tmp = a * xxyy;
isect1[0] = tmp + b * x1 * yy;
isect1[1] = tmp + c * x0 * yy;
tmp = d * xxyy;
isect2[0] = tmp + e * xx * y1;
isect2[1] = tmp + f * xx * y0;
SORT(isect1[0], isect1[1]);
SORT(isect2[0], isect2[1]);
if (isect1[1] < isect2[0] || isect2[1] < isect1[0])
return 0;
return 1;
}
template <typename T>
inline void isect2(vec3<T> VTX0, vec3<T> VTX1, vec3<T> VTX2, T VV0, T VV1,
T VV2, T D0, T D1, T D2, T *isect0, T *isect1,
vec3<T> isectpoint0, vec3<T> isectpoint1) {
T tmp = D0 / (D0 - D1);
T diff[3];
*isect0 = VV0 + (VV1 - VV0) * tmp;
SUB(diff, VTX1, VTX0);
MULT(diff, diff, tmp);
ADD(isectpoint0, diff, VTX0);
tmp = D0 / (D0 - D2);
*isect1 = VV0 + (VV2 - VV0) * tmp;
SUB(diff, VTX2, VTX0);
MULT(diff, diff, tmp);
ADD(isectpoint1, VTX0, diff);
}
template <typename T>
__host__ __device__ inline bool
compute_intervals_isectline(vec3<T> VERT0, vec3<T> VERT1, vec3<T> VERT2, T VV0,
T VV1, T VV2, T D0, T D1, T D2, T D0D1, T D0D2,
T *isect0, T *isect1, vec3<T> isectpoint0,
vec3<T> isectpoint1) {
if (D0D1 > 0.0f) {
/* here we know that D0D2<=0.0 */
/* that is D0, D1 are on the same side, D2 on the other or on the plane */
isect2(VERT2, VERT0, VERT1, VV2, VV0, VV1, D2, D0, D1, isect0, isect1,
isectpoint0, isectpoint1);
} else if (D0D2 > 0.0f) {
/* here we know that d0d1<=0.0 */
isect2(VERT1, VERT0, VERT2, VV1, VV0, VV2, D1, D0, D2, isect0, isect1,
isectpoint0, isectpoint1);
} else if (D1 * D2 > 0.0f || D0 != 0.0f) {
/* here we know that d0d1<=0.0 or that D0!=0.0 */
isect2(VERT0, VERT1, VERT2, VV0, VV1, VV2, D0, D1, D2, isect0, isect1,
isectpoint0, isectpoint1);
} else if (D1 != 0.0f) {
isect2(VERT1, VERT0, VERT2, VV1, VV0, VV2, D1, D0, D2, isect0, isect1,
isectpoint0, isectpoint1);
} else if (D2 != 0.0f) {
isect2(VERT2, VERT0, VERT1, VV2, VV0, VV1, D2, D0, D1, isect0, isect1,
isectpoint0, isectpoint1);
} else {
/* triangles are coplanar */
return 1;
}
return 0;
}
template <typename T>
__host__ __device__ inline bool tri_tri_intersect_with_isectline(
vec3<T> V0, vec3<T> V1, vec3<T> V2, vec3<T> U0, vec3<T> U1, vec3<T> U2,
bool *coplanar, vec3<T> isect_point1[3], vec3<T> isect_point2[3]) {
vec3<T> E1, E2;
vec3<T> N1, N2, d1, d2;
vec3<T> D;
T du0, du1, du2, dv0, dv1, dv2;
vec2<T> isect1, isect2;
vec3<T> isectpointA1, isectpointA2;
vec3<T> isectpointB1, isectpointB2;
T du0du1, du0du2, dv0dv1, dv0dv2;
short index;
T vp0, vp1, vp2;
T up0, up1, up2;
T b, c, max;
T tmp, diff[3];
int smallest1, smallest2;
/* compute plane equation of triangle(V0,V1,V2) */
E1 = V1 - V0;
E2 = V2 - V0;
N1 = cross(E1, E2) d1 = -dot(N1, V0);
/* plane equation 1: N1.X+d1=0 */
/* put U0,U1,U2 into plane equation 1 to compute signed distances to the
* plane*/
du0 = dot(N1, U0) + d1;
du1 = dot(N1, U1) + d1;
du2 = dot(N1, U2) + d1;
/* coplanarity robustness check */
#if USE_EPSILON_TEST == TRUE
if (fabs(du0) < EPSILON)
du0 = 0.0;
if (fabs(du1) < EPSILON)
du1 = 0.0;
if (fabs(du2) < EPSILON)
du2 = 0.0;
#endif
du0du1 = du0 * du1;
du0du2 = du0 * du2;
if (du0du1 > 0.0f &&
du0du2 > 0.0f) /* same sign on all of them + not equal 0 ? */
return 0; /* no intersection occurs */
/* compute plane of triangle (U0,U1,U2) */
E1 = U1 - U0;
E2 = U2 - U0;
N2 = cross(E1, E2);
d2 = -dot(N2, U0);
/* plane equation 2: N2.X+d2=0 */
/* put V0,V1,V2 into plane equation 2 */
dv0 = dot(N2, V0) + d2;
dv1 = dot(N2, V1) + d2;
dv2 = dot(N2, V2) + d2;
#if USE_EPSILON_TEST == TRUE
if (fabs(dv0) < EPSILON)
dv0 = 0.0;
if (fabs(dv1) < EPSILON)
dv1 = 0.0;
if (fabs(dv2) < EPSILON)
dv2 = 0.0;
#endif
dv0dv1 = dv0 * dv1;
dv0dv2 = dv0 * dv2;
/* same sign on all of them + not equal 0 ? */
if (dv0dv1 > 0.0f && dv0dv2 > 0.0f)
return 0; /* no intersection occurs */
/* compute direction of intersection line */
D = cross(N1, N2);
/* compute and index to the largest component of D */
max = fabs(D.x);
index = 0;
b = fabs(D.y);
c = fabs(D.z);
if (b > max)
max = b, index = 1;
if (c > max)
max = c, index = 2;
/* this is the simplified projection onto L*/
vp0 = V0[index];
vp1 = V1[index];
vp2 = V2[index];
up0 = U0[index];
up1 = U1[index];
up2 = U2[index];
/* compute interval for triangle 1 */
*coplanar = compute_intervals_isectline(
V0, V1, V2, vp0, vp1, vp2, dv0, dv1, dv2, dv0dv1, dv0dv2, &isect1[0],
&isect1[1], isectpointA1, isectpointA2);
if (*coplanar)
return coplanar_tri_tri(N1, V0, V1, V2, U0, U1, U2);
/* compute interval for triangle 2 */
compute_intervals_isectline(U0, U1, U2, up0, up1, up2, du0, du1, du2, du0du1,
du0du2, &isect2[0], &isect2[1], isectpointB1,
isectpointB2);
smallest1 = sort(&isect1[0], &isect1[1]);
smallest2 = sort(&isect2[0], &isect2[1]);
if (isect1[1] < isect2[0] || isect2[1] < isect1[0])
return 0;
/* at this point, we know that the triangles intersect */
if (isect2[0] < isect1[0]) {
if (smallest1 == 0) {
SET(isect_point1, isectpointA1);
} else {
SET(isect_point1, isectpointA2);
}
if (isect2[1] < isect1[1]) {
if (smallest2 == 0) {
SET(isect_point2, isectpointB2);
} else {
SET(isect_point2, isectpointB1);
}
} else {
if (smallest1 == 0) {
SET(isect_point2, isectpointA2);
} else {
SET(isect_point2, isectpointA1);
}
}
} else {
if (smallest2 == 0) {
SET(isect_point1, isectpointB1);
} else {
SET(isect_point1, isectpointB2);
}
if (isect2[1] > isect1[1]) {
if (smallest1 == 0) {
SET(isect_point2, isectpointA2);
} else {
SET(isect_point2, isectpointA1);
}
} else {
if (smallest2 == 0) {
SET(isect_point2, isectpointB2);
} else {
SET(isect_point2, isectpointB1);
}
}
}
return 1;
}
data/SHAPY/mesh-mesh-intersection/include/priority_queue.hpp
-169
View File
@@ -1,169 +0,0 @@
/*
* Max-Planck-Gesellschaft zur Förderung der Wissenschaften e.V. (MPG) is
* holder of all proprietary rights on this computer program.
* You can only use this computer program if you have closed
* a license agreement with MPG or you get the right to use the computer
* program from someone who is authorized to grant you that right.
* Any use of the computer program without a valid license is prohibited and
* liable to prosecution.
*
* Copyright©2019 Max-Planck-Gesellschaft zur Förderung
* der Wissenschaften e.V. (MPG). acting on behalf of its Max Planck Institute
* for Intelligent Systems. All rights reserved.
*
* @author Vasileios Choutas
* Contact: vassilis.choutas@tuebingen.mpg.de
* Contact: ps-license@tuebingen.mpg.de
*
*/
#ifndef PRIORITY_QUEUE_H
#define PRIORITY_QUEUE_H
#include <float.h>
#include <stdio.h>
#include <utility>
#include <cuda.h>
#include "device_launch_parameters.h"
#include <cuda_runtime.h>
template<typename T>
__host__ __device__
void swap_array_els(T* array, int i, int j) {
T tmp = array[i];
array[i] = array[j];
array[j] = tmp;
}
template <typename T, typename Obj, int QueueSize = 128, bool recursive = false>
class PriorityQueue {
public:
__host__ __device__
PriorityQueue() : heap_size(0) {}
inline
__host__ __device__
int get_size() {
return heap_size;
}
inline
__host__ __device__
int parent(int i) {
return (i - 1) / 2;
}
inline
__host__ __device__
int left_child(int i) {
return 2 * i + 1;
}
inline
__host__ __device__
int right_child(int i) {
return 2 * i + 2;
}
__host__ __device__
std::pair<T, Obj> get_min() {
if (heap_size > 0) {
return std::pair<T, Obj>(priority_heap[0], obj_heap[0]);
}
else {
return std::pair<T, Obj>(
std::is_same<T, float>::value ? FLT_MAX : DBL_MAX, nullptr);
}
}
__host__ __device__
void min_heapify(int index) {
if (recursive) {
int left = left_child(index);
int right = right_child(index);
int smallest = index;
if (left < heap_size && priority_heap[left] < priority_heap[index])
smallest = left;
if (right < heap_size && priority_heap[right] < priority_heap[index])
smallest = right;
if (smallest != index) {
swap_array_els(priority_heap, index, smallest);
swap_array_els(obj_heap, index, smallest);
min_heapify(smallest);
}
} else {
int ii = index;
int smallest;
while (true) {
int left = left_child(ii);
int right = right_child(ii);
smallest = ii;
if (left < heap_size && priority_heap[left] < priority_heap[ii])
smallest = left;
if (right < heap_size && priority_heap[right] < priority_heap[ii])
smallest = right;
if (smallest != ii) {
swap_array_els(priority_heap, ii, smallest);
swap_array_els(obj_heap, ii, smallest);
ii = smallest;
}
else
break;
}
}
}
__host__ __device__
void insert_key(T key, Obj obj) {
if (heap_size == QueueSize) {
printf("The queue has exceed its maximum size\n");
return;
}
heap_size++;
int ii = heap_size - 1;
priority_heap[ii] = key;
obj_heap[ii] = obj;
// Fix the min heap property if it is violated
min_heapify(0);
// while (ii != 0 && priority_heap[parent(ii)] > priority_heap[ii]) {
// swap_array_els(priority_heap, ii, parent(ii));
// swap_array_els(obj_heap, ii, parent(ii));
// ii = parent(ii);
// }
}
// void print() {
// for (int i = 0; i < heap_size; i++) {
// std::cout << i << ": " << heap[i] << std::endl;
// }
// }
__host__ __device__
std::pair<T, Obj> extract() {
if (heap_size <= 0)
return std::pair<T, Obj>(
std::is_same<T, float>::value ? FLT_MAX : DBL_MAX, nullptr);
T root_prio = priority_heap[0];
Obj root_obj = obj_heap[0];
// Replace the root with the last element
priority_heap[0] = priority_heap[heap_size - 1];
obj_heap[0] = obj_heap[heap_size - 1];
// Decrease the size of the heap
heap_size--;
min_heapify(0);
return std::pair<T, Obj>(root_prio, root_obj);
}
private:
T priority_heap[QueueSize];
Obj obj_heap[QueueSize];
int heap_size;
};
#endif // #ifndef PRIORITY_QUEUE_H
data/SHAPY/mesh-mesh-intersection/include/triangle.hpp
-66
View File
@@ -1,66 +0,0 @@
/*
* Max-Planck-Gesellschaft zur Förderung der Wissenschaften e.V. (MPG) is
* holder of all proprietary rights on this computer program.
* You can only use this computer program if you have closed
* a license agreement with MPG or you get the right to use the computer
* program from someone who is authorized to grant you that right.
* Any use of the computer program without a valid license is prohibited and
* liable to prosecution.
*
* Copyright©2019 Max-Planck-Gesellschaft zur Förderung
* der Wissenschaften e.V. (MPG). acting on behalf of its Max Planck Institute
* for Intelligent Systems. All rights reserved.
*
* @author Vasileios Choutas
* Contact: vassilis.choutas@tuebingen.mpg.de
* Contact: ps-license@tuebingen.mpg.de
*
*/
#ifndef TRIANGLE_H
#define TRIANGLE_H
#include "defs.hpp"
#include "double_vec_ops.hpp"
#include "helper_math.h"
#include "math_utils.hpp"
#include <cuda.h>
#include <cuda_profiler_api.h>
#include <cuda_runtime.h>
template <typename T>
__align__(48)
struct Triangle {
public:
vec3<T> v0;
vec3<T> v1;
vec3<T> v2;
__host__ __device__ Triangle() {}
__host__ __device__ Triangle(vec3<T> vertex0, vec3<T> vertex1,
vec3<T> vertex2)
: v0(vertex0), v1(vertex1), v2(vertex2){};
__host__ __device__ Triangle(const vec3<T> &vertex0, const vec3<T> &vertex1,
const vec3<T> &vertex2)
: v0(vertex0), v1(vertex1), v2(vertex2){};
__host__ __device__ AABB<T> bbox() const {
return AABB<T>(min(v0.x, min(v1.x, v2.x)), min(v0.y, min(v1.y, v2.y)),
min(v0.z, min(v1.z, v2.z)), max(v0.x, max(v1.x, v2.x)),
max(v0.y, max(v1.y, v2.y)), max(v0.z, max(v1.z, v2.z)));
}
};
template <typename T> using TrianglePtr = Triangle<T> *;
template <typename T>
std::ostream &operator<<(std::ostream &os, const Triangle<T> &x) {
os << x.v0 << std::endl;
os << x.v1 << std::endl;
os << x.v2 << std::endl;
return os;
}
#endif // TRIANGLE_H
data/SHAPY/mesh-mesh-intersection/mesh_mesh_intersection/__init__.py
-18
View File
@@ -1,18 +0,0 @@
# -*- coding: utf-8 -*-
# Max-Planck-Gesellschaft zur Förderung der Wissenschaften e.V. (MPG) is
# holder of all proprietary rights on this computer program.
# You can only use this computer program if you have closed
# a license agreement with MPG or you get the right to use the computer
# program from someone who is authorized to grant you that right.
# Any use of the computer program without a valid license is prohibited and
# liable to prosecution.
#
# Copyright©2020 Max-Planck-Gesellschaft zur Förderung
# der Wissenschaften e.V. (MPG). acting on behalf of its Max Planck Institute
# for Intelligent Systems. All rights reserved.
#
# Contact: ps-license@tuebingen.mpg.de
import torch
from .mesh_mesh_intersection import MeshMeshIntersection
data/SHAPY/mesh-mesh-intersection/mesh_mesh_intersection/loss.py
-317
View File
@@ -1,317 +0,0 @@
# -*- coding: utf-8 -*-
# Max-Planck-Gesellschaft zur Förderung der Wissenschaften e.V. (MPG) is
# holder of all proprietary rights on this computer program.
# You can only use this computer program if you have closed
# a license agreement with MPG or you get the right to use the computer
# program from someone who is authorized to grant you that right.
# Any use of the computer program without a valid license is prohibited and
# liable to prosecution.
#
# Copyright©2019 Max-Planck-Gesellschaft zur Förderung
# der Wissenschaften e.V. (MPG). acting on behalf of its Max Planck Institute
# for Intelligent Systems and the Max Planck Institute for Biological
# Cybernetics. All rights reserved.
#
# Contact: ps-license@tuebingen.mpg.de
from __future__ import absolute_import
from __future__ import print_function
from __future__ import division
import sys
import time
import torch
import torch.nn as nn
import torch.nn.functional as F
def calc_circumcircle(triangles, edge_cross_prod, idx=None):
''' Calculate the circumscribed circle for the given triangles
Args:
- triangles (torch.tensor BxTx3x3): The tensor that contains the
coordinates of the triangle vertices
- edge_cross_prod (torch.tensor BxCx3): Contains the unnormalized
perpendicular vector to the surface of the triangle.
Returns:
- circumradius (torch.tensor BxTx1): The radius of the
circumscribed circle
- circumcenter (torch.tensor BxTx3): The center of the
circumscribed circel
'''
alpha = triangles[:, :, 0] - triangles[:, :, 2]
beta = triangles[:, :, 1] - triangles[:, :, 2]
# Calculate the radius of the circumscribed circle
# Should be BxF
circumradius = (torch.norm(alpha - beta, dim=2, keepdim=True) /
(2 * torch.norm(edge_cross_prod, dim=2, keepdim=True)) *
torch.norm(alpha, dim=2, keepdim=True) *
torch.norm(beta, dim=2, keepdim=True))
# Calculate the coordinates of the circumcenter of each triangle
# Should BxFx3
circumcenter = torch.cross(
torch.sum(alpha ** 2, dim=2, keepdim=True) * beta -
torch.sum(beta ** 2, dim=2, keepdim=True) * alpha,
torch.cross(alpha, beta, dim=-1), dim=2)
circumcenter /= (2 * torch.sum(edge_cross_prod ** 2, dim=2, keepdim=True))
return circumradius, circumcenter + triangles[:, :, 2]
def repulsion_intensity(x, sigma=0.5, penalize_outside=True, linear_max=1000):
''' Penalizer function '''
quad_penalty = (-(1.0 - 2.0 * sigma) / (4.0 * sigma ** 2) *
x ** 2 - 1 / (2.0 * sigma) * x +
0.25 * (3 - 2 * sigma))
linear_region_mask = (x.le(-sigma) * x.gt(-linear_max)).to(dtype=x.dtype)
if penalize_outside:
quad_region_mask = (x.gt(-sigma) * x.lt(sigma)).to(dtype=x.dtype)
else:
quad_region_mask = (x.gt(-sigma) * x.lt(0)).to(dtype=x.dtype)
return (linear_region_mask * (-x + 1 - sigma) +
quad_region_mask * quad_penalty)
def dist_to_cone_axis(points_rel, dot_prod, cone_axis, cone_radius,
sigma=0.5, epsilon=1e-6, vectorized=True):
''' Computes the distance of each point to the axis
This function projects the points on the plane of the base of the cone
and computes the distance to the axis. This is subsequently normalized
by the radius of the cone at the height level of the point, so that
points with distance < 1 are in the code, distance == 1 means that the
point is on the surface and distance > 1 means that the point is
outside the cone.
Args:
- points_rel (torch.tensor BxCxNx3): The coordinates of the points
relative to the center of the cone
- dot_prod (torch.tensor BxCxN): The dot product of the points (in
relative coordinates with respect to the cone center) with the
axis of the cone
- cone_axis (torch.tensor BxCx3): The axis of the cone
- cone_radius (torch.tensor BxCx1): The radius of the cone
Keyword args:
- sigma (float = 0.5): The height of the cone
- epsilon (float = 1e-6): Numerical stability constant for the
float division
- vectorized (bool = True): Whether to use an iterative or a
vectorized version of the function
'''
if vectorized:
batch_size, num_collisions = cone_radius.shape[:2]
numerator = torch.norm(points_rel - dot_prod.unsqueeze(dim=-1) *
cone_axis.unsqueeze(dim=-2),
p=2, dim=-1)
denominator = -cone_radius / sigma * dot_prod + cone_radius
else:
batch_size, num_collisions = cone_radius.shape[:2]
numerator = torch.norm(points_rel - dot_prod.unsqueeze(-1) * cone_axis,
p=2, dim=-1)
denominator = -cone_radius.view(batch_size, num_collisions) / sigma * \
dot_prod + cone_radius.view(batch_size, num_collisions)
return numerator / (denominator + epsilon)
def conical_distance_field(triangle_points, cone_center, cone_radius,
cone_axis, sigma=0.5, vectorized=True,
penalize_outside=True, linear_max=1000):
''' Distance field calculation for a cone
Args:
- triangle_points (torch.tensor (BxCxNx3): Contains
the points whose distance from the cone we want to calculate.
- cone_center (torch.tensor (BxCx3)): The coordinates of the center
of the cone
- cone_radius (torch.tensor (BxC)): The radius of the base of the
cone
- cone_axis (torch.tensor(BxCx3)): The unit vector that represents
the axis of the cone
Keyword Arguments
- sigma (float = 0.5): The float value of the height of the cone
- vectorized (bool = True): Whether to use an iterative or a
vectorized version of the function
Returns:
- (torch.tensor BxCxN): The distance field values at the N points
for the cone
'''
if vectorized:
# Calculate the coordinates of the points relative to the center of
# the cone
points_rel = triangle_points - cone_center.unsqueeze(dim=-2)
# Calculate the dot product between the relative point coordinates and
# the axis (normal) of the cone. Essentially, it is the length of the
# projection of the relative vector on the axis of the cone
dot_prod = torch.sum(points_rel * cone_axis.unsqueeze(dim=-2), dim=-1)
# Calculate the distance of the projections of the points on the cone
# base plane to the center of cone, normalized by the height
axis_dist = dist_to_cone_axis(points_rel, dot_prod,
cone_axis, cone_radius,
sigma=sigma, vectorized=True)
circumcenter_dist = repulsion_intensity(
dot_prod, sigma=sigma, penalize_outside=penalize_outside,
linear_max=linear_max)
# Ignore the points with axis_dist > 1, since they are out of the cone
mask = axis_dist.lt(1).to(dtype=triangle_points.dtype)
distance_field = mask * ((1 - axis_dist) * circumcenter_dist).pow(2)
else:
batch_size, num_collisions, num_points = triangle_points.shape[:3]
distance_field = torch.zeros([batch_size, num_collisions, 3],
dtype=triangle_points.dtype,
device=triangle_points.device)
for idx in range(num_points):
# The relative coordinates of each point to the center of the cone
# BxCx3
points_rel = triangle_points[:, :, idx, :] - cone_center
# Calculate the dot product between the relative point coordinates
# and the axis (normal) of the cone. Essentially, it is the length
# of the projection of the relative vector on the axis of the cone
dot_prod = torch.sum(points_rel * cone_axis, dim=-1)
axis_dist = dist_to_cone_axis(points_rel, dot_prod,
cone_axis, cone_radius,
sigma=sigma,
vectorized=False)
circumcenter_dist = repulsion_intensity(
dot_prod, sigma=sigma, penalize_outside=penalize_outside)
mask = (axis_dist < 1).to(dtype=triangle_points.dtype)
distance_field[:, :, idx] = (1 - axis_dist) * mask * \
circumcenter_dist
return torch.pow(distance_field, 2)
class DistanceFieldPenetrationLoss(nn.Module):
def __init__(self, sigma=0.5, point2plane=False, vectorized=True,
penalize_outside=True, linear_max=1000):
super(DistanceFieldPenetrationLoss, self).__init__()
self.sigma = sigma
self.point2plane = point2plane
self.vectorized = vectorized
self.penalize_outside = penalize_outside
self.linear_max = linear_max
def forward(self, triangles, collision_idxs):
'''
Args:
- triangles: A torch tensor of size BxFx3x3 that contains the
coordinates of the triangle vertices
- collision_idxs: A torch tensor of size Bx(-1)x2 that contains the
indices of the colliding pairs
Returns:
A tensor with size B that contains the self penetration loss for
each mesh in the batch
'''
coll_idxs = collision_idxs[:, :, 0].ge(0).nonzero()
if len(coll_idxs) < 1:
return torch.zeros([triangles.shape[0]],
dtype=triangles.dtype,
device=triangles.device,
requires_grad=triangles.requires_grad)
receiver_faces = collision_idxs[coll_idxs[:, 0], coll_idxs[:, 1], 0]
intruder_faces = collision_idxs[coll_idxs[:, 0], coll_idxs[:, 1], 1]
batch_idxs = coll_idxs[:, 0]
num_collisions = coll_idxs.shape[0]
batch_size = triangles.shape[0]
if len(intruder_faces) < 1:
return torch.tensor(0.0, dtype=triangles.dtype,
device=triangles.device,
requires_grad=triangles.requires_grad)
# Calculate the edges of the triangles
# Size: BxFx3
edge0 = triangles[:, :, 1] - triangles[:, :, 0]
edge1 = triangles[:, :, 2] - triangles[:, :, 0]
# Compute the cross product of the edges to find the normal vector of
# the triangle
aCrossb = torch.cross(edge0, edge1, dim=2)
circumradius, circumcenter = calc_circumcircle(triangles, aCrossb)
# Normalize the result to get a unit vector
normals = aCrossb / torch.norm(aCrossb, 2, dim=2, keepdim=True)
recv_triangles = triangles[batch_idxs, receiver_faces]
intr_triangles = triangles[batch_idxs, intruder_faces]
recv_normals = normals[batch_idxs, receiver_faces]
recv_circumradius = circumradius[batch_idxs, receiver_faces]
recv_circumcenter = circumcenter[batch_idxs, receiver_faces]
intr_normals = normals[batch_idxs, intruder_faces]
intr_circumradius = circumradius[batch_idxs, intruder_faces]
intr_circumcenter = circumcenter[batch_idxs, intruder_faces]
# Compute the distance field for the intruding triangles
# B x NUM_COLLISIONS x 3
# For each batch element, for each collision pair, 3 distance values
# for the vertices of the intruding triangle
phi_receivers = conical_distance_field(
intr_triangles,
recv_circumcenter, recv_circumradius,
recv_normals,
sigma=self.sigma,
vectorized=self.vectorized,
penalize_outside=self.penalize_outside,
linear_max=self.linear_max)
# Compute the distance field for the intruding triangles
# B x NUM_COLLISIONS x 3
# For each batch element, for each collision pair, 3 distance values
# for the vertices of the intruding triangle
# Same as above, but now the receiver is the "intruder".
phi_intruders = conical_distance_field(
recv_triangles,
intr_circumcenter,
intr_circumradius,
intr_normals,
sigma=self.sigma,
vectorized=self.vectorized,
penalize_outside=self.penalize_outside,
linear_max=self.linear_max)
receiver_loss = torch.tensor(0, device=triangles.device,
dtype=torch.float32)
intruder_loss = torch.tensor(0, device=triangles.device,
dtype=torch.float32)
if self.point2plane:
receiver_loss = (-phi_receivers).pow(2).sum(dim=-1)
intruder_loss = (-phi_intruders).pow(2).sum(dim=-1)
else:
receiver_loss = torch.norm(-phi_receivers.unsqueeze(dim=-1) *
intr_normals.unsqueeze(dim=-2), p=2,
dim=-1).pow(2).sum(dim=-1)
intruder_loss = torch.norm(-phi_intruders.unsqueeze(dim=-1) *
recv_normals.unsqueeze(dim=-2), p=2,
dim=-1).pow(2).sum(dim=-1)
batch_ind = torch.arange(0, batch_size, dtype=batch_idxs.dtype,
device=triangles.device).unsqueeze(dim=1)
batch_mask = batch_ind.repeat([1, num_collisions]).eq(batch_idxs)\
.to(receiver_loss.dtype)
loss = torch.matmul(batch_mask, receiver_loss) + \
torch.matmul(batch_mask, intruder_loss)
return loss
data/SHAPY/mesh-mesh-intersection/mesh_mesh_intersection/mesh_mesh_intersection.py
-62
View File
@@ -1,62 +0,0 @@
# -*- coding: utf-8 -*-
# Max-Planck-Gesellschaft zur Förderung der Wissenschaften e.V. (MPG) is
# holder of all proprietary rights on this computer program.
# You can only use this computer program if you have closed
# a license agreement with MPG or you get the right to use the computer
# program from someone who is authorized to grant you that right.
# Any use of the computer program without a valid license is prohibited and
# liable to prosecution.
#
# Copyright©2019 Max-Planck-Gesellschaft zur Förderung
# der Wissenschaften e.V. (MPG). acting on behalf of its Max Planck Institute
# for Intelligent Systems. All rights reserved.
#
# Contact: ps-license@tuebingen.mpg.de
from __future__ import absolute_import
from __future__ import print_function
from __future__ import division
import sys
import torch
import torch.nn as nn
import torch.autograd as autograd
# from loguru import logger
import mesh_mesh_intersection
import mesh_mesh_intersect_cuda
class MeshMeshIntersectionFunction(autograd.Function):
@staticmethod
@torch.no_grad()
def forward(ctx, query_triangles, target_triangles, print_timings=False,
max_collisions=32,
*args, **kwargs):
outputs = mesh_mesh_intersect_cuda.mesh_to_mesh_forward(
query_triangles, target_triangles, print_timings=print_timings,
max_collisions=max_collisions)
# ctx.save_for_backward(query_triangles, outputs)
collision_faces, collision_bcs = outputs
return collision_faces, collision_bcs
@staticmethod
def backward(ctx, grad_output, *args, **kwargs):
raise NotImplementedError
class MeshMeshIntersection(nn.Module):
def __init__(self, max_collisions=32):
super(MeshMeshIntersection, self).__init__()
self.max_collisions = max_collisions
# MeshMeshIntersectionFunction.max_collisions = self.max_collisions
def forward(self, query_triangles, target_triangles,
print_timings=False):
return MeshMeshIntersectionFunction.apply(
query_triangles, target_triangles, print_timings,
self.max_collisions)
data/SHAPY/mesh-mesh-intersection/optional-requirements.txt
-4
View File
@@ -1,4 +0,0 @@
pyrender>=0.1.23
shapely
trimesh>=2.37.6
smplx
data/SHAPY/mesh-mesh-intersection/requirements.txt
-2
View File
@@ -1,2 +0,0 @@
numpy>=1.16.2
torch>=1.0
data/SHAPY/mesh-mesh-intersection/setup.py
-100
View File
@@ -1,100 +0,0 @@
# -*- coding: utf-8 -*-
# Max-Planck-Gesellschaft zur Förderung der Wissenschaften e.V. (MPG) is
# holder of all proprietary rights on this computer program.
# You can only use this computer program if you have closed
# a license agreement with MPG or you get the right to use the computer
# program from someone who is authorized to grant you that right.
# Any use of the computer program without a valid license is prohibited and
# liable to prosecution.
#
# Copyright©2019 Max-Planck-Gesellschaft zur Förderung
# der Wissenschaften e.V. (MPG). acting on behalf of its Max Planck Institute
# for Intelligent Systems and the Max Planck Institute for Biological
# Cybernetics. All rights reserved.
#
# Contact: ps-license@tuebingen.mpg.deimport io
import io
import os
import os.path as osp
from setuptools import find_packages, setup
import torch
from torch.utils.cpp_extension import BuildExtension, CUDAExtension
# Package meta-data.
NAME = 'mesh_mesh_intersection'
DESCRIPTION = 'PyTorch module for Mesh-Mesh intersection detection'
URL = ''
EMAIL = 'vassilis.choutas@tuebingen.mpg.de'
AUTHOR = 'Vassilis Choutas'
REQUIRES_PYTHON = '>=3.6.0'
VERSION = '0.2.0'
here = os.path.abspath(os.path.dirname(__file__))
try:
FileNotFoundError
except NameError:
FileNotFoundError = IOError
# Import the README and use it as the long-description.
# Note: this will only work if 'README.md' is present in your MANIFEST.in file!
try:
with io.open(os.path.join(here, 'README.md'), encoding='utf-8') as f:
long_description = '\n' + f.read()
except FileNotFoundError:
long_description = DESCRIPTION
# Load the package's __version__.py module as a dictionary.
about = {}
if not VERSION:
with open(os.path.join(here, NAME, '__version__.py')) as f:
exec(f.read(), about)
else:
about['__version__'] = VERSION
mesh_mesh_intersect_src_files = [
'src/mesh_mesh_intersect.cpp', 'src/mesh_mesh_intersect_cuda_op.cu']
mesh_mesh_intersect_include_dirs = torch.utils.cpp_extension.include_paths() + [
osp.abspath('include'),
osp.abspath(osp.expandvars('$CUDA_SAMPLES_INC'))]
mesh_mesh_intersect_extra_compile_args = {
'nvcc': ['-DPRINT_TIMINGS=0', '-DDEBUG_PRINT=0',
'-DERROR_CHECKING=1',
'-DCOLLISION_ORDERING=1'],
'cxx': []}
mesh_mesh_intersect_extension = CUDAExtension(
'mesh_mesh_intersect_cuda', mesh_mesh_intersect_src_files,
include_dirs=mesh_mesh_intersect_include_dirs,
extra_compile_args=mesh_mesh_intersect_extra_compile_args)
render_reqs = ['pyrender>=0.1.23', 'trimesh>=2.37.6', 'shapely']
setup(name=NAME,
version=about['__version__'],
description=DESCRIPTION,
long_description=long_description,
long_description_content_type='text/markdown',
author=AUTHOR,
author_email=EMAIL,
python_requires=REQUIRES_PYTHON,
url=URL,
packages=find_packages(),
ext_modules=[mesh_mesh_intersect_extension],
classifiers=[
"License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
"Environment :: Console",
"Programming Language :: Python",
"Programming Language :: Python :: 3.6",
"Programming Language :: Python :: 3.7"],
install_requires=[
'torch>=1.6.0',
],
extras_require={
'render': render_reqs,
'all': render_reqs
},
cmdclass={'build_ext': BuildExtension})
data/SHAPY/mesh-mesh-intersection/src/mesh_mesh_intersect.cpp
-64
View File
@@ -1,64 +0,0 @@
/*
Max-Planck-Gesellschaft zur Förderung der Wissenschaften e.V. (MPG) is
holder of all proprietary rights on this computer program.
You can only use this computer program if you have closed
a license agreement with MPG or you get the right to use the computer
program from someone who is authorized to grant you that right.
Any use of the computer program without a valid license is prohibited and
liable to prosecution.
Copyright©2019 Max-Planck-Gesellschaft zur Förderung
der Wissenschaften e.V. (MPG). acting on behalf of its Max Planck Institute
for Intelligent Systems. All rights reserved.
Contact: ps-license@tuebingen.mpg.de
*/
#include <torch/extension.h>
#include <vector>
#define CHECK_CUDA(x) \
TORCH_CHECK(x.device().type() == torch::kCUDA, #x " must be a CUDA tensor")
#define CHECK_CONTIGUOUS(x) \
TORCH_CHECK(x.is_contiguous(), #x " must be contiguous")
#define CHECK_INPUT(x) \
CHECK_CUDA(x); \
CHECK_CONTIGUOUS(x)
void mesh_mesh_intersection_forward(const torch::Tensor &query_triangles,
const torch::Tensor &target_triangles,
torch::Tensor &collision_faces,
torch::Tensor &collision_bcs,
int max_collisions = 16,
bool print_timings = false);
std::vector<torch::Tensor>
mesh_to_mesh_intersection(torch::Tensor query_triangles,
torch::Tensor target_triangles,
int max_collisions = 16, bool print_timings = false) {
CHECK_INPUT(query_triangles);
CHECK_INPUT(target_triangles);
torch::Tensor collision_faces =
-1 * torch::ones({query_triangles.size(0),
query_triangles.size(1) * max_collisions},
torch::device(query_triangles.device())
.dtype(torch::ScalarType::Long));
torch::Tensor collision_bcs = torch::zeros(
{query_triangles.size(0), query_triangles.size(1) * max_collisions, 2, 3},
torch::device(query_triangles.device()).dtype(query_triangles.dtype()));
mesh_mesh_intersection_forward(query_triangles, target_triangles,
collision_faces, collision_bcs,
max_collisions);
return {torch::autograd::make_variable(collision_faces, false),
torch::autograd::make_variable(collision_bcs, false)};
}
PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {
m.def("mesh_to_mesh_forward", &mesh_to_mesh_intersection,
"BVH mesh-to-mesh intersection forward (CUDA)",
py::arg("query_triangles"), py::arg("target_triangles"),
py::arg("max_collisions") = 16, py::arg("print_timings") = false);
}
data/SHAPY/mesh-mesh-intersection/src/mesh_mesh_intersect_cuda_op.cu
-1079
View File
File diff suppressed because it is too large Load Diff
data/SHAPY/test_submission_format.py
-62
View File
@@ -1,62 +0,0 @@
""" Ref: https://github.com/muelea/shapy/blob/master/regressor/hbw_evaluation/test_submission_format.py """
import argparse
import numpy as np
def test_submission_file_format(
npz_file: str,
model_type: str = 'smplx'
):
submission = np.load(npz_file)
# check if keys are named correctly
keys = [x for x in submission.keys()]
assert 'image_name' in keys and 'v_shaped' in keys, \
f"Keys are not correct. Got {keys}, but expected ['image_name', 'v_shaped']"
image_names = submission['image_name']
v_shapeds = submission['v_shaped']
# check if shape and type are correct
assert type(image_names) == np.ndarray, \
f"Type of key image_name is not correct. {type(image_names)} given, but np.ndarray expected."
assert image_names.shape == (1631,), \
f"Shape of key image_name is not correct. {image_names.shape} given, but (1631,) expected."
assert type(v_shapeds) == np.ndarray, \
f"Type of key v_shaped is not correct. {type(image_names)} given, but np.ndarray expected."
if model_type == 'smplx':
assert v_shapeds.shape == (1631, 10475, 3), \
f"Shape of key v_shaped is not correct. {v_shapeds.shape} given, but (1631, 10475, 3) expected."
else:
assert v_shapeds.shape == (1631, 6890, 3), \
f"Shape of key v_shaped is not correct. {v_shapeds.shape} given, but (1631, 6890, 3) expected."
# check if each image has a prediction
hbw_images_gt = np.load('../data/SHAPY/hbw_testset_image_names.npy')
check_prediction_available = np.isin(hbw_images_gt, image_names)
assert np.all(check_prediction_available), \
f"Images without predition exist! Missing predictions: \
\n {hbw_images_gt[~check_prediction_available]}"
print(f'Your submission file passed the test.')
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument('--input-npz-file',
dest='input_npz_file', type=str, required=True,
help='npz containing labels and body shape parameters.')
parser.add_argument('--model-type', choices=['smpl', 'smplx'], type=str,
default='smplx',
help='The model type used for body shape prediction. ')
args = parser.parse_args()
test_submission_file_format(
npz_file=args.input_npz_file,
model_type=args.model_type
)
data/SPEC/SPEC.py
-51
View File
@@ -1,51 +0,0 @@
import os
import os.path as osp
import numpy as np
import torch
import cv2
import json
import copy
from pycocotools.coco import COCO
from config import cfg
from utils.human_models import smpl_x
from utils.preprocessing import load_img, process_bbox, augmentation, process_db_coord, process_human_model_output, \
get_fitting_error_3D
from utils.transforms import world2cam, cam2pixel, rigid_align
from humandata import HumanDataset
class SPEC(HumanDataset):
def __init__(self, transform, data_split):
super(SPEC, self).__init__(transform, data_split)
pre_prc_file_train = 'spec_train_smpl.npz'
pre_prc_file_test = 'spec_test_smpl.npz'
if self.data_split == 'train':
filename = getattr(cfg, 'filename', pre_prc_file_train)
else:
filename = getattr(cfg, 'filename', pre_prc_file_test)
self.img_dir = osp.join(cfg.data_dir, 'SPEC')
self.annot_path = osp.join(cfg.data_dir, 'preprocessed_datasets', filename)
self.annot_path_cache = osp.join(cfg.data_dir, 'cache', filename)
self.use_cache = getattr(cfg, 'use_cache', False)
self.img_shape = (1080, 1920) # (h, w)
self.cam_param = {}
# check image shape
img_path = osp.join(self.img_dir, np.load(self.annot_path)['image_path'][0])
img_shape = cv2.imread(img_path).shape[:2]
assert self.img_shape == img_shape, 'image shape is incorrect: {} vs {}'.format(self.img_shape, img_shape)
# load data or cache
if self.use_cache and osp.isfile(self.annot_path_cache):
print(f'[{self.__class__.__name__}] loading cache from {self.annot_path_cache}')
self.datalist = self.load_cache(self.annot_path_cache)
else:
if self.use_cache:
print(f'[{self.__class__.__name__}] Cache not found, generating cache...')
self.datalist = self.load_data(
train_sample_interval=getattr(cfg, f'{self.__class__.__name__}_train_sample_interval', 1))
if self.use_cache:
self.save_cache(self.annot_path_cache, self.datalist)
data/SSP3D/SSP3D.py
-52
View File
@@ -1,52 +0,0 @@
import os
import os.path as osp
import numpy as np
import torch
import cv2
import json
import copy
from pycocotools.coco import COCO
from config import cfg
from utils.human_models import smpl_x
from utils.preprocessing import load_img, process_bbox, augmentation, process_db_coord, process_human_model_output, \
get_fitting_error_3D
from utils.transforms import world2cam, cam2pixel, rigid_align
from humandata import HumanDataset
class SSP3D(HumanDataset):
def __init__(self, transform, data_split):
super(SSP3D, self).__init__(transform, data_split)
self.use_cache = getattr(cfg, 'use_cache', False)
self.annot_path_cache = osp.join(cfg.data_dir, 'cache', 'ssp3d_230525_311.npz')
self.img_shape = (512, 512) # (h, w)
self.cam_param = {}
if self.use_cache and osp.isfile(self.annot_path_cache):
print(f'[{self.__class__.__name__}] loading cache from {self.annot_path_cache}')
self.datalist = self.load_cache(self.annot_path_cache)
else:
if self.use_cache:
print(f'[{self.__class__.__name__}] Cache not found, generating cache...')
if self.data_split == 'train':
filename = getattr(cfg, 'filename', 'ssp3d_230525_311.npz')
else:
raise ValueError('SSP3D test set is not support')
self.img_dir = osp.join(cfg.data_dir, 'SSP3D')
self.annot_path = osp.join(cfg.data_dir, 'preprocessed_datasets', filename)
# check image shape
img_path = osp.join(self.img_dir, np.load(self.annot_path)['image_path'][0])
img_shape = cv2.imread(img_path).shape[:2]
assert self.img_shape == img_shape, 'image shape is incorrect: {} vs {}'.format(self.img_shape, img_shape)
# load data or cache
self.datalist = self.load_data(
train_sample_interval=getattr(cfg, f'{self.__class__.__name__}_train_sample_interval', 1))
if self.use_cache:
self.save_cache(self.annot_path_cache, self.datalist)
data/SynBody/SynBody.py
-47
View File
@@ -1,47 +0,0 @@
import os
import os.path as osp
import numpy as np
import torch
import cv2
import json
import copy
from pycocotools.coco import COCO
from config import cfg
from utils.human_models import smpl_x
from utils.preprocessing import load_img, process_bbox, augmentation, process_db_coord, process_human_model_output, \
get_fitting_error_3D
from utils.transforms import world2cam, cam2pixel, rigid_align
from humandata import HumanDataset
class SynBody(HumanDataset):
def __init__(self, transform, data_split):
super(SynBody, self).__init__(transform, data_split)
filename = 'synbody_train_230521_04000_fix_betas.npz'
self.img_dir = osp.join(cfg.data_dir, 'SynBody')
self.annot_path = osp.join(cfg.data_dir, 'preprocessed_datasets', filename)
self.annot_path_cache = osp.join(cfg.data_dir, 'cache', filename)
self.use_cache = getattr(cfg, 'use_cache', False)
self.img_shape = (720, 1280) # (h, w)
self.cam_param = {
'focal': (540, 540), # (fx, fy)
'princpt': (640, 360) # (cx, cy)
}
# check image shape
img_path = osp.join(self.img_dir, np.load(self.annot_path)['image_path'][0])
img_shape = cv2.imread(img_path).shape[:2]
assert self.img_shape == img_shape, 'image shape is incorrect: {} vs {}'.format(self.img_shape, img_shape)
# load data or cache
if self.use_cache and osp.isfile(self.annot_path_cache):
print(f'[{self.__class__.__name__}] loading cache from {self.annot_path_cache}')
self.datalist = self.load_cache(self.annot_path_cache)
else:
if self.use_cache:
print(f'[{self.__class__.__name__}] Cache not found, generating cache...')
self.datalist = self.load_data(
train_sample_interval=getattr(cfg, f'{self.__class__.__name__}_train_sample_interval', 1))
if self.use_cache:
self.save_cache(self.annot_path_cache, self.datalist)
data/Talkshow/Talkshow.py
-59
View File
@@ -1,59 +0,0 @@
import os
import os.path as osp
import numpy as np
import torch
import cv2
import json
import copy
from pycocotools.coco import COCO
from config import cfg
from utils.human_models import smpl_x
from utils.preprocessing import load_img, process_bbox, augmentation, process_db_coord, process_human_model_output, \
get_fitting_error_3D
from utils.transforms import world2cam, cam2pixel, rigid_align
from humandata import HumanDataset
# 'talkshow_smplx_chemistry_path.npz' zipfile.BadZipFile: File is not a zip file
# ['talkshow_smplx_conan.npz',
# 'talkshow_smplx_oliver_path.npz', 'talkshow_smplx_seth.npz']:
class Talkshow(HumanDataset):
def __init__(self, transform, data_split):
super(Talkshow, self).__init__(transform, data_split)
sample_rate = getattr(cfg, 'Talkshow_train_sample_interval', 1)
self.use_cache = getattr(cfg, 'use_cache', False)
self.annot_path_cache = osp.join(cfg.data_dir, 'cache', 'talkshow_smplx.npz')
self.img_shape = None # (h, w)
self.cam_param = {}
# load data or cache
if self.use_cache and osp.isfile(self.annot_path_cache):
print(f'[{self.__class__.__name__}] loading cache from {self.annot_path_cache}')
self.datalist = self.load_cache(self.annot_path_cache)
else:
if self.use_cache:
print(f'[{self.__class__.__name__}] Cache not found, generating cache...')
self.datalist = []
for pre_prc_file in ['talkshow_smplx_chemistry.npz', 'talkshow_smplx_conan.npz',
'talkshow_smplx_oliver.npz', 'talkshow_smplx_seth.npz']:
if self.data_split == 'train':
filename = getattr(cfg, 'filename', pre_prc_file)
else:
raise ValueError('Talkshow test set is not support')
self.img_dir = osp.join(cfg.data_dir, 'Talkshow')
self.annot_path = osp.join(cfg.data_dir, 'preprocessed_datasets', filename)
# check image shape
# img_path = osp.join(self.img_dir, np.load(self.annot_path)['image_path'][0])
# img_shape = cv2.imread(img_path).shape[:2]
# assert self.img_shape == img_shape, 'image shape is incorrect: {} vs {}'.format(self.img_shape, img_shape)
# load data
datalist_slice = self.load_data(sample_rate)
self.datalist.extend(datalist_slice)
if self.use_cache:
self.save_cache(self.annot_path_cache, self.datalist)
data/UBody/UBody.py
-1079
View File
File diff suppressed because it is too large Load Diff
data/UP3D/UP3D.py
-44
View File
@@ -1,44 +0,0 @@
import os
import os.path as osp
import numpy as np
import torch
import cv2
import json
import copy
from pycocotools.coco import COCO
from config import cfg
from utils.human_models import smpl_x
from utils.preprocessing import load_img, process_bbox, augmentation, process_db_coord, process_human_model_output, \
get_fitting_error_3D
from utils.transforms import world2cam, cam2pixel, rigid_align
from humandata import HumanDataset
class UP3D(HumanDataset):
def __init__(self, transform, data_split):
super(UP3D, self).__init__(transform, data_split)
if self.data_split == 'train':
filename = getattr(cfg, 'filename', 'up3d_trainval.npz')
else:
raise ValueError('UP3D test set is not support')
self.img_dir = osp.join(cfg.data_dir, 'UP3D')
self.annot_path = osp.join(cfg.data_dir, 'preprocessed_datasets', filename)
self.annot_path_cache = osp.join(cfg.data_dir, 'cache', filename)
self.use_cache = getattr(cfg, 'use_cache', False)
self.img_shape = None # (h, w)
self.cam_param = {}
# load data or cache
if self.use_cache and osp.isfile(self.annot_path_cache):
print(f'[{self.__class__.__name__}] loading cache from {self.annot_path_cache}')
self.datalist = self.load_cache(self.annot_path_cache)
else:
if self.use_cache:
print(f'[{self.__class__.__name__}] Cache not found, generating cache...')
self.datalist = self.load_data(
train_sample_interval=getattr(cfg, f'{self.__class__.__name__}_train_sample_interval', 1))
if self.use_cache:
self.save_cache(self.annot_path_cache, self.datalist)
data/dataset.py
-78
View File
@@ -1,78 +0,0 @@
import random
import numpy as np
from torch.utils.data.dataset import Dataset
from config import cfg
class MultipleDatasets(Dataset):
def __init__(self, dbs, make_same_len=True, total_len=None, verbose=False):
self.dbs = dbs
self.db_num = len(self.dbs)
self.max_db_data_num = max([len(db) for db in dbs])
self.db_len_cumsum = np.cumsum([len(db) for db in dbs])
self.make_same_len = make_same_len
if total_len == 'auto':
self.total_len = self.db_len_cumsum[-1]
self.auto_total_len = True
else:
self.total_len = total_len
self.auto_total_len = False
if total_len is not None:
self.per_db_len = self.total_len // self.db_num
if verbose:
print('datasets:', [len(self.dbs[i]) for i in range(self.db_num)])
print(f'Auto total length: {self.auto_total_len}, {self.total_len}')
def __len__(self):
# all dbs have the same length
if self.make_same_len:
if self.total_len is None:
# match the longest length
return self.max_db_data_num * self.db_num
else:
# each dataset has the same length and total len is fixed
return self.total_len
else:
# each db has different length, simply concat
return sum([len(db) for db in self.dbs])
def __getitem__(self, index):
if self.make_same_len:
if self.total_len is None:
# match the longest length
db_idx = index // self.max_db_data_num
data_idx = index % self.max_db_data_num
if data_idx >= len(self.dbs[db_idx]) * (self.max_db_data_num // len(self.dbs[db_idx])): # last batch: random sampling
data_idx = random.randint(0,len(self.dbs[db_idx])-1)
else: # before last batch: use modular
data_idx = data_idx % len(self.dbs[db_idx])
else:
db_idx = index // self.per_db_len
data_idx = index % self.per_db_len
if db_idx > (self.db_num - 1):
# last batch: randomly choose one dataset
db_idx = random.randint(0,self.db_num - 1)
if len(self.dbs[db_idx]) < self.per_db_len and \
data_idx >= len(self.dbs[db_idx]) * (self.per_db_len // len(self.dbs[db_idx])):
# last batch: random sampling in this dataset
data_idx = random.randint(0,len(self.dbs[db_idx]) - 1)
else:
# before last batch: use modular
data_idx = data_idx % len(self.dbs[db_idx])
else:
for i in range(self.db_num):
if index < self.db_len_cumsum[i]:
db_idx = i
break
if db_idx == 0:
data_idx = index
else:
data_idx = index - self.db_len_cumsum[db_idx-1]
return self.dbs[db_idx][data_idx]
data/humandata.py
-807
View File
@@ -1,807 +0,0 @@
import os
import os.path as osp
import numpy as np
import torch
import cv2
import json
import copy
from pycocotools.coco import COCO
from config import cfg
from utils.human_models import smpl_x
from utils.preprocessing import load_img, process_bbox, augmentation, process_db_coord, process_human_model_output, \
get_fitting_error_3D
from utils.transforms import world2cam, cam2pixel, rigid_align
import tqdm
import time
import random
KPS2D_KEYS = ['keypoints2d', 'keypoints2d_smplx', 'keypoints2d_smpl', 'keypoints2d_original']
KPS3D_KEYS = ['keypoints3d_cam', 'keypoints3d', 'keypoints3d_smplx','keypoints3d_smpl' ,'keypoints3d_original']
# keypoints3d_cam with root-align has higher priority, followed by old version key keypoints3d
# when there is keypoints3d_smplx, use this rather than keypoints3d_original
hands_meanr = np.array([ 0.11167871, -0.04289218, 0.41644183, 0.10881133, 0.06598568,
0.75622 , -0.09639297, 0.09091566, 0.18845929, -0.11809504,
-0.05094385, 0.5295845 , -0.14369841, -0.0552417 , 0.7048571 ,
-0.01918292, 0.09233685, 0.3379135 , -0.45703298, 0.19628395,
0.6254575 , -0.21465237, 0.06599829, 0.50689423, -0.36972436,
0.06034463, 0.07949023, -0.1418697 , 0.08585263, 0.63552827,
-0.3033416 , 0.05788098, 0.6313892 , -0.17612089, 0.13209307,
0.37335458, 0.8509643 , -0.27692273, 0.09154807, -0.49983943,
-0.02655647, -0.05288088, 0.5355592 , -0.04596104, 0.27735803]).reshape(15, -1)
hands_meanl = np.array([ 0.11167871, 0.04289218, -0.41644183, 0.10881133, -0.06598568,
-0.75622 , -0.09639297, -0.09091566, -0.18845929, -0.11809504,
0.05094385, -0.5295845 , -0.14369841, 0.0552417 , -0.7048571 ,
-0.01918292, -0.09233685, -0.3379135 , -0.45703298, -0.19628395,
-0.6254575 , -0.21465237, -0.06599829, -0.50689423, -0.36972436,
-0.06034463, -0.07949023, -0.1418697 , -0.08585263, -0.63552827,
-0.3033416 , -0.05788098, -0.6313892 , -0.17612089, -0.13209307,
-0.37335458, 0.8509643 , 0.27692273, -0.09154807, -0.49983943,
0.02655647, 0.05288088, 0.5355592 , 0.04596104, -0.27735803]).reshape(15, -1)
class Cache():
""" A custom implementation for SMPLer_X pipeline
Need to run tool/cache/fix_cache.py to fix paths
"""
def __init__(self, load_path=None):
if load_path is not None:
self.load(load_path)
def load(self, load_path):
self.load_path = load_path
self.cache = np.load(load_path, allow_pickle=True)
self.data_len = self.cache['data_len']
self.data_strategy = self.cache['data_strategy']
assert self.data_len == len(self.cache) - 2 # data_len, data_strategy
self.cache = None
@classmethod
def save(cls, save_path, data_list, data_strategy):
assert save_path is not None, 'save_path is None'
data_len = len(data_list)
cache = {}
for i, data in enumerate(data_list):
cache[str(i)] = data
assert len(cache) == data_len
# update meta
cache.update({
'data_len': data_len,
'data_strategy': data_strategy})
np.savez_compressed(save_path, **cache)
print(f'Cache saved to {save_path}.')
# def shuffle(self):
# random.shuffle(self.mapping)
def __len__(self):
return self.data_len
def __getitem__(self, idx):
if self.cache is None:
self.cache = np.load(self.load_path, allow_pickle=True)
# mapped_idx = self.mapping[idx]
# cache_data = self.cache[str(mapped_idx)]
cache_data = self.cache[str(idx)]
data = cache_data.item()
return data
class HumanDataset(torch.utils.data.Dataset):
# same mapping for 144->137 and 190->137
SMPLX_137_MAPPING = [
0, 1, 2, 4, 5, 7, 8, 12, 16, 17, 18, 19, 20, 21, 60, 61, 62, 63, 64, 65, 59, 58, 57, 56, 55, 37, 38, 39, 66,
25, 26, 27, 67, 28, 29, 30, 68, 34, 35, 36, 69, 31, 32, 33, 70, 52, 53, 54, 71, 40, 41, 42, 72, 43, 44, 45,
73, 49, 50, 51, 74, 46, 47, 48, 75, 22, 15, 56, 57, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89,
90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113,
114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135,
136, 137, 138, 139, 140, 141, 142, 143]
def __init__(self, transform, data_split):
self.transform = transform
self.data_split = data_split
# dataset information, to be filled by child class
self.img_dir = None
self.annot_path = None
self.annot_path_cache = None
self.use_cache = False
self.save_idx = 0
self.img_shape = None # (h, w)
self.cam_param = None # {'focal_length': (fx, fy), 'princpt': (cx, cy)}
self.use_betas_neutral = False
self.joint_set = {
'joint_num': smpl_x.joint_num,
'joints_name': smpl_x.joints_name,
'flip_pairs': smpl_x.flip_pairs}
self.joint_set['root_joint_idx'] = self.joint_set['joints_name'].index('Pelvis')
def load_cache(self, annot_path_cache):
datalist = Cache(annot_path_cache)
assert datalist.data_strategy == getattr(cfg, 'data_strategy', None), \
f'Cache data strategy {datalist.data_strategy} does not match current data strategy ' \
f'{getattr(cfg, "data_strategy", None)}'
return datalist
def save_cache(self, annot_path_cache, datalist):
print(f'[{self.__class__.__name__}] Caching datalist to {self.annot_path_cache}...')
Cache.save(
annot_path_cache,
datalist,
data_strategy=getattr(cfg, 'data_strategy', None)
)
def load_data(self, train_sample_interval=1, test_sample_interval=1):
content = np.load(self.annot_path, allow_pickle=True)
num_examples = len(content['image_path'])
if 'meta' in content:
meta = content['meta'].item()
print('meta keys:', meta.keys())
else:
meta = None
print('No meta info provided! Please give height and width manually')
print(f'Start loading humandata {self.annot_path} into memory...\nDataset includes: {content.files}'); tic = time.time()
image_path = content['image_path']
if meta is not None and 'height' in meta:
height = np.array(meta['height'])
width = np.array(meta['width'])
image_shape = np.stack([height, width], axis=-1)
else:
image_shape = None
bbox_xywh = content['bbox_xywh']
if 'smplx' in content:
smplx = content['smplx'].item()
as_smplx = 'smplx'
elif 'smpl' in content:
smplx = content['smpl'].item()
as_smplx = 'smpl'
elif 'smplh' in content:
smplx = content['smplh'].item()
as_smplx = 'smplh'
# TODO: temp solution, should be more general. But SHAPY is very special
elif self.__class__.__name__ == 'SHAPY':
smplx = {}
else:
raise KeyError('No SMPL for SMPLX available, please check keys:\n'
f'{content.files}')
print('Smplx param', smplx.keys())
if 'lhand_bbox_xywh' in content and 'rhand_bbox_xywh' in content:
lhand_bbox_xywh = content['lhand_bbox_xywh']
rhand_bbox_xywh = content['rhand_bbox_xywh']
else:
lhand_bbox_xywh = np.zeros_like(bbox_xywh)
rhand_bbox_xywh = np.zeros_like(bbox_xywh)
if 'face_bbox_xywh' in content:
face_bbox_xywh = content['face_bbox_xywh']
else:
face_bbox_xywh = np.zeros_like(bbox_xywh)
decompressed = False
if content['__keypoints_compressed__']:
decompressed_kps = self.decompress_keypoints(content)
decompressed = True
keypoints3d = None
valid_kps3d = False
keypoints3d_mask = None
valid_kps3d_mask = False
for kps3d_key in KPS3D_KEYS:
if kps3d_key in content:
keypoints3d = decompressed_kps[kps3d_key][:, self.SMPLX_137_MAPPING, :3] if decompressed \
else content[kps3d_key][:, self.SMPLX_137_MAPPING, :3]
valid_kps3d = True
if f'{kps3d_key}_mask' in content:
keypoints3d_mask = content[f'{kps3d_key}_mask'][self.SMPLX_137_MAPPING]
valid_kps3d_mask = True
elif 'keypoints3d_mask' in content:
keypoints3d_mask = content['keypoints3d_mask'][self.SMPLX_137_MAPPING]
valid_kps3d_mask = True
break
for kps2d_key in KPS2D_KEYS:
if kps2d_key in content:
keypoints2d = decompressed_kps[kps2d_key][:, self.SMPLX_137_MAPPING, :2] if decompressed \
else content[kps2d_key][:, self.SMPLX_137_MAPPING, :2]
if f'{kps2d_key}_mask' in content:
keypoints2d_mask = content[f'{kps2d_key}_mask'][self.SMPLX_137_MAPPING]
elif 'keypoints2d_mask' in content:
keypoints2d_mask = content['keypoints2d_mask'][self.SMPLX_137_MAPPING]
break
mask = keypoints3d_mask if valid_kps3d_mask \
else keypoints2d_mask
print('Done. Time: {:.2f}s'.format(time.time() - tic))
datalist = []
for i in tqdm.tqdm(range(int(num_examples))):
if self.data_split == 'train' and i % train_sample_interval != 0:
continue
if self.data_split == 'test' and i % test_sample_interval != 0:
continue
img_path = osp.join(self.img_dir, image_path[i])
img_shape = image_shape[i] if image_shape is not None else self.img_shape
bbox = bbox_xywh[i][:4]
if hasattr(cfg, 'bbox_ratio'):
bbox_ratio = cfg.bbox_ratio * 0.833 # preprocess body bbox is giving 1.2 box padding
else:
bbox_ratio = 1.25
bbox = process_bbox(bbox, img_width=img_shape[1], img_height=img_shape[0], ratio=bbox_ratio)
if bbox is None: continue
# hand/face bbox
lhand_bbox = lhand_bbox_xywh[i]
rhand_bbox = rhand_bbox_xywh[i]
face_bbox = face_bbox_xywh[i]
if lhand_bbox[-1] > 0: # conf > 0
lhand_bbox = lhand_bbox[:4]
if hasattr(cfg, 'bbox_ratio'):
lhand_bbox = process_bbox(lhand_bbox, img_width=img_shape[1], img_height=img_shape[0], ratio=cfg.bbox_ratio)
if lhand_bbox is not None:
lhand_bbox[2:] += lhand_bbox[:2] # xywh -> xyxy
else:
lhand_bbox = None
if rhand_bbox[-1] > 0:
rhand_bbox = rhand_bbox[:4]
if hasattr(cfg, 'bbox_ratio'):
rhand_bbox = process_bbox(rhand_bbox, img_width=img_shape[1], img_height=img_shape[0], ratio=cfg.bbox_ratio)
if rhand_bbox is not None:
rhand_bbox[2:] += rhand_bbox[:2] # xywh -> xyxy
else:
rhand_bbox = None
if face_bbox[-1] > 0:
face_bbox = face_bbox[:4]
if hasattr(cfg, 'bbox_ratio'):
face_bbox = process_bbox(face_bbox, img_width=img_shape[1], img_height=img_shape[0], ratio=cfg.bbox_ratio)
if face_bbox is not None:
face_bbox[2:] += face_bbox[:2] # xywh -> xyxy
else:
face_bbox = None
joint_img = keypoints2d[i]
joint_valid = mask.reshape(-1, 1)
# num_joints = joint_cam.shape[0]
# joint_valid = np.ones((num_joints, 1))
if valid_kps3d:
joint_cam = keypoints3d[i]
else:
joint_cam = None
smplx_param = {k: v[i] for k, v in smplx.items()}
smplx_param['root_pose'] = smplx_param.pop('global_orient', None)
smplx_param['shape'] = smplx_param.pop('betas', None)
smplx_param['trans'] = smplx_param.pop('transl', np.zeros(3))
smplx_param['lhand_pose'] = smplx_param.pop('left_hand_pose', None)
smplx_param['rhand_pose'] = smplx_param.pop('right_hand_pose', None)
smplx_param['expr'] = smplx_param.pop('expression', None)
# TODO do not fix betas, give up shape supervision
if 'betas_neutral' in smplx_param:
smplx_param['shape'] = smplx_param.pop('betas_neutral')
# TODO fix shape of poses
if self.__class__.__name__ == 'Talkshow':
smplx_param['body_pose'] = smplx_param['body_pose'].reshape(21, 3)
smplx_param['lhand_pose'] = smplx_param['lhand_pose'].reshape(15, 3)
smplx_param['rhand_pose'] = smplx_param['lhand_pose'].reshape(15, 3)
smplx_param['expr'] = smplx_param['expr'][:10]
if self.__class__.__name__ == 'BEDLAM':
smplx_param['shape'] = smplx_param['shape'][:10]
# manually set flat_hand_mean = True
smplx_param['lhand_pose'] -= hands_meanl
smplx_param['rhand_pose'] -= hands_meanr
if as_smplx == 'smpl':
smplx_param['shape'] = np.zeros(10, dtype=np.float32) # drop smpl betas for smplx
smplx_param['body_pose'] = smplx_param['body_pose'][:21, :] # use smpl body_pose on smplx
if as_smplx == 'smplh':
smplx_param['shape'] = np.zeros(10, dtype=np.float32) # drop smpl betas for smplx
if smplx_param['lhand_pose'] is None:
smplx_param['lhand_valid'] = False
else:
smplx_param['lhand_valid'] = True
if smplx_param['rhand_pose'] is None:
smplx_param['rhand_valid'] = False
else:
smplx_param['rhand_valid'] = True
if smplx_param['expr'] is None:
smplx_param['face_valid'] = False
else:
smplx_param['face_valid'] = True
if joint_cam is not None and np.any(np.isnan(joint_cam)):
continue
datalist.append({
'img_path': img_path,
'img_shape': img_shape,
'bbox': bbox,
'lhand_bbox': lhand_bbox,
'rhand_bbox': rhand_bbox,
'face_bbox': face_bbox,
'joint_img': joint_img,
'joint_cam': joint_cam,
'joint_valid': joint_valid,
'smplx_param': smplx_param,
'smplx': smplx})
# save memory
del content, image_path, bbox_xywh, lhand_bbox_xywh, rhand_bbox_xywh, face_bbox_xywh, keypoints3d, keypoints2d
if self.data_split == 'train':
print(f'[{self.__class__.__name__} train] original size:', int(num_examples),
'. Sample interval:', train_sample_interval,
'. Sampled size:', len(datalist))
if (getattr(cfg, 'data_strategy', None) == 'balance' and self.data_split == 'train') or \
getattr(cfg, 'eval_on_train', False):
print(f'[{self.__class__.__name__}] Using [balance] strategy with datalist shuffled...')
random.seed(2023)
random.shuffle(datalist)
if getattr(cfg, 'eval_on_train', False):
return datalist[:10000]
return datalist
def __len__(self):
return len(self.datalist)
def __getitem__(self, idx):
try:
data = copy.deepcopy(self.datalist[idx])
except Exception as e:
print(f'[{self.__class__.__name__}] Error loading data {idx}')
print(e)
exit(0)
img_path, img_shape, bbox = data['img_path'], data['img_shape'], data['bbox']
# img
img = load_img(img_path)
img, img2bb_trans, bb2img_trans, rot, do_flip = augmentation(img, bbox, self.data_split)
img = self.transform(img.astype(np.float32)) / 255.
if self.data_split == 'train':
# h36m gt
joint_cam = data['joint_cam']
if joint_cam is not None:
dummy_cord = False
joint_cam = joint_cam - joint_cam[self.joint_set['root_joint_idx'], None, :] # root-relative
else:
# dummy cord as joint_cam
dummy_cord = True
joint_cam = np.zeros((self.joint_set['joint_num'], 3), dtype=np.float32)
joint_img = data['joint_img']
joint_img = np.concatenate((joint_img[:, :2], joint_cam[:, 2:]), 1) # x, y, depth
if not dummy_cord:
joint_img[:, 2] = (joint_img[:, 2] / (cfg.body_3d_size / 2) + 1) / 2. * cfg.output_hm_shape[0] # discretize depth
joint_img_aug, joint_cam_wo_ra, joint_cam_ra, joint_valid, joint_trunc = process_db_coord(
joint_img, joint_cam, data['joint_valid'], do_flip, img_shape,
self.joint_set['flip_pairs'], img2bb_trans, rot, self.joint_set['joints_name'], smpl_x.joints_name)
# smplx coordinates and parameters
smplx_param = data['smplx_param']
smplx_joint_img, smplx_joint_cam, smplx_joint_trunc, smplx_pose, smplx_shape, smplx_expr, \
smplx_pose_valid, smplx_joint_valid, smplx_expr_valid, smplx_mesh_cam_orig = process_human_model_output(
smplx_param, self.cam_param, do_flip, img_shape, img2bb_trans, rot, 'smplx',
joint_img=None if self.cam_param else joint_img, # if cam not provided, we take joint_img as smplx joint 2d, which is commonly the case for our processed humandata
)
# TODO temp fix keypoints3d for renbody
if 'RenBody' in self.__class__.__name__:
joint_cam_ra = smplx_joint_cam.copy()
joint_cam_wo_ra = smplx_joint_cam.copy()
joint_cam_wo_ra[smpl_x.joint_part['lhand'], :] = joint_cam_wo_ra[smpl_x.joint_part['lhand'], :] \
+ joint_cam_wo_ra[smpl_x.lwrist_idx, None, :] # left hand root-relative
joint_cam_wo_ra[smpl_x.joint_part['rhand'], :] = joint_cam_wo_ra[smpl_x.joint_part['rhand'], :] \
+ joint_cam_wo_ra[smpl_x.rwrist_idx, None, :] # right hand root-relative
joint_cam_wo_ra[smpl_x.joint_part['face'], :] = joint_cam_wo_ra[smpl_x.joint_part['face'], :] \
+ joint_cam_wo_ra[smpl_x.neck_idx, None,: ] # face root-relative
# change smplx_shape if use_betas_neutral
# processing follows that in process_human_model_output
if self.use_betas_neutral:
smplx_shape = smplx_param['betas_neutral'].reshape(1, -1)
smplx_shape[(np.abs(smplx_shape) > 3).any(axis=1)] = 0.
smplx_shape = smplx_shape.reshape(-1)
# SMPLX pose parameter validity
# for name in ('L_Ankle', 'R_Ankle', 'L_Wrist', 'R_Wrist'):
# smplx_pose_valid[smpl_x.orig_joints_name.index(name)] = 0
smplx_pose_valid = np.tile(smplx_pose_valid[:, None], (1, 3)).reshape(-1)
# SMPLX joint coordinate validity
# for name in ('L_Big_toe', 'L_Small_toe', 'L_Heel', 'R_Big_toe', 'R_Small_toe', 'R_Heel'):
# smplx_joint_valid[smpl_x.joints_name.index(name)] = 0
smplx_joint_valid = smplx_joint_valid[:, None]
smplx_joint_trunc = smplx_joint_valid * smplx_joint_trunc
if not (smplx_shape == 0).all():
smplx_shape_valid = True
else:
smplx_shape_valid = False
# hand and face bbox transform
lhand_bbox, lhand_bbox_valid = self.process_hand_face_bbox(data['lhand_bbox'], do_flip, img_shape, img2bb_trans)
rhand_bbox, rhand_bbox_valid = self.process_hand_face_bbox(data['rhand_bbox'], do_flip, img_shape, img2bb_trans)
face_bbox, face_bbox_valid = self.process_hand_face_bbox(data['face_bbox'], do_flip, img_shape, img2bb_trans)
if do_flip:
lhand_bbox, rhand_bbox = rhand_bbox, lhand_bbox
lhand_bbox_valid, rhand_bbox_valid = rhand_bbox_valid, lhand_bbox_valid
lhand_bbox_center = (lhand_bbox[0] + lhand_bbox[1]) / 2.
rhand_bbox_center = (rhand_bbox[0] + rhand_bbox[1]) / 2.
face_bbox_center = (face_bbox[0] + face_bbox[1]) / 2.
lhand_bbox_size = lhand_bbox[1] - lhand_bbox[0]
rhand_bbox_size = rhand_bbox[1] - rhand_bbox[0]
face_bbox_size = face_bbox[1] - face_bbox[0]
inputs = {'img': img}
targets = {'joint_img': joint_img_aug, # keypoints2d
'smplx_joint_img': joint_img_aug, #smplx_joint_img, # projected smplx if valid cam_param, else same as keypoints2d
'joint_cam': joint_cam_wo_ra, # joint_cam actually not used in any loss, # raw kps3d probably without ra
'smplx_joint_cam': smplx_joint_cam if dummy_cord else joint_cam_ra, # kps3d with body, face, hand ra
'smplx_pose': smplx_pose,
'smplx_shape': smplx_shape,
'smplx_expr': smplx_expr,
'lhand_bbox_center': lhand_bbox_center, 'lhand_bbox_size': lhand_bbox_size,
'rhand_bbox_center': rhand_bbox_center, 'rhand_bbox_size': rhand_bbox_size,
'face_bbox_center': face_bbox_center, 'face_bbox_size': face_bbox_size}
meta_info = {'joint_valid': joint_valid,
'joint_trunc': joint_trunc,
'smplx_joint_valid': smplx_joint_valid if dummy_cord else joint_valid,
'smplx_joint_trunc': smplx_joint_trunc if dummy_cord else joint_trunc,
'smplx_pose_valid': smplx_pose_valid,
'smplx_shape_valid': float(smplx_shape_valid),
'smplx_expr_valid': float(smplx_expr_valid),
'is_3D': float(False) if dummy_cord else float(True),
'lhand_bbox_valid': lhand_bbox_valid,
'rhand_bbox_valid': rhand_bbox_valid, 'face_bbox_valid': face_bbox_valid}
if self.__class__.__name__ == 'SHAPY':
meta_info['img_path'] = img_path
return inputs, targets, meta_info
# TODO: temp solution, should be more general. But SHAPY is very special
elif self.__class__.__name__ == 'SHAPY':
inputs = {'img': img}
if cfg.shapy_eval_split == 'val':
targets = {'smplx_shape': smplx_shape}
else:
targets = {}
meta_info = {'img_path': img_path}
return inputs, targets, meta_info
else:
joint_cam = data['joint_cam']
if joint_cam is not None:
dummy_cord = False
joint_cam = joint_cam - joint_cam[self.joint_set['root_joint_idx'], None, :] # root-relative
else:
# dummy cord as joint_cam
dummy_cord = True
joint_cam = np.zeros((self.joint_set['joint_num'], 3), dtype=np.float32)
joint_img = data['joint_img']
joint_img = np.concatenate((joint_img[:, :2], joint_cam[:, 2:]), 1) # x, y, depth
if not dummy_cord:
joint_img[:, 2] = (joint_img[:, 2] / (cfg.body_3d_size / 2) + 1) / 2. * cfg.output_hm_shape[0] # discretize depth
joint_img, joint_cam, joint_cam_ra, joint_valid, joint_trunc = process_db_coord(
joint_img, joint_cam, data['joint_valid'], do_flip, img_shape,
self.joint_set['flip_pairs'], img2bb_trans, rot, self.joint_set['joints_name'], smpl_x.joints_name)
# smplx coordinates and parameters
smplx_param = data['smplx_param']
smplx_cam_trans = np.array(smplx_param['trans']) if 'trans' in smplx_param else None
smplx_joint_img, smplx_joint_cam, smplx_joint_trunc, smplx_pose, smplx_shape, smplx_expr, \
smplx_pose_valid, smplx_joint_valid, smplx_expr_valid, smplx_mesh_cam_orig = process_human_model_output(
smplx_param, self.cam_param, do_flip, img_shape, img2bb_trans, rot, 'smplx',
joint_img=None if self.cam_param else joint_img
) # if cam not provided, we take joint_img as smplx joint 2d, which is commonly the case for our processed humandata
inputs = {'img': img}
targets = {'smplx_pose': smplx_pose,
'smplx_shape': smplx_shape,
'smplx_expr': smplx_expr,
'smplx_cam_trans' : smplx_cam_trans,
}
meta_info = {'img_path': img_path,
'bb2img_trans': bb2img_trans,
'gt_smplx_transl':smplx_cam_trans}
return inputs, targets, meta_info
def process_hand_face_bbox(self, bbox, do_flip, img_shape, img2bb_trans):
if bbox is None:
bbox = np.array([0, 0, 1, 1], dtype=np.float32).reshape(2, 2) # dummy value
bbox_valid = float(False) # dummy value
else:
# reshape to top-left (x,y) and bottom-right (x,y)
bbox = bbox.reshape(2, 2)
# flip augmentation
if do_flip:
bbox[:, 0] = img_shape[1] - bbox[:, 0] - 1
bbox[0, 0], bbox[1, 0] = bbox[1, 0].copy(), bbox[0, 0].copy() # xmin <-> xmax swap
# make four points of the bbox
bbox = bbox.reshape(4).tolist()
xmin, ymin, xmax, ymax = bbox
bbox = np.array([[xmin, ymin], [xmax, ymin], [xmax, ymax], [xmin, ymax]], dtype=np.float32).reshape(4, 2)
# affine transformation (crop, rotation, scale)
bbox_xy1 = np.concatenate((bbox, np.ones_like(bbox[:, :1])), 1)
bbox = np.dot(img2bb_trans, bbox_xy1.transpose(1, 0)).transpose(1, 0)[:, :2]
bbox[:, 0] = bbox[:, 0] / cfg.input_img_shape[1] * cfg.output_hm_shape[2]
bbox[:, 1] = bbox[:, 1] / cfg.input_img_shape[0] * cfg.output_hm_shape[1]
# make box a rectangle without rotation
xmin = np.min(bbox[:, 0])
xmax = np.max(bbox[:, 0])
ymin = np.min(bbox[:, 1])
ymax = np.max(bbox[:, 1])
bbox = np.array([xmin, ymin, xmax, ymax], dtype=np.float32)
bbox_valid = float(True)
bbox = bbox.reshape(2, 2)
return bbox, bbox_valid
def evaluate(self, outs, cur_sample_idx=None):
sample_num = len(outs)
eval_result = {'pa_mpvpe_all': [], 'pa_mpvpe_l_hand': [], 'pa_mpvpe_r_hand': [], 'pa_mpvpe_hand': [], 'pa_mpvpe_face': [],
'mpvpe_all': [], 'mpvpe_l_hand': [], 'mpvpe_r_hand': [], 'mpvpe_hand': [], 'mpvpe_face': [],
'pa_mpjpe_body': [], 'pa_mpjpe_l_hand': [], 'pa_mpjpe_r_hand': [], 'pa_mpjpe_hand': []}
if getattr(cfg, 'vis', False):
import csv
csv_file = f'{cfg.vis_dir}/{cfg.testset}_smplx_error.csv'
file = open(csv_file, 'a', newline='')
writer = csv.writer(file)
for n in range(sample_num):
out = outs[n]
mesh_gt = out['smplx_mesh_cam_pseudo_gt']
mesh_out = out['smplx_mesh_cam']
# MPVPE from all vertices
mesh_out_align = mesh_out - np.dot(smpl_x.J_regressor, mesh_out)[smpl_x.J_regressor_idx['pelvis'], None,
:] + np.dot(smpl_x.J_regressor, mesh_gt)[smpl_x.J_regressor_idx['pelvis'], None,
:]
mpvpe_all = np.sqrt(np.sum((mesh_out_align - mesh_gt) ** 2, 1)).mean() * 1000
eval_result['mpvpe_all'].append(mpvpe_all)
mesh_out_align = rigid_align(mesh_out, mesh_gt)
pa_mpvpe_all = np.sqrt(np.sum((mesh_out_align - mesh_gt) ** 2, 1)).mean() * 1000
eval_result['pa_mpvpe_all'].append(pa_mpvpe_all)
# MPVPE from hand vertices
mesh_gt_lhand = mesh_gt[smpl_x.hand_vertex_idx['left_hand'], :]
mesh_out_lhand = mesh_out[smpl_x.hand_vertex_idx['left_hand'], :]
mesh_gt_rhand = mesh_gt[smpl_x.hand_vertex_idx['right_hand'], :]
mesh_out_rhand = mesh_out[smpl_x.hand_vertex_idx['right_hand'], :]
mesh_out_lhand_align = mesh_out_lhand - np.dot(smpl_x.J_regressor, mesh_out)[
smpl_x.J_regressor_idx['lwrist'], None, :] + np.dot(
smpl_x.J_regressor, mesh_gt)[smpl_x.J_regressor_idx['lwrist'], None, :]
mesh_out_rhand_align = mesh_out_rhand - np.dot(smpl_x.J_regressor, mesh_out)[
smpl_x.J_regressor_idx['rwrist'], None, :] + np.dot(
smpl_x.J_regressor, mesh_gt)[smpl_x.J_regressor_idx['rwrist'], None, :]
eval_result['mpvpe_l_hand'].append(np.sqrt(
np.sum((mesh_out_lhand_align - mesh_gt_lhand) ** 2, 1)).mean() * 1000)
eval_result['mpvpe_r_hand'].append(np.sqrt(
np.sum((mesh_out_rhand_align - mesh_gt_rhand) ** 2, 1)).mean() * 1000)
eval_result['mpvpe_hand'].append((np.sqrt(
np.sum((mesh_out_lhand_align - mesh_gt_lhand) ** 2, 1)).mean() * 1000 + np.sqrt(
np.sum((mesh_out_rhand_align - mesh_gt_rhand) ** 2, 1)).mean() * 1000) / 2.)
mesh_out_lhand_align = rigid_align(mesh_out_lhand, mesh_gt_lhand)
mesh_out_rhand_align = rigid_align(mesh_out_rhand, mesh_gt_rhand)
eval_result['pa_mpvpe_l_hand'].append(np.sqrt(
np.sum((mesh_out_lhand_align - mesh_gt_lhand) ** 2, 1)).mean() * 1000)
eval_result['pa_mpvpe_r_hand'].append(np.sqrt(
np.sum((mesh_out_rhand_align - mesh_gt_rhand) ** 2, 1)).mean() * 1000)
eval_result['pa_mpvpe_hand'].append((np.sqrt(
np.sum((mesh_out_lhand_align - mesh_gt_lhand) ** 2, 1)).mean() * 1000 + np.sqrt(
np.sum((mesh_out_rhand_align - mesh_gt_rhand) ** 2, 1)).mean() * 1000) / 2.)
# MPVPE from face vertices
mesh_gt_face = mesh_gt[smpl_x.face_vertex_idx, :]
mesh_out_face = mesh_out[smpl_x.face_vertex_idx, :]
mesh_out_face_align = mesh_out_face - np.dot(smpl_x.J_regressor, mesh_out)[smpl_x.J_regressor_idx['neck'],
None, :] + np.dot(smpl_x.J_regressor, mesh_gt)[
smpl_x.J_regressor_idx['neck'], None, :]
eval_result['mpvpe_face'].append(
np.sqrt(np.sum((mesh_out_face_align - mesh_gt_face) ** 2, 1)).mean() * 1000)
mesh_out_face_align = rigid_align(mesh_out_face, mesh_gt_face)
eval_result['pa_mpvpe_face'].append(
np.sqrt(np.sum((mesh_out_face_align - mesh_gt_face) ** 2, 1)).mean() * 1000)
# MPJPE from body joints
joint_gt_body = np.dot(smpl_x.j14_regressor, mesh_gt)
joint_out_body = np.dot(smpl_x.j14_regressor, mesh_out)
joint_out_body_align = rigid_align(joint_out_body, joint_gt_body)
eval_result['pa_mpjpe_body'].append(
np.sqrt(np.sum((joint_out_body_align - joint_gt_body) ** 2, 1)).mean() * 1000)
# MPJPE from hand joints
joint_gt_lhand = np.dot(smpl_x.orig_hand_regressor['left'], mesh_gt)
joint_out_lhand = np.dot(smpl_x.orig_hand_regressor['left'], mesh_out)
joint_out_lhand_align = rigid_align(joint_out_lhand, joint_gt_lhand)
joint_gt_rhand = np.dot(smpl_x.orig_hand_regressor['right'], mesh_gt)
joint_out_rhand = np.dot(smpl_x.orig_hand_regressor['right'], mesh_out)
joint_out_rhand_align = rigid_align(joint_out_rhand, joint_gt_rhand)
eval_result['pa_mpjpe_l_hand'].append(np.sqrt(
np.sum((joint_out_lhand_align - joint_gt_lhand) ** 2, 1)).mean() * 1000)
eval_result['pa_mpjpe_r_hand'].append(np.sqrt(
np.sum((joint_out_rhand_align - joint_gt_rhand) ** 2, 1)).mean() * 1000)
eval_result['pa_mpjpe_hand'].append((np.sqrt(
np.sum((joint_out_lhand_align - joint_gt_lhand) ** 2, 1)).mean() * 1000 + np.sqrt(
np.sum((joint_out_rhand_align - joint_gt_rhand) ** 2, 1)).mean() * 1000) / 2.)
if getattr(cfg, 'vis', False):
img_path = out['img_path']
rel_img_path = img_path.split('..')[-1]
smplx_pred = {}
smplx_pred['global_orient'] = out['smplx_root_pose'].reshape(-1,3)
smplx_pred['body_pose'] = out['smplx_body_pose'].reshape(-1,3)
smplx_pred['left_hand_pose'] = out['smplx_lhand_pose'].reshape(-1,3)
smplx_pred['right_hand_pose'] = out['smplx_rhand_pose'].reshape(-1,3)
smplx_pred['jaw_pose'] = out['smplx_jaw_pose'].reshape(-1,3)
smplx_pred['leye_pose'] = np.zeros((1, 3))
smplx_pred['reye_pose'] = np.zeros((1, 3))
smplx_pred['betas'] = out['smplx_shape'].reshape(-1,10)
smplx_pred['expression'] = out['smplx_expr'].reshape(-1,10)
smplx_pred['transl'] = out['gt_smplx_transl'].reshape(-1,3)
smplx_pred['img_path'] = rel_img_path
npz_path = os.path.join(cfg.vis_dir, f'{self.save_idx}.npz')
np.savez(npz_path, **smplx_pred)
# save img path and error
new_line = [self.save_idx, rel_img_path, mpvpe_all, pa_mpvpe_all]
# Append the new line to the CSV file
writer.writerow(new_line)
self.save_idx += 1
if getattr(cfg, 'vis', False):
file.close()
return eval_result
def print_eval_result(self, eval_result):
print(f'======{cfg.testset}======')
print(f'{cfg.vis_dir}')
print('PA MPVPE (All): %.2f mm' % np.mean(eval_result['pa_mpvpe_all']))
print('PA MPVPE (L-Hands): %.2f mm' % np.mean(eval_result['pa_mpvpe_l_hand']))
print('PA MPVPE (R-Hands): %.2f mm' % np.mean(eval_result['pa_mpvpe_r_hand']))
print('PA MPVPE (Hands): %.2f mm' % np.mean(eval_result['pa_mpvpe_hand']))
print('PA MPVPE (Face): %.2f mm' % np.mean(eval_result['pa_mpvpe_face']))
print()
print('MPVPE (All): %.2f mm' % np.mean(eval_result['mpvpe_all']))
print('MPVPE (L-Hands): %.2f mm' % np.mean(eval_result['mpvpe_l_hand']))
print('MPVPE (R-Hands): %.2f mm' % np.mean(eval_result['mpvpe_r_hand']))
print('MPVPE (Hands): %.2f mm' % np.mean(eval_result['mpvpe_hand']))
print('MPVPE (Face): %.2f mm' % np.mean(eval_result['mpvpe_face']))
print()
print('PA MPJPE (Body): %.2f mm' % np.mean(eval_result['pa_mpjpe_body']))
print('PA MPJPE (L-Hands): %.2f mm' % np.mean(eval_result['pa_mpjpe_l_hand']))
print('PA MPJPE (R-Hands): %.2f mm' % np.mean(eval_result['pa_mpjpe_r_hand']))
print('PA MPJPE (Hands): %.2f mm' % np.mean(eval_result['pa_mpjpe_hand']))
print()
print(f"{np.mean(eval_result['pa_mpvpe_all'])},{np.mean(eval_result['pa_mpvpe_l_hand'])},{np.mean(eval_result['pa_mpvpe_r_hand'])},{np.mean(eval_result['pa_mpvpe_hand'])},{np.mean(eval_result['pa_mpvpe_face'])},"
f"{np.mean(eval_result['mpvpe_all'])},{np.mean(eval_result['mpvpe_l_hand'])},{np.mean(eval_result['mpvpe_r_hand'])},{np.mean(eval_result['mpvpe_hand'])},{np.mean(eval_result['mpvpe_face'])},"
f"{np.mean(eval_result['pa_mpjpe_body'])},{np.mean(eval_result['pa_mpjpe_l_hand'])},{np.mean(eval_result['pa_mpjpe_r_hand'])},{np.mean(eval_result['pa_mpjpe_hand'])}")
print()
f = open(os.path.join(cfg.result_dir, 'result.txt'), 'w')
f.write(f'{cfg.testset} dataset \n')
f.write('PA MPVPE (All): %.2f mm\n' % np.mean(eval_result['pa_mpvpe_all']))
f.write('PA MPVPE (L-Hands): %.2f mm' % np.mean(eval_result['pa_mpvpe_l_hand']))
f.write('PA MPVPE (R-Hands): %.2f mm' % np.mean(eval_result['pa_mpvpe_r_hand']))
f.write('PA MPVPE (Hands): %.2f mm\n' % np.mean(eval_result['pa_mpvpe_hand']))
f.write('PA MPVPE (Face): %.2f mm\n' % np.mean(eval_result['pa_mpvpe_face']))
f.write('MPVPE (All): %.2f mm\n' % np.mean(eval_result['mpvpe_all']))
f.write('MPVPE (L-Hands): %.2f mm' % np.mean(eval_result['mpvpe_l_hand']))
f.write('MPVPE (R-Hands): %.2f mm' % np.mean(eval_result['mpvpe_r_hand']))
f.write('MPVPE (Hands): %.2f mm' % np.mean(eval_result['mpvpe_hand']))
f.write('MPVPE (Face): %.2f mm\n' % np.mean(eval_result['mpvpe_face']))
f.write('PA MPJPE (Body): %.2f mm\n' % np.mean(eval_result['pa_mpjpe_body']))
f.write('PA MPJPE (L-Hands): %.2f mm' % np.mean(eval_result['pa_mpjpe_l_hand']))
f.write('PA MPJPE (R-Hands): %.2f mm' % np.mean(eval_result['pa_mpjpe_r_hand']))
f.write('PA MPJPE (Hands): %.2f mm\n' % np.mean(eval_result['pa_mpjpe_hand']))
f.write(f"{np.mean(eval_result['pa_mpvpe_all'])},{np.mean(eval_result['pa_mpvpe_l_hand'])},{np.mean(eval_result['pa_mpvpe_r_hand'])},{np.mean(eval_result['pa_mpvpe_hand'])},{np.mean(eval_result['pa_mpvpe_face'])},"
f"{np.mean(eval_result['mpvpe_all'])},{np.mean(eval_result['mpvpe_l_hand'])},{np.mean(eval_result['mpvpe_r_hand'])},{np.mean(eval_result['mpvpe_hand'])},{np.mean(eval_result['mpvpe_face'])},"
f"{np.mean(eval_result['pa_mpjpe_body'])},{np.mean(eval_result['pa_mpjpe_l_hand'])},{np.mean(eval_result['pa_mpjpe_r_hand'])},{np.mean(eval_result['pa_mpjpe_hand'])}")
if getattr(cfg, 'eval_on_train', False):
import csv
csv_file = f'{cfg.root_dir}/output/{cfg.testset}_eval_on_train.csv'
exp_id = cfg.exp_name.split('_')[1]
new_line = [exp_id,np.mean(eval_result['pa_mpvpe_all']),np.mean(eval_result['pa_mpvpe_l_hand']),np.mean(eval_result['pa_mpvpe_r_hand']),np.mean(eval_result['pa_mpvpe_hand']),np.mean(eval_result['pa_mpvpe_face']),
np.mean(eval_result['mpvpe_all']),np.mean(eval_result['mpvpe_l_hand']),np.mean(eval_result['mpvpe_r_hand']),np.mean(eval_result['mpvpe_hand']),np.mean(eval_result['mpvpe_face']),
np.mean(eval_result['pa_mpjpe_body']),np.mean(eval_result['pa_mpjpe_l_hand']),np.mean(eval_result['pa_mpjpe_r_hand']),np.mean(eval_result['pa_mpjpe_hand'])]
# Append the new line to the CSV file
with open(csv_file, 'a', newline='') as file:
writer = csv.writer(file)
writer.writerow(new_line)
def decompress_keypoints(self, humandata) -> None:
"""If a key contains 'keypoints', and f'{key}_mask' is in self.keys(),
invalid zeros will be inserted to the right places and f'{key}_mask'
will be unlocked.
Raises:
KeyError:
A key contains 'keypoints' has been found
but its corresponding mask is missing.
"""
assert bool(humandata['__keypoints_compressed__']) is True
key_pairs = []
for key in humandata.files:
if key not in KPS2D_KEYS + KPS3D_KEYS:
continue
mask_key = f'{key}_mask'
if mask_key in humandata.files:
print(f'Decompress {key}...')
key_pairs.append([key, mask_key])
decompressed_dict = {}
for kpt_key, mask_key in key_pairs:
mask_array = np.asarray(humandata[mask_key])
compressed_kpt = humandata[kpt_key]
kpt_array = \
self.add_zero_pad(compressed_kpt, mask_array)
decompressed_dict[kpt_key] = kpt_array
del humandata
return decompressed_dict
def add_zero_pad(self, compressed_array: np.ndarray,
mask_array: np.ndarray) -> np.ndarray:
"""Pad zeros to a compressed keypoints array.
Args:
compressed_array (np.ndarray):
A compressed keypoints array.
mask_array (np.ndarray):
The mask records compression relationship.
Returns:
np.ndarray:
A keypoints array in full-size.
"""
assert mask_array.sum() == compressed_array.shape[1]
data_len, _, dim = compressed_array.shape
mask_len = mask_array.shape[0]
ret_value = np.zeros(
shape=[data_len, mask_len, dim], dtype=compressed_array.dtype)
valid_mask_index = np.where(mask_array == 1)[0]
ret_value[:, valid_mask_index, :] = compressed_array
return ret_value
demo.py
+74
View File
@@ -0,0 +1,74 @@
import os
import sys
import os.path as osp
import argparse
from pathlib import Path
import cv2
import torch
import math
import mmpose
import shutil
import time
from OpenGL import GL
from OpenGL.GL import *
os.environ["PYOPENGL_PLATFORM"] = "osmesa"
import pyrender
try:
import mmpose
except:
os.system('pip install main/transformer_utils')
def parse_args():
parser = argparse.ArgumentParser()
parser.add_argument('--show_verts', action="store_true")
parser.add_argument('--multi_person', action="store_true")
parser.add_argument('--in_threshold', type=float, default=0.5)
parser.add_argument('--output_folder', type=str, default='demo_out')
parser.add_argument('--pretrained_model', type=str, default='smpler_x_h32')
parser.add_argument('--input_video', type=str, default='')
args = parser.parse_args()
return args
def infer():
args = parse_args()
os.makedirs(args.output_folder, exist_ok=True)
num_gpus = 1 if torch.cuda.is_available() else -1
from main.inference import Inferer
inferer = Inferer(args.pretrained_model, num_gpus, args.output_folder)
cap = cv2.VideoCapture(args.input_video)
fps = math.ceil(cap.get(5))
width = int(cap.get(3))
height = int(cap.get(4))
fourcc = cv2.VideoWriter_fourcc(*'mp4v')
video_path = osp.join(args.output_folder, f'out.m4v')
final_video_path = osp.join(args.output_folder, f'out.mp4')
video_output = cv2.VideoWriter(video_path, fourcc, fps, (width, height))
success = 1
frame = 0
while success:
success, original_img = cap.read()
if not success:
break
frame += 1
img, mesh_paths, smplx_paths = inferer.infer(original_img, args.in_threshold, frame, args.multi_person, args.show_verts)
video_output.write(img)
cap.release()
video_output.release()
cv2.destroyAllWindows()
os.system(f'ffmpeg -i {video_path} -c copy {final_video_path}')
#Compress mesh and smplx files
save_path_mesh = os.path.join(args.output_folder, 'mesh')
save_mesh_file = os.path.join(args.output_folder, 'mesh.zip')
os.makedirs(save_path_mesh, exist_ok= True)
save_path_smplx = os.path.join(args.output_folder, 'smplx')
save_smplx_file = os.path.join(args.output_folder, 'smplx.zip')
os.makedirs(save_path_smplx, exist_ok= True)
os.system(f'zip -r {save_mesh_file} {save_path_mesh}')
os.system(f'zip -r {save_smplx_file} {save_path_smplx}')
if __name__ == "__main__":
main()
main/SMPLer_X.py
+6 -6
View File
@@ -9,7 +9,7 @@ from config import cfg
import math
import copy
from mmpose.models import build_posenet
from mmcv import Config
from mmengine.config import Config
class Model(nn.Module):
def __init__(self, encoder, body_position_net, body_rotation_net, box_net, hand_position_net, hand_roi_net,
@@ -30,7 +30,7 @@ class Model(nn.Module):
# face
self.face_regressor = face_regressor
self.smplx_layer = copy.deepcopy(smpl_x.layer['neutral']).cuda()
self.smplx_layer = copy.deepcopy(smpl_x.layer['neutral']).to(cfg.device)
self.coord_loss = CoordLoss()
self.param_loss = ParamLoss()
self.ce_loss = CELoss()
@@ -70,14 +70,14 @@ class Model(nn.Module):
t_xy = cam_param[:, :2]
gamma = torch.sigmoid(cam_param[:, 2]) # apply sigmoid to make it positive
k_value = torch.FloatTensor([math.sqrt(cfg.focal[0] * cfg.focal[1] * cfg.camera_3d_size * cfg.camera_3d_size / (
cfg.input_body_shape[0] * cfg.input_body_shape[1]))]).cuda().view(-1)
cfg.input_body_shape[0] * cfg.input_body_shape[1]))]).to(cfg.device).view(-1)
t_z = k_value * gamma
cam_trans = torch.cat((t_xy, t_z[:, None]), 1)
return cam_trans
def get_coord(self, root_pose, body_pose, lhand_pose, rhand_pose, jaw_pose, shape, expr, cam_trans, mode):
batch_size = root_pose.shape[0]
zero_pose = torch.zeros((1, 3)).float().cuda().repeat(batch_size, 1) # eye poses
zero_pose = torch.zeros((1, 3)).float().to(cfg.device).repeat(batch_size, 1) # eye poses
output = self.smplx_layer(betas=shape, body_pose=body_pose, global_orient=root_pose, right_hand_pose=rhand_pose,
left_hand_pose=lhand_pose, jaw_pose=jaw_pose, leye_pose=zero_pose,
reye_pose=zero_pose, expression=expr)
@@ -318,7 +318,7 @@ class Model(nn.Module):
for bid in range(coord.shape[0]):
mask = meta_info['joint_trunc'][bid, smpl_x.joint_part[part_name], 0] == 1
if torch.sum(mask) == 0:
trans.append(torch.zeros((2)).float().cuda())
trans.append(torch.zeros((2)).float().to(cfg.device))
else:
trans.append((-coord[bid, mask, :2] + targets['joint_img'][:, smpl_x.joint_part[part_name], :][
bid, mask, :2]).mean(0))
@@ -334,7 +334,7 @@ class Model(nn.Module):
for bid in range(coord.shape[0]):
mask = meta_info['joint_trunc'][bid, smpl_x.joint_part['face'], 0] == 1
if torch.sum(mask) == 0:
trans.append(torch.zeros((2)).float().cuda())
trans.append(torch.zeros((2)).float().to(cfg.device))
else:
trans.append((-coord[bid, mask, :2] + targets['joint_img'][:, smpl_x.joint_part['face'], :][bid,
mask, :2]).mean(0))
main/config.py
+7 -13
View File
@@ -2,7 +2,8 @@ import os
import os.path as osp
import sys
import datetime
from mmcv import Config as MMConfig
from mmengine.config import Config as MMConfig
class Config:
def get_config_fromfile(self, config_path):
@@ -18,13 +19,6 @@ class Config:
## add some paths to the system root dir
sys.path.insert(0, osp.join(self.root_dir, 'common'))
from utils.dir import add_pypath
add_pypath(osp.join(self.data_dir))
for dataset in os.listdir(osp.join(self.root_dir, 'data')):
if dataset not in ['humandata.py', '__pycache__', 'dataset.py']:
add_pypath(osp.join(self.root_dir, 'data', dataset))
add_pypath(osp.join(self.root_dir, 'data'))
add_pypath(self.data_dir)
def prepare_dirs(self, exp_name):
time_str = datetime.datetime.now().strftime('%Y%m%d_%H%M%S')
@@ -59,14 +53,14 @@ class Config:
self.eval_on_train = eval_on_train
self.vis = vis
def update_config(self, num_gpus, exp_name):
def update_config(self, num_gpus, pretrained_model_path, output_folder, device):
self.num_gpus = num_gpus
self.exp_name = exp_name
self.prepare_dirs(self.exp_name)
self.pretrained_model_path = pretrained_model_path
self.log_dir = output_folder
self.device = device
# Save
cfg_save = MMConfig(self.__dict__)
cfg_save.dump(osp.join(self.code_dir,'config_base.py'))
cfg = Config()
main/config/config_ft_3dpw.py
-101
View File
@@ -1,101 +0,0 @@
import os
import os.path as osp
# will be update in exp
num_gpus = -1
exp_name = 'output/exp1/pre_analysis'
# quick access
save_epoch = 1
lr = 1e-5
min_lr = 5e-7
end_epoch = 5
train_batch_size = 16
syncbn = True
bbox_ratio = 1.2
# continue
continue_train = True
start_over = True
pretrained_model_path = '../path_to_smpler_x_h32/snapshot.pth.tar'
# dataset setting
agora_fix_betas = True
agora_fix_global_orient_transl = True
agora_valid_root_pose = True
# for ubody ft
dataset_list = ['Human36M', 'MSCOCO', 'MPII', 'AGORA', 'EHF', 'SynBody', 'GTA_Human2', \
'EgoBody_Egocentric', 'EgoBody_Kinect', 'UBody', 'PW3D', 'MuCo', 'PROX']
trainset_3d = ['PW3D']
trainset_2d = []
trainset_humandata = ['GTA_Human2', 'EgoBody_Kinect', 'InstaVariety', 'HumanSC3D']
testset = 'EHF'
use_cache = True
# strategy
data_strategy = 'balance' # 'balance' need to define total_data_len
total_data_len = 1000000 # assign number or 'auto' for concat length
Talkshow_train_sample_interval = 10
# fine-tune
fine_tune = None # 'backbone', 'head', None for full network tuning
smplx_loss_weight = 1.0 #2 for agora_model for smplx shape
smplx_pose_weight = 10.0
smplx_kps_3d_weight = 100.0
smplx_kps_2d_weight = 1.0
net_kps_2d_weight = 1.0
agora_benchmark = 'agora_model' # 'agora_model', 'test_only'
model_type = 'smpler_x_h'
encoder_config_file = 'transformer_utils/configs/smpler_x/encoder/body_encoder_huge.py'
encoder_pretrained_model_path = '../pretrained_models/vitpose_huge.pth'
feat_dim = 1280
## =====FIXED ARGS============================================================
## model setting
upscale = 4
hand_pos_joint_num = 20
face_pos_joint_num = 72
num_task_token = 24
num_noise_sample = 0
## UBody setting
train_sample_interval = 10
test_sample_interval = 100
make_same_len = False
## input, output size
input_img_shape = (512, 384)
input_body_shape = (256, 192)
output_hm_shape = (16, 16, 12)
input_hand_shape = (256, 256)
output_hand_hm_shape = (16, 16, 16)
output_face_hm_shape = (8, 8, 8)
input_face_shape = (192, 192)
focal = (5000, 5000) # virtual focal lengths
princpt = (input_body_shape[1] / 2, input_body_shape[0] / 2) # virtual principal point position
body_3d_size = 2
hand_3d_size = 0.3
face_3d_size = 0.3
camera_3d_size = 2.5
## training config
print_iters = 100
lr_mult = 1
## testing config
test_batch_size = 32
## others
num_thread = 4
vis = False
## directory
output_dir, model_dir, vis_dir, log_dir, result_dir, code_dir = None, None, None, None, None, None
main/config/config_ft_agora.py
-101
View File
@@ -1,101 +0,0 @@
import os
import os.path as osp
# will be update in exp
num_gpus = -1
exp_name = 'output/exp1/pre_analysis'
# quick access
save_epoch = 1
lr = 1e-5
min_lr = 5e-7
end_epoch = 5
train_batch_size = 16
syncbn = True
bbox_ratio = 1.2
# continue
continue_train = True
start_over = True
pretrained_model_path = '../path_to_smpler_x_h32/snapshot.pth.tar'
# dataset setting
agora_fix_betas = True
agora_fix_global_orient_transl = True
agora_valid_root_pose = True
# for ubody ft
dataset_list = ['Human36M', 'MSCOCO', 'MPII', 'AGORA', 'EHF', 'SynBody', 'GTA_Human2', \
'EgoBody_Egocentric', 'EgoBody_Kinect', 'UBody', 'PW3D', 'MuCo', 'PROX']
trainset_3d = ['AGORA']
trainset_2d = []
trainset_humandata = ['GTA_Human2', 'BEDLAM', 'SynBody', 'SPEC']
testset = 'EHF'
use_cache = True
# strategy
data_strategy = 'balance' # 'balance' need to define total_data_len
total_data_len = 1000000 # assign number or 'auto' for concat length
Talkshow_train_sample_interval = 10
# fine-tune
fine_tune = None # 'backbone', 'head', None for full network tuning
smplx_loss_weight = 1.0 #2 for agora_model for smplx shape
smplx_pose_weight = 10.0
smplx_kps_3d_weight = 100.0
smplx_kps_2d_weight = 1.0
net_kps_2d_weight = 1.0
agora_benchmark = 'agora_model' # 'agora_model', 'test_only'
model_type = 'smpler_x_h'
encoder_config_file = 'transformer_utils/configs/smpler_x/encoder/body_encoder_huge.py'
encoder_pretrained_model_path = '../pretrained_models/vitpose_huge.pth'
feat_dim = 1280
## =====FIXED ARGS============================================================
## model setting
upscale = 4
hand_pos_joint_num = 20
face_pos_joint_num = 72
num_task_token = 24
num_noise_sample = 0
## UBody setting
train_sample_interval = 10
test_sample_interval = 100
make_same_len = False
## input, output size
input_img_shape = (512, 384)
input_body_shape = (256, 192)
output_hm_shape = (16, 16, 12)
input_hand_shape = (256, 256)
output_hand_hm_shape = (16, 16, 16)
output_face_hm_shape = (8, 8, 8)
input_face_shape = (192, 192)
focal = (5000, 5000) # virtual focal lengths
princpt = (input_body_shape[1] / 2, input_body_shape[0] / 2) # virtual principal point position
body_3d_size = 2
hand_3d_size = 0.3
face_3d_size = 0.3
camera_3d_size = 2.5
## training config
print_iters = 100
lr_mult = 1
## testing config
test_batch_size = 32
## others
num_thread = 4
vis = False
## directory
output_dir, model_dir, vis_dir, log_dir, result_dir, code_dir = None, None, None, None, None, None
main/config/config_ft_arctic.py
-100
View File
@@ -1,100 +0,0 @@
import os
import os.path as osp
# will be update in exp
num_gpus = -1
exp_name = 'output/exp1/pre_analysis'
# quick access
save_epoch = 1
lr = 1e-5
min_lr = 5e-7
end_epoch = 5
train_batch_size = 16
syncbn = True
bbox_ratio = 1.2
# continue
continue_train = True
start_over = True
pretrained_model_path = '../path_to_smpler_x_h32/snapshot.pth.tar'
# dataset setting
agora_fix_betas = True
agora_fix_global_orient_transl = True
agora_valid_root_pose = True
# for ubody ft
dataset_list = ['Human36M', 'MSCOCO', 'MPII', 'AGORA', 'EHF', 'SynBody', 'GTA_Human2', \
'EgoBody_Egocentric', 'EgoBody_Kinect', 'UBody', 'PW3D', 'MuCo', 'PROX']
trainset_3d = ['MSCOCO']
trainset_2d = []
trainset_humandata = ['ARCTIC', 'BEDLAM', 'EgoBody_Egocentric', 'PoseTrack']
testset = 'EHF'
use_cache = True
# strategy
data_strategy = 'balance' # 'balance' need to define total_data_len
total_data_len = 1000000 # assign number or 'auto' for concat length
Talkshow_train_sample_interval = 10
# fine-tune
fine_tune = None # 'backbone', 'head', None for full network tuning
smplx_loss_weight = 1.0 #2 for agora_model for smplx shape
smplx_pose_weight = 10.0
smplx_kps_3d_weight = 100.0
smplx_kps_2d_weight = 1.0
net_kps_2d_weight = 1.0
agora_benchmark = 'agora_model' # 'agora_model', 'test_only'
model_type = 'smpler_x_h'
encoder_config_file = 'transformer_utils/configs/smpler_x/encoder/body_encoder_huge.py'
encoder_pretrained_model_path = '../pretrained_models/vitpose_huge.pth'
feat_dim = 1280
## =====FIXED ARGS============================================================
## model setting
upscale = 4
hand_pos_joint_num = 20
face_pos_joint_num = 72
num_task_token = 24
num_noise_sample = 0
## UBody setting
train_sample_interval = 10
test_sample_interval = 100
make_same_len = False
## input, output size
input_img_shape = (512, 384)
input_body_shape = (256, 192)
output_hm_shape = (16, 16, 12)
input_hand_shape = (256, 256)
output_hand_hm_shape = (16, 16, 16)
output_face_hm_shape = (8, 8, 8)
input_face_shape = (192, 192)
focal = (5000, 5000) # virtual focal lengths
princpt = (input_body_shape[1] / 2, input_body_shape[0] / 2) # virtual principal point position
body_3d_size = 2
hand_3d_size = 0.3
face_3d_size = 0.3
camera_3d_size = 2.5
## training config
print_iters = 100
lr_mult = 1
## testing config
test_batch_size = 32
## others
num_thread = 4
vis = False
## directory
output_dir, model_dir, vis_dir, log_dir, result_dir, code_dir = None, None, None, None, None, None
main/config/config_ft_egobody.py
-100
View File
@@ -1,100 +0,0 @@
import os
import os.path as osp
# will be update in exp
num_gpus = -1
exp_name = 'output/exp1/pre_analysis'
# quick access
save_epoch = 1
lr = 1e-5
min_lr = 5e-7
end_epoch = 5
train_batch_size = 16
syncbn = True
bbox_ratio = 1.2
# continue
continue_train = True
start_over = True
pretrained_model_path = '../path_to_smpler_x_h32/snapshot.pth.tar'
# dataset setting
agora_fix_betas = True
agora_fix_global_orient_transl = True
agora_valid_root_pose = True
# for ubody ft
dataset_list = ['Human36M', 'MSCOCO', 'MPII', 'AGORA', 'EHF', 'SynBody', 'GTA_Human2', \
'EgoBody_Egocentric', 'EgoBody_Kinect', 'UBody', 'PW3D', 'MuCo', 'PROX']
trainset_3d = ['MSCOCO']
trainset_2d = []
trainset_humandata = ['EgoBody_Egocentric', 'EgoBody_Kinect', 'BEDLAM', 'PROX']
testset = 'EHF'
use_cache = True
# strategy
data_strategy = 'balance' # 'balance' need to define total_data_len
total_data_len = 1000000 # assign number or 'auto' for concat length
Talkshow_train_sample_interval = 10
# fine-tune
fine_tune = None # 'backbone', 'head', None for full network tuning
smplx_loss_weight = 1.0 #2 for agora_model for smplx shape
smplx_pose_weight = 10.0
smplx_kps_3d_weight = 100.0
smplx_kps_2d_weight = 1.0
net_kps_2d_weight = 1.0
agora_benchmark = 'agora_model' # 'agora_model', 'test_only'
model_type = 'smpler_x_h'
encoder_config_file = 'transformer_utils/configs/smpler_x/encoder/body_encoder_huge.py'
encoder_pretrained_model_path = '../pretrained_models/vitpose_huge.pth'
feat_dim = 1280
## =====FIXED ARGS============================================================
## model setting
upscale = 4
hand_pos_joint_num = 20
face_pos_joint_num = 72
num_task_token = 24
num_noise_sample = 0
## UBody setting
train_sample_interval = 10
test_sample_interval = 100
make_same_len = False
## input, output size
input_img_shape = (512, 384)
input_body_shape = (256, 192)
output_hm_shape = (16, 16, 12)
input_hand_shape = (256, 256)
output_hand_hm_shape = (16, 16, 16)
output_face_hm_shape = (8, 8, 8)
input_face_shape = (192, 192)
focal = (5000, 5000) # virtual focal lengths
princpt = (input_body_shape[1] / 2, input_body_shape[0] / 2) # virtual principal point position
body_3d_size = 2
hand_3d_size = 0.3
face_3d_size = 0.3
camera_3d_size = 2.5
## training config
print_iters = 100
lr_mult = 1
## testing config
test_batch_size = 32
## others
num_thread = 4
vis = False
## directory
output_dir, model_dir, vis_dir, log_dir, result_dir, code_dir = None, None, None, None, None, None
main/config/config_ft_renbody.py
-100
View File
@@ -1,100 +0,0 @@
import os
import os.path as osp
# will be update in exp
num_gpus = -1
exp_name = 'output/exp1/pre_analysis'
# quick access
save_epoch = 1
lr = 1e-5
min_lr = 5e-7
end_epoch = 5
train_batch_size = 16
syncbn = True
bbox_ratio = 1.2
# continue
continue_train = True
start_over = True
pretrained_model_path = '../path_to_smpler_x_h32/snapshot.pth.tar'
# dataset setting
agora_fix_betas = True
agora_fix_global_orient_transl = True
agora_valid_root_pose = True
# for ubody ft
dataset_list = ['Human36M', 'MSCOCO', 'MPII', 'AGORA', 'EHF', 'SynBody', 'GTA_Human2', \
'EgoBody_Egocentric', 'EgoBody_Kinect', 'UBody', 'PW3D', 'MuCo', 'PROX']
trainset_3d = []
trainset_2d = []
trainset_humandata = ['RenBody_HiRes', 'RenBody', 'BEDLAM', 'SynBody', 'CHI3D']
testset = 'EHF'
use_cache = True
# strategy
data_strategy = 'balance' # 'balance' need to define total_data_len
total_data_len = 1000000 # assign number or 'auto' for concat length
Talkshow_train_sample_interval = 10
# fine-tune
fine_tune = None # 'backbone', 'head', None for full network tuning
smplx_loss_weight = 1.0 #2 for agora_model for smplx shape
smplx_pose_weight = 10.0
smplx_kps_3d_weight = 100.0
smplx_kps_2d_weight = 1.0
net_kps_2d_weight = 1.0
agora_benchmark = 'agora_model' # 'agora_model', 'test_only'
model_type = 'smpler_x_h'
encoder_config_file = 'transformer_utils/configs/smpler_x/encoder/body_encoder_huge.py'
encoder_pretrained_model_path = '../pretrained_models/vitpose_huge.pth'
feat_dim = 1280
## =====FIXED ARGS============================================================
## model setting
upscale = 4
hand_pos_joint_num = 20
face_pos_joint_num = 72
num_task_token = 24
num_noise_sample = 0
## UBody setting
train_sample_interval = 10
test_sample_interval = 100
make_same_len = False
## input, output size
input_img_shape = (512, 384)
input_body_shape = (256, 192)
output_hm_shape = (16, 16, 12)
input_hand_shape = (256, 256)
output_hand_hm_shape = (16, 16, 16)
output_face_hm_shape = (8, 8, 8)
input_face_shape = (192, 192)
focal = (5000, 5000) # virtual focal lengths
princpt = (input_body_shape[1] / 2, input_body_shape[0] / 2) # virtual principal point position
body_3d_size = 2
hand_3d_size = 0.3
face_3d_size = 0.3
camera_3d_size = 2.5
## training config
print_iters = 100
lr_mult = 1
## testing config
test_batch_size = 32
## others
num_thread = 4
vis = False
## directory
output_dir, model_dir, vis_dir, log_dir, result_dir, code_dir = None, None, None, None, None, None
main/config/config_ft_ubody.py
-101
View File
@@ -1,101 +0,0 @@
import os
import os.path as osp
# will be update in exp
num_gpus = -1
exp_name = 'output/exp1/pre_analysis'
# quick access
save_epoch = 1
lr = 1e-5
min_lr = 5e-7
end_epoch = 5
train_batch_size = 16
syncbn = True
bbox_ratio = 1.2
# continue
continue_train = True
start_over = True
pretrained_model_path = '../path_to_smpler_x_h32/snapshot.pth.tar'
# dataset setting
agora_fix_betas = True
agora_fix_global_orient_transl = True
agora_valid_root_pose = True
# for ubody ft
dataset_list = ['Human36M', 'MSCOCO', 'MPII', 'AGORA', 'EHF', 'SynBody', 'GTA_Human2', \
'EgoBody_Egocentric', 'EgoBody_Kinect', 'UBody', 'PW3D', 'MuCo', 'PROX']
trainset_3d = ['UBody', 'MSCOCO']
trainset_2d = []
trainset_humandata = ['EgoBody_Egocentric', 'PoseTrack', 'Talkshow']
testset = 'EHF'
use_cache = True
# strategy
data_strategy = 'balance' # 'balance' need to define total_data_len
total_data_len = 1000000 # assign number or 'auto' for concat length
Talkshow_train_sample_interval = 10
# fine-tune
fine_tune = None # 'backbone', 'head', None for full network tuning
smplx_loss_weight = 1.0 #2 for agora_model for smplx shape
smplx_pose_weight = 10.0
smplx_kps_3d_weight = 100.0
smplx_kps_2d_weight = 1.0
net_kps_2d_weight = 1.0
agora_benchmark = 'agora_model' # 'agora_model', 'test_only'
model_type = 'smpler_x_h'
encoder_config_file = 'transformer_utils/configs/smpler_x/encoder/body_encoder_huge.py'
encoder_pretrained_model_path = '../pretrained_models/vitpose_huge.pth'
feat_dim = 1280
## =====FIXED ARGS============================================================
## model setting
upscale = 4
hand_pos_joint_num = 20
face_pos_joint_num = 72
num_task_token = 24
num_noise_sample = 0
## UBody setting
train_sample_interval = 10
test_sample_interval = 100
make_same_len = False
## input, output size
input_img_shape = (512, 384)
input_body_shape = (256, 192)
output_hm_shape = (16, 16, 12)
input_hand_shape = (256, 256)
output_hand_hm_shape = (16, 16, 16)
output_face_hm_shape = (8, 8, 8)
input_face_shape = (192, 192)
focal = (5000, 5000) # virtual focal lengths
princpt = (input_body_shape[1] / 2, input_body_shape[0] / 2) # virtual principal point position
body_3d_size = 2
hand_3d_size = 0.3
face_3d_size = 0.3
camera_3d_size = 2.5
## training config
print_iters = 100
lr_mult = 1
## testing config
test_batch_size = 32
## others
num_thread = 4
vis = False
## directory
output_dir, model_dir, vis_dir, log_dir, result_dir, code_dir = None, None, None, None, None, None
main/config/config_smpler_x_b10.py
-108
View File
@@ -1,108 +0,0 @@
import os
import os.path as osp
# will be update in exp
num_gpus = -1
exp_name = 'output/exp1/pre_analysis'
# quick access
save_epoch = 1
lr = 1e-5
end_epoch = 10
train_batch_size = 32
syncbn = True
bbox_ratio = 1.2
# continue
continue_train = False
start_over = True
# dataset setting
agora_fix_betas = True
agora_fix_global_orient_transl = True
agora_valid_root_pose = True
# top10
dataset_list = ['Human36M', 'MSCOCO', 'MPII', 'AGORA', 'EHF', 'SynBody', 'GTA_Human2', \
'EgoBody_Egocentric', 'EgoBody_Kinect', 'UBody', 'PW3D', 'MuCo', 'PROX']
trainset_3d = ['MSCOCO','AGORA']
trainset_2d = []
trainset_humandata = ['BEDLAM', 'SPEC', 'GTA_Human2','SynBody', 'PoseTrack',
'EgoBody_Egocentric', 'PROX', 'CrowdPose']
testset = 'EHF'
use_cache = True
# downsample
BEDLAM_train_sample_interval = 5
EgoBody_Kinect_train_sample_interval = 10
train_sample_interval = 10 # UBody
MPI_INF_3DHP_train_sample_interval = 5
InstaVariety_train_sample_interval = 10
RenBody_HiRes_train_sample_interval = 5
ARCTIC_train_sample_interval = 10
RenBody_train_sample_interval = 10
FIT3D_train_sample_interval = 10
Talkshow_train_sample_interval = 10
# strategy
data_strategy = 'balance' # 'balance' need to define total_data_len
total_data_len = 1500000
# model
smplx_loss_weight = 1.0 #2 for agora_model for smplx shape
smplx_pose_weight = 10.0
smplx_kps_3d_weight = 100.0
smplx_kps_2d_weight = 1.0
net_kps_2d_weight = 1.0
agora_benchmark = 'agora_model' # 'agora_model', 'test_only'
model_type = 'smpler_x_b'
encoder_config_file = 'transformer_utils/configs/smpler_x/encoder/body_encoder_base.py'
encoder_pretrained_model_path = '../pretrained_models/vitpose_base.pth'
feat_dim = 768
## =====FIXED ARGS============================================================
## model setting
upscale = 4
hand_pos_joint_num = 20
face_pos_joint_num = 72
num_task_token = 24
num_noise_sample = 0
## UBody setting
train_sample_interval = 10
test_sample_interval = 100
make_same_len = False
## input, output size
input_img_shape = (512, 384)
input_body_shape = (256, 192)
output_hm_shape = (16, 16, 12)
input_hand_shape = (256, 256)
output_hand_hm_shape = (16, 16, 16)
output_face_hm_shape = (8, 8, 8)
input_face_shape = (192, 192)
focal = (5000, 5000) # virtual focal lengths
princpt = (input_body_shape[1] / 2, input_body_shape[0] / 2) # virtual principal point position
body_3d_size = 2
hand_3d_size = 0.3
face_3d_size = 0.3
camera_3d_size = 2.5
## training config
print_iters = 100
lr_mult = 1
## testing config
test_batch_size = 32
## others
num_thread = 4
vis = False
## directory
output_dir, model_dir, vis_dir, log_dir, result_dir, code_dir = None, None, None, None, None, None
main/config/config_smpler_x_b20.py
-110
View File
@@ -1,110 +0,0 @@
import os
import os.path as osp
# will be update in exp
num_gpus = -1
exp_name = 'output/exp1/pre_analysis'
# quick access
save_epoch = 1
lr = 1e-5
end_epoch = 10
train_batch_size = 32
syncbn = True
bbox_ratio = 1.2
# continue
continue_train = False
start_over = True
# dataset setting
agora_fix_betas = True
agora_fix_global_orient_transl = True
agora_valid_root_pose = True
# top20
dataset_list = ['Human36M', 'MSCOCO', 'MPII', 'AGORA', 'EHF', 'SynBody', 'GTA_Human2', \
'EgoBody_Egocentric', 'EgoBody_Kinect', 'UBody', 'PW3D', 'MuCo', 'PROX']
trainset_3d = ['MSCOCO','AGORA', 'UBody']
trainset_2d = ['PW3D']
trainset_humandata = ['BEDLAM', 'SPEC', 'GTA_Human2','SynBody', 'PoseTrack',
'EgoBody_Egocentric', 'PROX', 'CrowdPose',
'EgoBody_Kinect', 'MPI_INF_3DHP', 'RICH', 'MuCo', 'InstaVariety',
'Behave', 'UP3D', 'ARCTIC' ]
testset = 'EHF'
use_cache = True
# downsample
BEDLAM_train_sample_interval = 5
EgoBody_Kinect_train_sample_interval = 10
train_sample_interval = 10 # UBody
MPI_INF_3DHP_train_sample_interval = 5
InstaVariety_train_sample_interval = 10
RenBody_HiRes_train_sample_interval = 5
ARCTIC_train_sample_interval = 10
RenBody_train_sample_interval = 10
FIT3D_train_sample_interval = 10
Talkshow_train_sample_interval = 10
# strategy
data_strategy = 'balance' # 'balance' need to define total_data_len
total_data_len = 3000000
# model
smplx_loss_weight = 1.0 #2 for agora_model for smplx shape
smplx_pose_weight = 10.0
smplx_kps_3d_weight = 100.0
smplx_kps_2d_weight = 1.0
net_kps_2d_weight = 1.0
agora_benchmark = 'agora_model' # 'agora_model', 'test_only'
model_type = 'smpler_x_b'
encoder_config_file = 'transformer_utils/configs/smpler_x/encoder/body_encoder_base.py'
encoder_pretrained_model_path = '../pretrained_models/vitpose_base.pth'
feat_dim = 768
## =====FIXED ARGS============================================================
## model setting
upscale = 4
hand_pos_joint_num = 20
face_pos_joint_num = 72
num_task_token = 24
num_noise_sample = 0
## UBody setting
train_sample_interval = 10
test_sample_interval = 100
make_same_len = False
## input, output size
input_img_shape = (512, 384)
input_body_shape = (256, 192)
output_hm_shape = (16, 16, 12)
input_hand_shape = (256, 256)
output_hand_hm_shape = (16, 16, 16)
output_face_hm_shape = (8, 8, 8)
input_face_shape = (192, 192)
focal = (5000, 5000) # virtual focal lengths
princpt = (input_body_shape[1] / 2, input_body_shape[0] / 2) # virtual principal point position
body_3d_size = 2
hand_3d_size = 0.3
face_3d_size = 0.3
camera_3d_size = 2.5
## training config
print_iters = 100
lr_mult = 1
## testing config
test_batch_size = 32
## others
num_thread = 4
vis = False
## directory
output_dir, model_dir, vis_dir, log_dir, result_dir, code_dir = None, None, None, None, None, None
main/config/config_smpler_x_b32.py
+2 -2
View File
@@ -64,8 +64,8 @@ net_kps_2d_weight = 1.0
agora_benchmark = 'agora_model' # 'agora_model', 'test_only'
model_type = 'smpler_x_b'
encoder_config_file = 'transformer_utils/configs/smpler_x/encoder/body_encoder_base.py'
encoder_pretrained_model_path = '../pretrained_models/vitpose_base.pth'
encoder_config_file = 'main/transformer_utils/configs/smpler_x/encoder/body_encoder_base.py'
encoder_pretrained_model_path = 'pretrained_models/vitpose_base.pth'
feat_dim = 768
main/config/config_smpler_x_b5.py
-106
View File
@@ -1,106 +0,0 @@
import os
import os.path as osp
# will be update in exp
num_gpus = -1
exp_name = 'output/exp1/pre_analysis'
# quick access
save_epoch = 1
lr = 1e-5
end_epoch = 10
train_batch_size = 32
syncbn = True
bbox_ratio = 1.2
# continue
continue_train = False
start_over = True
# dataset setting
agora_fix_betas = True
agora_fix_global_orient_transl = True
agora_valid_root_pose = True
# top5
dataset_list = ['Human36M', 'MSCOCO', 'MPII', 'AGORA', 'EHF', 'SynBody', 'GTA_Human2', \
'EgoBody_Egocentric', 'EgoBody_Kinect', 'UBody', 'PW3D', 'MuCo', 'PROX']
trainset_3d = ['MSCOCO','AGORA']
trainset_2d = []
trainset_humandata = ['BEDLAM', 'SPEC', 'GTA_Human2']
testset = 'EHF'
use_cache = True
# downsample
BEDLAM_train_sample_interval = 5
EgoBody_Kinect_train_sample_interval = 10
train_sample_interval = 10 # UBody
MPI_INF_3DHP_train_sample_interval = 5
InstaVariety_train_sample_interval = 10
RenBody_HiRes_train_sample_interval = 5
ARCTIC_train_sample_interval = 10
RenBody_train_sample_interval = 10
FIT3D_train_sample_interval = 10
Talkshow_train_sample_interval = 10
# strategy
data_strategy = 'balance' # 'balance' need to define total_data_len
total_data_len = 750000
# model
smplx_loss_weight = 1.0 #2 for agora_model for smplx shape
smplx_pose_weight = 10.0
smplx_kps_3d_weight = 100.0
smplx_kps_2d_weight = 1.0
net_kps_2d_weight = 1.0
agora_benchmark = 'agora_model' # 'agora_model', 'test_only'
model_type = 'smpler_x_b'
encoder_config_file = 'transformer_utils/configs/smpler_x/encoder/body_encoder_base.py'
encoder_pretrained_model_path = '../pretrained_models/vitpose_base.pth'
feat_dim = 768
## =====FIXED ARGS============================================================
## model setting
upscale = 4
hand_pos_joint_num = 20
face_pos_joint_num = 72
num_task_token = 24
num_noise_sample = 0
## UBody setting
train_sample_interval = 10
test_sample_interval = 100
make_same_len = False
## input, output size
input_img_shape = (512, 384)
input_body_shape = (256, 192)
output_hm_shape = (16, 16, 12)
input_hand_shape = (256, 256)
output_hand_hm_shape = (16, 16, 16)
output_face_hm_shape = (8, 8, 8)
input_face_shape = (192, 192)
focal = (5000, 5000) # virtual focal lengths
princpt = (input_body_shape[1] / 2, input_body_shape[0] / 2) # virtual principal point position
body_3d_size = 2
hand_3d_size = 0.3
face_3d_size = 0.3
camera_3d_size = 2.5
## training config
print_iters = 100
lr_mult = 1
## testing config
test_batch_size = 32
## others
num_thread = 4
vis = False
## directory
output_dir, model_dir, vis_dir, log_dir, result_dir, code_dir = None, None, None, None, None, None
main/config/config_smpler_x_h10.py
-107
View File
@@ -1,107 +0,0 @@
import os
import os.path as osp
# will be update in exp
num_gpus = -1
exp_name = 'output/exp1/pre_analysis'
# quick access
save_epoch = 1
lr = 1e-5
end_epoch = 10
train_batch_size = 16
syncbn = True
bbox_ratio = 1.2
# continue
continue_train = False
start_over = True
# dataset setting
agora_fix_betas = True
agora_fix_global_orient_transl = True
agora_valid_root_pose = True
# top10
dataset_list = ['Human36M', 'MSCOCO', 'MPII', 'AGORA', 'EHF', 'SynBody', 'GTA_Human2', \
'EgoBody_Egocentric', 'EgoBody_Kinect', 'UBody', 'PW3D', 'MuCo', 'PROX']
trainset_3d = ['MSCOCO','AGORA']
trainset_2d = []
trainset_humandata = ['BEDLAM', 'SPEC', 'GTA_Human2','SynBody', 'PoseTrack',
'EgoBody_Egocentric', 'PROX', 'CrowdPose']
testset = 'EHF'
use_cache = True
# downsample
BEDLAM_train_sample_interval = 5
EgoBody_Kinect_train_sample_interval = 10
train_sample_interval = 10 # UBody
MPI_INF_3DHP_train_sample_interval = 5
InstaVariety_train_sample_interval = 10
RenBody_HiRes_train_sample_interval = 5
ARCTIC_train_sample_interval = 10
RenBody_train_sample_interval = 10
FIT3D_train_sample_interval = 10
Talkshow_train_sample_interval = 10
# strategy
data_strategy = 'balance' # 'balance' need to define total_data_len
total_data_len = 1500000
# model
smplx_loss_weight = 1.0 #2 for agora_model for smplx shape
smplx_pose_weight = 10.0
smplx_kps_3d_weight = 100.0
smplx_kps_2d_weight = 1.0
net_kps_2d_weight = 1.0
agora_benchmark = 'agora_model' # 'agora_model', 'test_only'
model_type = 'smpler_x_h'
encoder_config_file = 'transformer_utils/configs/smpler_x/encoder/body_encoder_huge.py'
encoder_pretrained_model_path = '../pretrained_models/vitpose_huge.pth'
feat_dim = 1280
## =====FIXED ARGS============================================================
## model setting
upscale = 4
hand_pos_joint_num = 20
face_pos_joint_num = 72
num_task_token = 24
num_noise_sample = 0
## UBody setting
train_sample_interval = 10
test_sample_interval = 100
make_same_len = False
## input, output size
input_img_shape = (512, 384)
input_body_shape = (256, 192)
output_hm_shape = (16, 16, 12)
input_hand_shape = (256, 256)
output_hand_hm_shape = (16, 16, 16)
output_face_hm_shape = (8, 8, 8)
input_face_shape = (192, 192)
focal = (5000, 5000) # virtual focal lengths
princpt = (input_body_shape[1] / 2, input_body_shape[0] / 2) # virtual principal point position
body_3d_size = 2
hand_3d_size = 0.3
face_3d_size = 0.3
camera_3d_size = 2.5
## training config
print_iters = 100
lr_mult = 1
## testing config
test_batch_size = 32
## others
num_thread = 4
vis = False
## directory
output_dir, model_dir, vis_dir, log_dir, result_dir, code_dir = None, None, None, None, None, None
main/config/config_smpler_x_h20.py
-109
View File
@@ -1,109 +0,0 @@
import os
import os.path as osp
# will be update in exp
num_gpus = -1
exp_name = 'output/exp1/pre_analysis'
# quick access
save_epoch = 1
lr = 1e-5
end_epoch = 10
train_batch_size = 16
syncbn = True
bbox_ratio = 1.2
# continue
continue_train = False
start_over = True
# dataset setting
agora_fix_betas = True
agora_fix_global_orient_transl = True
agora_valid_root_pose = True
# top20
dataset_list = ['Human36M', 'MSCOCO', 'MPII', 'AGORA', 'EHF', 'SynBody', 'GTA_Human2', \
'EgoBody_Egocentric', 'EgoBody_Kinect', 'UBody', 'PW3D', 'MuCo', 'PROX']
trainset_3d = ['MSCOCO','AGORA', 'UBody']
trainset_2d = ['PW3D']
trainset_humandata = ['BEDLAM', 'SPEC', 'GTA_Human2','SynBody', 'PoseTrack',
'EgoBody_Egocentric', 'PROX', 'CrowdPose',
'EgoBody_Kinect', 'MPI_INF_3DHP', 'RICH', 'MuCo', 'InstaVariety',
'Behave', 'UP3D', 'ARCTIC' ]
testset = 'EHF'
use_cache = True
# downsample
BEDLAM_train_sample_interval = 5
EgoBody_Kinect_train_sample_interval = 10
train_sample_interval = 10 # UBody
MPI_INF_3DHP_train_sample_interval = 5
InstaVariety_train_sample_interval = 10
RenBody_HiRes_train_sample_interval = 5
ARCTIC_train_sample_interval = 10
RenBody_train_sample_interval = 10
FIT3D_train_sample_interval = 10
Talkshow_train_sample_interval = 10
# strategy
data_strategy = 'balance' # 'balance' need to define total_data_len
total_data_len = 3000000
# model
smplx_loss_weight = 1.0 #2 for agora_model for smplx shape
smplx_pose_weight = 10.0
smplx_kps_3d_weight = 100.0
smplx_kps_2d_weight = 1.0
net_kps_2d_weight = 1.0
agora_benchmark = 'agora_model' # 'agora_model', 'test_only'
model_type = 'smpler_x_h'
encoder_config_file = 'transformer_utils/configs/smpler_x/encoder/body_encoder_huge.py'
encoder_pretrained_model_path = '../pretrained_models/vitpose_huge.pth'
feat_dim = 1280
## =====FIXED ARGS============================================================
## model setting
upscale = 4
hand_pos_joint_num = 20
face_pos_joint_num = 72
num_task_token = 24
num_noise_sample = 0
## UBody setting
train_sample_interval = 10
test_sample_interval = 100
make_same_len = False
## input, output size
input_img_shape = (512, 384)
input_body_shape = (256, 192)
output_hm_shape = (16, 16, 12)
input_hand_shape = (256, 256)
output_hand_hm_shape = (16, 16, 16)
output_face_hm_shape = (8, 8, 8)
input_face_shape = (192, 192)
focal = (5000, 5000) # virtual focal lengths
princpt = (input_body_shape[1] / 2, input_body_shape[0] / 2) # virtual principal point position
body_3d_size = 2
hand_3d_size = 0.3
face_3d_size = 0.3
camera_3d_size = 2.5
## training config
print_iters = 100
lr_mult = 1
## testing config
test_batch_size = 32
## others
num_thread = 4
vis = False
## directory
output_dir, model_dir, vis_dir, log_dir, result_dir, code_dir = None, None, None, None, None, None
main/config/config_smpler_x_h32.py
+2 -2
View File
@@ -64,8 +64,8 @@ net_kps_2d_weight = 1.0
agora_benchmark = 'agora_model' # 'agora_model', 'test_only'
model_type = 'smpler_x_h'
encoder_config_file = 'transformer_utils/configs/smpler_x/encoder/body_encoder_huge.py'
encoder_pretrained_model_path = '../pretrained_models/vitpose_huge.pth'
encoder_config_file = 'main/transformer_utils/configs/smpler_x/encoder/body_encoder_huge.py'
encoder_pretrained_model_path = 'pretrained_models/vitpose_huge.pth'
feat_dim = 1280
## =====FIXED ARGS============================================================
main/config/config_smpler_x_h5.py
-106
View File
@@ -1,106 +0,0 @@
import os
import os.path as osp
# will be update in exp
num_gpus = -1
exp_name = 'output/exp1/pre_analysis'
# quick access
save_epoch = 1
lr = 1e-5
end_epoch = 10
train_batch_size = 16
syncbn = True
bbox_ratio = 1.2
# continue
continue_train = False
start_over = True
# dataset setting
agora_fix_betas = True
agora_fix_global_orient_transl = True
agora_valid_root_pose = True
# top5
dataset_list = ['Human36M', 'MSCOCO', 'MPII', 'AGORA', 'EHF', 'SynBody', 'GTA_Human2', \
'EgoBody_Egocentric', 'EgoBody_Kinect', 'UBody', 'PW3D', 'MuCo', 'PROX']
trainset_3d = ['MSCOCO','AGORA']
trainset_2d = []
trainset_humandata = ['BEDLAM', 'SPEC', 'GTA_Human2']
testset = 'EHF'
use_cache = True
# downsample
BEDLAM_train_sample_interval = 5
EgoBody_Kinect_train_sample_interval = 10
train_sample_interval = 10 # UBody
MPI_INF_3DHP_train_sample_interval = 5
InstaVariety_train_sample_interval = 10
RenBody_HiRes_train_sample_interval = 5
ARCTIC_train_sample_interval = 10
RenBody_train_sample_interval = 10
FIT3D_train_sample_interval = 10
Talkshow_train_sample_interval = 10
# strategy
data_strategy = 'balance' # 'balance' need to define total_data_len
total_data_len = 750000
# model
smplx_loss_weight = 1.0 #2 for agora_model for smplx shape
smplx_pose_weight = 10.0
smplx_kps_3d_weight = 100.0
smplx_kps_2d_weight = 1.0
net_kps_2d_weight = 1.0
agora_benchmark = 'agora_model' # 'agora_model', 'test_only'
model_type = 'smpler_x_h'
encoder_config_file = 'transformer_utils/configs/smpler_x/encoder/body_encoder_huge.py'
encoder_pretrained_model_path = '../pretrained_models/vitpose_huge.pth'
feat_dim = 1280
## =====FIXED ARGS============================================================
## model setting
upscale = 4
hand_pos_joint_num = 20
face_pos_joint_num = 72
num_task_token = 24
num_noise_sample = 0
## UBody setting
train_sample_interval = 10
test_sample_interval = 100
make_same_len = False
## input, output size
input_img_shape = (512, 384)
input_body_shape = (256, 192)
output_hm_shape = (16, 16, 12)
input_hand_shape = (256, 256)
output_hand_hm_shape = (16, 16, 16)
output_face_hm_shape = (8, 8, 8)
input_face_shape = (192, 192)
focal = (5000, 5000) # virtual focal lengths
princpt = (input_body_shape[1] / 2, input_body_shape[0] / 2) # virtual principal point position
body_3d_size = 2
hand_3d_size = 0.3
face_3d_size = 0.3
camera_3d_size = 2.5
## training config
print_iters = 100
lr_mult = 1
## testing config
test_batch_size = 32
## others
num_thread = 4
vis = False
## directory
output_dir, model_dir, vis_dir, log_dir, result_dir, code_dir = None, None, None, None, None, None
main/config/config_smpler_x_l10.py
-109
View File
@@ -1,109 +0,0 @@
import os
import os.path as osp
# will be update in exp
num_gpus = -1
exp_name = 'output/exp1/pre_analysis'
# quick access
save_epoch = 1
lr = 1e-5
end_epoch = 10
train_batch_size = 32
syncbn = True
bbox_ratio = 1.2
# continue
continue_train = False
start_over = True
# dataset setting
agora_fix_betas = True
agora_fix_global_orient_transl = True
agora_valid_root_pose = True
# top10
dataset_list = ['Human36M', 'MSCOCO', 'MPII', 'AGORA', 'EHF', 'SynBody', 'GTA_Human2', \
'EgoBody_Egocentric', 'EgoBody_Kinect', 'UBody', 'PW3D', 'MuCo', 'PROX']
trainset_3d = ['MSCOCO','AGORA']
trainset_2d = []
trainset_humandata = ['BEDLAM', 'SPEC', 'GTA_Human2','SynBody', 'PoseTrack',
'EgoBody_Egocentric', 'PROX', 'CrowdPose']
testset = 'EHF'
use_cache = True
# downsample
BEDLAM_train_sample_interval = 5
EgoBody_Kinect_train_sample_interval = 10
train_sample_interval = 10 # UBody
MPI_INF_3DHP_train_sample_interval = 5
InstaVariety_train_sample_interval = 10
RenBody_HiRes_train_sample_interval = 5
ARCTIC_train_sample_interval = 10
RenBody_train_sample_interval = 10
FIT3D_train_sample_interval = 10
Talkshow_train_sample_interval = 10
# strategy
data_strategy = 'balance' # 'balance' need to define total_data_len
total_data_len = 1500000
# model
smplx_loss_weight = 1.0 #2 for agora_model for smplx shape
smplx_pose_weight = 10.0
smplx_kps_3d_weight = 100.0
smplx_kps_2d_weight = 1.0
net_kps_2d_weight = 1.0
agora_benchmark = 'agora_model' # 'agora_model', 'test_only'
model_type = 'smpler_x_l'
encoder_config_file = 'transformer_utils/configs/smpler_x/encoder/body_encoder_large.py'
encoder_pretrained_model_path = '../pretrained_models/vitpose_large.pth'
feat_dim = 1024
## =====FIXED ARGS============================================================
## model setting
upscale = 4
hand_pos_joint_num = 20
face_pos_joint_num = 72
num_task_token = 24
num_noise_sample = 0
## UBody setting
train_sample_interval = 10
test_sample_interval = 100
make_same_len = False
## input, output size
input_img_shape = (512, 384)
input_body_shape = (256, 192)
output_hm_shape = (16, 16, 12)
input_hand_shape = (256, 256)
output_hand_hm_shape = (16, 16, 16)
output_face_hm_shape = (8, 8, 8)
input_face_shape = (192, 192)
focal = (5000, 5000) # virtual focal lengths
princpt = (input_body_shape[1] / 2, input_body_shape[0] / 2) # virtual principal point position
body_3d_size = 2
hand_3d_size = 0.3
face_3d_size = 0.3
camera_3d_size = 2.5
## training config
print_iters = 100
lr_mult = 1
## testing config
test_batch_size = 32
## others
num_thread = 4
vis = False
## directory
output_dir, model_dir, vis_dir, log_dir, result_dir, code_dir = None, None, None, None, None, None
main/config/config_smpler_x_l20.py
-110
View File
@@ -1,110 +0,0 @@
import os
import os.path as osp
# will be update in exp
num_gpus = -1
exp_name = 'output/exp1/pre_analysis'
# quick access
save_epoch = 1
lr = 1e-5
end_epoch = 10
train_batch_size = 32
syncbn = True
bbox_ratio = 1.2
# continue
continue_train = False
start_over = True
# dataset setting
agora_fix_betas = True
agora_fix_global_orient_transl = True
agora_valid_root_pose = True
# top20
dataset_list = ['Human36M', 'MSCOCO', 'MPII', 'AGORA', 'EHF', 'SynBody', 'GTA_Human2', \
'EgoBody_Egocentric', 'EgoBody_Kinect', 'UBody', 'PW3D', 'MuCo', 'PROX']
trainset_3d = ['MSCOCO','AGORA', 'UBody']
trainset_2d = ['PW3D']
trainset_humandata = ['BEDLAM', 'SPEC', 'GTA_Human2','SynBody', 'PoseTrack',
'EgoBody_Egocentric', 'PROX', 'CrowdPose',
'EgoBody_Kinect', 'MPI_INF_3DHP', 'RICH', 'MuCo', 'InstaVariety',
'Behave', 'UP3D', 'ARCTIC' ]
testset = 'EHF'
use_cache = True
# downsample
BEDLAM_train_sample_interval = 5
EgoBody_Kinect_train_sample_interval = 10
train_sample_interval = 10 # UBody
MPI_INF_3DHP_train_sample_interval = 5
InstaVariety_train_sample_interval = 10
RenBody_HiRes_train_sample_interval = 5
ARCTIC_train_sample_interval = 10
RenBody_train_sample_interval = 10
FIT3D_train_sample_interval = 10
Talkshow_train_sample_interval = 10
# strategy
data_strategy = 'balance' # 'balance' need to define total_data_len
total_data_len = 3000000
# model
smplx_loss_weight = 1.0 #2 for agora_model for smplx shape
smplx_pose_weight = 10.0
smplx_kps_3d_weight = 100.0
smplx_kps_2d_weight = 1.0
net_kps_2d_weight = 1.0
agora_benchmark = 'agora_model' # 'agora_model', 'test_only'
model_type = 'smpler_x_l'
encoder_config_file = 'transformer_utils/configs/smpler_x/encoder/body_encoder_large.py'
encoder_pretrained_model_path = '../pretrained_models/vitpose_large.pth'
feat_dim = 1024
## =====FIXED ARGS============================================================
## model setting
upscale = 4
hand_pos_joint_num = 20
face_pos_joint_num = 72
num_task_token = 24
num_noise_sample = 0
## UBody setting
train_sample_interval = 10
test_sample_interval = 100
make_same_len = False
## input, output size
input_img_shape = (512, 384)
input_body_shape = (256, 192)
output_hm_shape = (16, 16, 12)
input_hand_shape = (256, 256)
output_hand_hm_shape = (16, 16, 16)
output_face_hm_shape = (8, 8, 8)
input_face_shape = (192, 192)
focal = (5000, 5000) # virtual focal lengths
princpt = (input_body_shape[1] / 2, input_body_shape[0] / 2) # virtual principal point position
body_3d_size = 2
hand_3d_size = 0.3
face_3d_size = 0.3
camera_3d_size = 2.5
## training config
print_iters = 100
lr_mult = 1
## testing config
test_batch_size = 32
## others
num_thread = 4
vis = False
## directory
output_dir, model_dir, vis_dir, log_dir, result_dir, code_dir = None, None, None, None, None, None
main/config/config_smpler_x_l32.py
+2 -2
View File
@@ -64,8 +64,8 @@ net_kps_2d_weight = 1.0
agora_benchmark = 'agora_model' # 'agora_model', 'test_only'
model_type = 'smpler_x_l'
encoder_config_file = 'transformer_utils/configs/smpler_x/encoder/body_encoder_large.py'
encoder_pretrained_model_path = '../pretrained_models/vitpose_large.pth'
encoder_config_file = 'main/transformer_utils/configs/smpler_x/encoder/body_encoder_large.py'
encoder_pretrained_model_path = 'pretrained_models/vitpose_large.pth'
feat_dim = 1024
main/config/config_smpler_x_l5.py
-107
View File
@@ -1,107 +0,0 @@
import os
import os.path as osp
# will be update in exp
num_gpus = -1
exp_name = 'output/exp1/pre_analysis'
# quick access
save_epoch = 1
lr = 2e-5
end_epoch = 10
train_batch_size = 32
syncbn = True
bbox_ratio = 1.2
# continue
continue_train = False
start_over = True
# dataset setting
agora_fix_betas = True
agora_fix_global_orient_transl = True
agora_valid_root_pose = True
# top5
dataset_list = ['Human36M', 'MSCOCO', 'MPII', 'AGORA', 'EHF', 'SynBody', 'GTA_Human2', \
'EgoBody_Egocentric', 'EgoBody_Kinect', 'UBody', 'PW3D', 'MuCo', 'PROX']
trainset_3d = ['MSCOCO','AGORA']
trainset_2d = []
trainset_humandata = ['BEDLAM', 'SPEC', 'GTA_Human2']
testset = 'EHF'
use_cache = True
# downsample
BEDLAM_train_sample_interval = 5
EgoBody_Kinect_train_sample_interval = 10
train_sample_interval = 10 # UBody
MPI_INF_3DHP_train_sample_interval = 5
InstaVariety_train_sample_interval = 10
RenBody_HiRes_train_sample_interval = 5
ARCTIC_train_sample_interval = 10
RenBody_train_sample_interval = 10
FIT3D_train_sample_interval = 10
Talkshow_train_sample_interval = 10
# strategy
data_strategy = 'balance' # 'balance' need to define total_data_len
total_data_len = 750000
# model
smplx_loss_weight = 1.0 #2 for agora_model for smplx shape
smplx_pose_weight = 10.0
smplx_kps_3d_weight = 100.0
smplx_kps_2d_weight = 1.0
net_kps_2d_weight = 1.0
agora_benchmark = 'agora_model' # 'agora_model', 'test_only'
model_type = 'smpler_x_l'
encoder_config_file = 'transformer_utils/configs/smpler_x/encoder/body_encoder_large.py'
encoder_pretrained_model_path = '../pretrained_models/vitpose_large.pth'
feat_dim = 1024
## =====FIXED ARGS============================================================
## model setting
upscale = 4
hand_pos_joint_num = 20
face_pos_joint_num = 72
num_task_token = 24
num_noise_sample = 0
## UBody setting
train_sample_interval = 10
test_sample_interval = 100
make_same_len = False
## input, output size
input_img_shape = (512, 384)
input_body_shape = (256, 192)
output_hm_shape = (16, 16, 12)
input_hand_shape = (256, 256)
output_hand_hm_shape = (16, 16, 16)
output_face_hm_shape = (8, 8, 8)
input_face_shape = (192, 192)
focal = (5000, 5000) # virtual focal lengths
princpt = (input_body_shape[1] / 2, input_body_shape[0] / 2) # virtual principal point position
body_3d_size = 2
hand_3d_size = 0.3
face_3d_size = 0.3
camera_3d_size = 2.5
## training config
print_iters = 100
lr_mult = 1
## testing config
test_batch_size = 32
## others
num_thread = 4
vis = False
## directory
output_dir, model_dir, vis_dir, log_dir, result_dir, code_dir = None, None, None, None, None, None
main/config/config_smpler_x_s10.py
-108
View File
@@ -1,108 +0,0 @@
import os
import os.path as osp
# will be update in exp
num_gpus = -1
exp_name = 'output/exp1/pre_analysis'
# quick access
save_epoch = 1
lr = 1e-5
end_epoch = 10
train_batch_size = 32
syncbn = True
bbox_ratio = 1.2
# continue
continue_train = False
start_over = True
# dataset setting
agora_fix_betas = True
agora_fix_global_orient_transl = True
agora_valid_root_pose = True
# top10
dataset_list = ['Human36M', 'MSCOCO', 'MPII', 'AGORA', 'EHF', 'SynBody', 'GTA_Human2', \
'EgoBody_Egocentric', 'EgoBody_Kinect', 'UBody', 'PW3D', 'MuCo', 'PROX']
trainset_3d = ['MSCOCO','AGORA']
trainset_2d = []
trainset_humandata = ['BEDLAM', 'SPEC', 'GTA_Human2','SynBody', 'PoseTrack',
'EgoBody_Egocentric', 'PROX', 'CrowdPose']
testset = 'EHF'
use_cache = True
# downsample
BEDLAM_train_sample_interval = 5
EgoBody_Kinect_train_sample_interval = 10
train_sample_interval = 10 # UBody
MPI_INF_3DHP_train_sample_interval = 5
InstaVariety_train_sample_interval = 10
RenBody_HiRes_train_sample_interval = 5
ARCTIC_train_sample_interval = 10
RenBody_train_sample_interval = 10
FIT3D_train_sample_interval = 10
Talkshow_train_sample_interval = 10
# strategy
data_strategy = 'balance' # 'balance' need to define total_data_len
total_data_len = 1500000
# model
smplx_loss_weight = 1.0 #2 for agora_model for smplx shape
smplx_pose_weight = 10.0
smplx_kps_3d_weight = 100.0
smplx_kps_2d_weight = 1.0
net_kps_2d_weight = 1.0
agora_benchmark = 'agora_model' # 'agora_model', 'test_only'
model_type = 'smpler_x_s'
encoder_config_file = 'transformer_utils/configs/smpler_x/encoder/body_encoder_small.py'
encoder_pretrained_model_path = '../pretrained_models/vitpose_small.pth'
feat_dim = 384
## =====FIXED ARGS============================================================
## model setting
upscale = 4
hand_pos_joint_num = 20
face_pos_joint_num = 72
num_task_token = 24
num_noise_sample = 0
## UBody setting
train_sample_interval = 10
test_sample_interval = 100
make_same_len = False
## input, output size
input_img_shape = (512, 384)
input_body_shape = (256, 192)
output_hm_shape = (16, 16, 12)
input_hand_shape = (256, 256)
output_hand_hm_shape = (16, 16, 16)
output_face_hm_shape = (8, 8, 8)
input_face_shape = (192, 192)
focal = (5000, 5000) # virtual focal lengths
princpt = (input_body_shape[1] / 2, input_body_shape[0] / 2) # virtual principal point position
body_3d_size = 2
hand_3d_size = 0.3
face_3d_size = 0.3
camera_3d_size = 2.5
## training config
print_iters = 100
lr_mult = 1
## testing config
test_batch_size = 32
## others
num_thread = 4
vis = False
## directory
output_dir, model_dir, vis_dir, log_dir, result_dir, code_dir = None, None, None, None, None, None
main/config/config_smpler_x_s20.py
-110
View File
@@ -1,110 +0,0 @@
import os
import os.path as osp
# will be update in exp
num_gpus = -1
exp_name = 'output/exp1/pre_analysis'
# quick access
save_epoch = 1
lr = 1e-5
end_epoch = 10
train_batch_size = 32
syncbn = True
bbox_ratio = 1.2
# continue
continue_train = False
start_over = True
# dataset setting
agora_fix_betas = True
agora_fix_global_orient_transl = True
agora_valid_root_pose = True
# top20
dataset_list = ['Human36M', 'MSCOCO', 'MPII', 'AGORA', 'EHF', 'SynBody', 'GTA_Human2', \
'EgoBody_Egocentric', 'EgoBody_Kinect', 'UBody', 'PW3D', 'MuCo', 'PROX']
trainset_3d = ['MSCOCO','AGORA', 'UBody']
trainset_2d = ['PW3D']
trainset_humandata = ['BEDLAM', 'SPEC', 'GTA_Human2','SynBody', 'PoseTrack',
'EgoBody_Egocentric', 'PROX', 'CrowdPose',
'EgoBody_Kinect', 'MPI_INF_3DHP', 'RICH', 'MuCo', 'InstaVariety',
'Behave', 'UP3D', 'ARCTIC' ]
testset = 'EHF'
use_cache = True
# downsample
BEDLAM_train_sample_interval = 5
EgoBody_Kinect_train_sample_interval = 10
train_sample_interval = 10 # UBody
MPI_INF_3DHP_train_sample_interval = 5
InstaVariety_train_sample_interval = 10
RenBody_HiRes_train_sample_interval = 5
ARCTIC_train_sample_interval = 10
RenBody_train_sample_interval = 10
FIT3D_train_sample_interval = 10
Talkshow_train_sample_interval = 10
# strategy
data_strategy = 'balance' # 'balance' need to define total_data_len
total_data_len = 3000000
# model
smplx_loss_weight = 1.0 #2 for agora_model for smplx shape
smplx_pose_weight = 10.0
smplx_kps_3d_weight = 100.0
smplx_kps_2d_weight = 1.0
net_kps_2d_weight = 1.0
agora_benchmark = 'agora_model' # 'agora_model', 'test_only'
model_type = 'smpler_x_s'
encoder_config_file = 'transformer_utils/configs/smpler_x/encoder/body_encoder_small.py'
encoder_pretrained_model_path = '../pretrained_models/vitpose_small.pth'
feat_dim = 384
## =====FIXED ARGS============================================================
## model setting
upscale = 4
hand_pos_joint_num = 20
face_pos_joint_num = 72
num_task_token = 24
num_noise_sample = 0
## UBody setting
train_sample_interval = 10
test_sample_interval = 100
make_same_len = False
## input, output size
input_img_shape = (512, 384)
input_body_shape = (256, 192)
output_hm_shape = (16, 16, 12)
input_hand_shape = (256, 256)
output_hand_hm_shape = (16, 16, 16)
output_face_hm_shape = (8, 8, 8)
input_face_shape = (192, 192)
focal = (5000, 5000) # virtual focal lengths
princpt = (input_body_shape[1] / 2, input_body_shape[0] / 2) # virtual principal point position
body_3d_size = 2
hand_3d_size = 0.3
face_3d_size = 0.3
camera_3d_size = 2.5
## training config
print_iters = 100
lr_mult = 1
## testing config
test_batch_size = 32
## others
num_thread = 4
vis = False
## directory
output_dir, model_dir, vis_dir, log_dir, result_dir, code_dir = None, None, None, None, None, None
main/config/config_smpler_x_s32.py
+2 -2
View File
@@ -64,8 +64,8 @@ net_kps_2d_weight = 1.0
agora_benchmark = 'agora_model' # 'agora_model', 'test_only'
model_type = 'smpler_x_s'
encoder_config_file = 'transformer_utils/configs/smpler_x/encoder/body_encoder_small.py'
encoder_pretrained_model_path = '../pretrained_models/vitpose_small.pth'
encoder_config_file = 'main/transformer_utils/configs/smpler_x/encoder/body_encoder_small.py'
encoder_pretrained_model_path = 'pretrained_models/vitpose_small.pth'
feat_dim = 384
## =====FIXED ARGS============================================================
main/config/config_smpler_x_s5.py
-107
View File
@@ -1,107 +0,0 @@
import os
import os.path as osp
# will be update in exp
num_gpus = -1
exp_name = 'output/exp1/pre_analysis'
# quick access
save_epoch = 1
lr = 1e-5
end_epoch = 10
train_batch_size = 32
syncbn = True
bbox_ratio = 1.2
# continue
continue_train = False
start_over = True
# dataset setting
agora_fix_betas = True
agora_fix_global_orient_transl = True
agora_valid_root_pose = True
# top5
dataset_list = ['Human36M', 'MSCOCO', 'MPII', 'AGORA', 'EHF', 'SynBody', 'GTA_Human2', \
'EgoBody_Egocentric', 'EgoBody_Kinect', 'UBody', 'PW3D', 'MuCo', 'PROX']
trainset_3d = ['MSCOCO','AGORA']
trainset_2d = []
trainset_humandata = ['BEDLAM', 'SPEC', 'GTA_Human2']
testset = 'EHF'
use_cache = True
# downsample
BEDLAM_train_sample_interval = 5
EgoBody_Kinect_train_sample_interval = 10
train_sample_interval = 10 # UBody
MPI_INF_3DHP_train_sample_interval = 5
InstaVariety_train_sample_interval = 10
RenBody_HiRes_train_sample_interval = 5
ARCTIC_train_sample_interval = 10
RenBody_train_sample_interval = 10
FIT3D_train_sample_interval = 10
Talkshow_train_sample_interval = 10
# strategy
data_strategy = 'balance' # 'balance' need to define total_data_len
total_data_len = 750000
# model
smplx_loss_weight = 1.0 #2 for agora_model for smplx shape
smplx_pose_weight = 10.0
smplx_kps_3d_weight = 100.0
smplx_kps_2d_weight = 1.0
net_kps_2d_weight = 1.0
agora_benchmark = 'agora_model' # 'agora_model', 'test_only'
model_type = 'smpler_x_s'
encoder_config_file = 'transformer_utils/configs/smpler_x/encoder/body_encoder_small.py'
encoder_pretrained_model_path = '../pretrained_models/vitpose_small.pth'
feat_dim = 384
## =====FIXED ARGS============================================================
## model setting
upscale = 4
hand_pos_joint_num = 20
face_pos_joint_num = 72
num_task_token = 24
num_noise_sample = 0
## UBody setting
train_sample_interval = 10
test_sample_interval = 100
make_same_len = False
## input, output size
input_img_shape = (512, 384)
input_body_shape = (256, 192)
output_hm_shape = (16, 16, 12)
input_hand_shape = (256, 256)
output_hand_hm_shape = (16, 16, 16)
output_face_hm_shape = (8, 8, 8)
input_face_shape = (192, 192)
focal = (5000, 5000) # virtual focal lengths
princpt = (input_body_shape[1] / 2, input_body_shape[0] / 2) # virtual principal point position
body_3d_size = 2
hand_3d_size = 0.3
face_3d_size = 0.3
camera_3d_size = 2.5
## training config
print_iters = 100
lr_mult = 1
## testing config
test_batch_size = 32
## others
num_thread = 4
vis = False
## directory
output_dir, model_dir, vis_dir, log_dir, result_dir, code_dir = None, None, None, None, None, None

Some files were not shown because too many files have changed in this diff Show More