Detectron2 - 2

Posted on Edited on In ,

For my FIRST blog Detectron2 - 1 has already demonstrated its stunning capability, I NEVER thought I'll have a 2nd chance to write a 2nd blog about detectron2.

In this crazy year 2021 (yeah, today is THE HOTTEST EVER DAY recorded in Canada ), detectron2 has some updates as in its What's New. Today, I'd love to test out Densepose in depth. By the way, today, I finally received OpenCV AI KIT. We'll have some fun on that next week.

1. My Environments

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
➜  ~ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.2 LTS
Release:	20.04
Codename:	focal
➜  ~ python --version
Python 3.8.5
➜  ~ pip show torch
Name: torch
Version: 1.10.0a0+git3137bbe
Summary: Tensors and Dynamic neural networks in Python with strong GPU acceleration
Home-page: https://pytorch.org/
Author: PyTorch Team
Author-email: packages@pytorch.org
License: BSD-3
Location: ~/.local/lib/python3.8/site-packages
Requires: typing-extensions
Required-by: torchvision, torchaudio, torchtext
➜  ~ pip show torchvision
Name: torchvision
Version: 0.11.0a0+0013d93
Summary: image and video datasets and models for torch deep learning
Home-page: https://github.com/pytorch/vision
Author: PyTorch Core Team
Author-email: soumith@pytorch.org
License: BSD
Location: ~/.local/lib/python3.8/site-packages/torchvision-0.11.0a0+0013d93-py3.8-linux-x86_64.egg
Requires: numpy, torch, pillow
Required-by: 
➜  ~ pip show detectron2
Name: detectron2
Version: 0.4.1
Summary: Detectron2 is FAIR's next-generation research platform for object detection and segmentation.
Home-page: https://github.com/facebookresearch/detectron2
Author: FAIR
Author-email: None
License: UNKNOWN
Location: ~/.local/lib/python3.8/site-packages/detectron2-0.4.1-py3.8-linux-x86_64.egg
Requires: Pillow, matplotlib, pycocotools, termcolor, yacs, tabulate, cloudpickle, tqdm, tensorboard, fvcore, iopath, future, pydot, omegaconf, hydra-core, black
Required-by: 
➜  ~ apt show opencv
Package: opencv
Version: 4.5.2-6
Status: install ok installed
Priority: extra
Section: checkinstall
Maintainer: root@lvision-MS-7C59
Installed-Size: 2,188 MB
Provides: build
Download-Size: unknown
APT-Manual-Installed: yes
APT-Sources: /var/lib/dpkg/status
Description: Package created with checkinstall 1.6.3
➜  ~ gcc --version
gcc (GCC) 10.3.1 20210627
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

2. DensePose in Detectron2

2.1 Demo on Single Image

We first extract a single image from a youtube video Dance Workout- EXO: Love Shot, follow by Kkardio. The reasons why we select this video are: there are two parts of the video: - the left part contains multiple front-facing dancers - the right part contains a single back-facing fllower

2.1.1 Command Line for IUV

1
2
3
4
5
6
7
8
9
10
11
12
➜  DensePose git:(master) ✗ python apply_net.py show configs/densepose_rcnn_R_50_FPN_s1x.yaml ./pkl/IUV/R_50_FPN_s1x.pkl image.png dp_contour,bbox --output image_densepose_contour.png 
The checkpoint state_dict contains keys that are not used by the model:
  proposal_generator.anchor_generator.cell_anchors.{0, 1, 2, 3, 4}
~/.local/lib/python3.8/site-packages/torch/_tensor.py:585: UserWarning: floor_divide is deprecated, and will be removed in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values.
To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). (Triggered internally at  ../aten/src/ATen/native/BinaryOps.cpp:461.)
  return torch.floor_divide(self, other)
~/.local/lib/python3.8/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  ../c10/core/TensorImpl.h:1292.)
  return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
....../detectron2/projects/DensePose/densepose/vis/densepose_results.py:133: UserWarning: The following kwargs were not used by contour: 'cfg', 'texture_atlas', 'texture_atlases_dict'
  plt.contour(u, self.levels, extent=extent, **self.plot_args)
....../detectron2/projects/DensePose/densepose/vis/densepose_results.py:134: UserWarning: The following kwargs were not used by contour: 'cfg', 'texture_atlas', 'texture_atlases_dict'
  plt.contour(v, self.levels, extent=extent, **self.plot_args)

2.1.2 Command Line for CSE

It looks smpl_uvSnapshot_colors.jpg is missing?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
➜  DensePose git:(master) ✗ python apply_net.py show configs/cse/densepose_rcnn_R_50_FPN_s1x.yaml ./pkl/CSE/R_50_FPN_s1x.pkl image.png dp_vertex,bbox -v
[06/28 16:19:54 apply_net]: Loading config from configs/cse/densepose_rcnn_R_50_FPN_s1x.yaml
[06/28 16:19:54 apply_net]: Loading model from ./pkl/CSE/R_50_FPN_s1x.pkl
The checkpoint state_dict contains keys that are not used by the model:
  pixel_mean
  pixel_std
  proposal_generator.anchor_generator.cell_anchors.{0, 1, 2, 3, 4}
[06/28 16:19:56 apply_net]: Loading data from image.png
~/.local/lib/python3.8/site-packages/torch/_tensor.py:585: UserWarning: floor_divide is deprecated, and will be removed in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values.
To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). (Triggered internally at  ../aten/src/ATen/native/BinaryOps.cpp:461.)
  return torch.floor_divide(self, other)
~/.local/lib/python3.8/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  ../c10/core/TensorImpl.h:1292.)
  return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
[06/28 16:19:57 apply_net]: Processing image.png
~/.local/lib/python3.8/site-packages/torch/nn/functional.py:3609: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
  warnings.warn(
[06/28 16:19:58 apply_net]: Output saved to outputres.0001.png

2.1.3 Results

Dance Workout- EXO: Love Shot, follow by Kkardio
Dance Workout- EXO: Love Shot, follow by Kkardio
Dense Pose Contour IUV
Dense Pose Contour IUV
Dense Pose Continuous Surface Embedding
Dense Pose CSE

From the CSE result, it's quite obvious that detectron2 Densepose CSE is able to tell the left leg from the right leg.

2.2 Demo on Video

The original video is embedded in the following:

2.2.1 Code

The code I used for my demo heavily relies on detectron2 issue 481 and detectron2 Densepose apply_net.py. Please modify the code by yourself according to your own applications.

2.2.2 Results

2.2.2.1 IUV

IUV Model Zoo

2.2.2.2 CSE

CSE Model Zoo

Again, from the CSE result, it's quite obvious that detectron2 Densepose CSE is able to tell the left leg from the right leg.