Introduction

This is the AI Watch A1 docum entation for C++. AI Watch A1 uses one Intel RealSense D435 camera and OpenPose in order to achieve multi-person 3D skeleton detection. Once this task is completed, AI Watch A1 provides support to send each detected skeleton and its joint points' 3D coordinates (room and D435's details have to be specified) via Kafka, in order to let further detached computation possible.

Keynote 📋

Demo 💻

Other Docs 📜

Bachelor Thesis in Computer Science: PDF TesiOnline

Installation 🚀

Note: Instructions for MacOS with Intel CPU / M1 chip

1) Install RealSense SDK 2.0 and its own dependencies. The following guide is really helpful: build RealSense for macOS Monterey (Intel + Apple Silicon).

2) Install OpenPose and its own dependencies. Useful guides: build openpose with/without GPU support for macOS, OpenPose for M1/Intel.

In order to properly choose the GPU_MODE during OpenPose installation and/or in order to install OpenPose on other operating systems, check out this page: OpenPose Docs. If you have MacOS with Intel CPU and a GPU that matches OpenPose prerequisites, then you can set GPU_MODE to OPENCL (GPU-accelerated, it is harder to install but provides a faster runtime speed). Otherwise, if you have a MacOS with M1 chip, it's suggested to set GPU_MODE to CPU_ONLY.

CMake Settings for Intel

CMake Settings for >= M1

3) Install Apache Kafka, Confluent and their own dependencies. Confluent is not mandatory, but without it, you will have to set up the Kafka environment on your own. Useful guides:

how to install Apache Kafka on Mac, how to install Confluent.

4) Run the following command in your terminal:

git clone --recursive git@github.com:dennewbie/AI_Watch_A1.git

After that, remove all the files named emptyFileForPadding.txt in the cloned folder.

5) From /AI_Watch_A1/src/AI_Watch_A1/ folder, run the following commands in your terminal:

mkdir build && cd build

6) Copy OpenPose's models folder to the project's build folder.

7) Copy OpenPose's BoostConfig.cmake, FindGFlags.cmake and FindGlog.cmake files to the build/cmake/modules/ folder.

8) Update caffe lib path inside CMakeLists.txt. Then run the following commands in your terminal:

cmake .. && make -j `sysctl -n hw.logicalcpu`

9) This step is optional. Run the following command in your terminal:

sudo make install

10) Now let's start the Kafka environment. Run the following commands in a new terminal session located on the parent folder of the confluent folder. Set the environment variable for the Confluent Platform home directory:

export CONFLUENT_HOME=confluent-7.2.1

Add the Confluent Platform bin directory to your PATH:

export PATH=$PATH:$CONFLUENT_HOME/bin

Test your installation by running the confluent command:

confluent --help

Your output should show the available commands for managing Confluent Platform.

Start Confluent Platform by using the Confluent CLI confluent local services start command. This command starts all of the Confluent Platform components, including Kafka, ZooKeeper, Schema Registry, HTTP REST Proxy for Kafka, Kafka Connect, ksqlDB, and Control Center.

confluent local services start

Your output should resemble:

Starting Zookeeper
Zookeeper is [UP]
Starting Kafka
Kafka is [UP]
Starting Schema Registry
Schema Registry is [UP]
Starting Kafka REST
Kafka REST is [UP]
Starting Connect
Connect is [UP]
Starting KSQL Server
KSQL Server is [UP]
Starting Control Center
Control Center is [UP]

Important #1 📌

The confluent local commands are intended for a single-node development environment and are not suitable for a production environment. The data that are produced are transient and are intended to be temporary. For production-ready workflows, check Confluent website.

Important #2 📌

The Confluent CLI requires Java version 1.8 or 1.11. See Confluent versions interoperability.

Important #3 📌

In order to improve speed and general performance, you could disable images' showing and change the number of frames captured per second. In this last option, capturing too few frames per second reduces output' quality. Furthermore, it's also possible to reduce OpenPose's network resolution, but a significant drawback is reduced accuracy.

Important #4 📌

Place the camera in a way that it can "see" the left and the right wall of the room.

11) Navigate to http://localhost:9021 and create a new topic topic1 with default settings. Now go to /AI_Watch_A1/src/AI_Watch_A1/ and set up your Kafka parameters within the configuration_file.ini file. At this point:

if internal OpenPose execution is chosen, then run the following command in your terminal:

sudo ./AI_Watch_A1.bin --num_gpu 1 --num_gpu_start 2 --image_dir rs_images/rgb --write_json op_output/op --logging_level 255

if external OpenPose execution is chosen, then set up your OpenPose parameters within the conf.conf file and run the following command in your terminal:

sudo ./AI_Watch_A1.bin

Note that the internal OpenPose execution is suggested.

Known issues ⚠️

1) The module can rarely get stuck on the following invocation located in RealSenseD435Manager::23:

rs2::pipeline_profile myPipelineProfile = pipelineStream.start(myConfiguration);

This means some errors have occurred, due to the USB connection while starting the camera's environment. In order to fix that, just exit the program, unplug the RealSense camera from the USB cable, and connect it again.

System Structure 🏛

A1 System Structure

Tools 🛠

Contributing 👨🏻👳🏾‍♂️👩🏼‍🦱👽

In order to contribute to AI Watch A1, please follow the contribution guidelines.

License ☢️

Contacts 🪪

LinkedIn

Citation 📖

@report{AIWatchA1,
    author = {Denny Caruso},
    title = {Multi-person 3D skeleton detection with depth cameras​},
    institution = {University of Naples, Parthenope},
    year = {2022}
}

Supervisor

Dr. Alessio Ferone, CS Assistant Professor at the University of Naples Parthenope.

Other modules

a.a. 2021/2022