OpenCV_Manager Class Reference

OpenCV_Manager class is used for general-purpose tasks on the captured frames. More...

#include <OpenCV_Manager.hpp>

Public Member Functions
virtual	~OpenCV_Manager ()=default
	Destroy the OpenCV_Manager object. More...
cv::Mat	realsenseFrameToMat (const rs2::frame &singleFrame)
	This method converts a frame of type rs2::frame to cv::Mat. More...
void	getVideoFramesCV (unsigned int user_nFrame, rs2::pipeline &pipelineStream, float scale, const unsigned short int framesToSkip)
	Get user_nFrame video frames from the pipeline by applying a specific scaling factor. More...
void	showSkeletonsCV (unsigned int user_nFrame, const float skeletonThreshold)
	Retrieve OpenPose's output, convert it to RealSense coordinate's space, show results and save them. More...

Detailed Description

OpenCV_Manager class is used for general-purpose tasks on the captured frames.

OpenCV_Manager class is a class that is responsible for converting a rs2::frame to a cv::Mat, retrieving video frames from the camera and handling them, showing the skeletons, producing related output, and saving results.

Definition at line 23 of file OpenCV_Manager.hpp.

Constructor & Destructor Documentation

~OpenCV_Manager()

virtual OpenCV_Manager::~OpenCV_Manager ( )

virtualdefault

Destroy the OpenCV_Manager object.

Member Function Documentation

getVideoFramesCV()

void OpenCV_Manager::getVideoFramesCV	(	unsigned int	user_nFrame,
		rs2::pipeline &	pipelineStream,
		float	scale,
		const unsigned short int	framesToSkip
	)

Get user_nFrame video frames from the pipeline by applying a specific scaling factor.

Parameters

user_nFrame	frame's number to capture according to user choice.
pipelineStream	The pipeline simplifies the user interaction with the device and computer vision processing modules. The class abstracts the camera configuration and streaming, and the vision modules triggering and threading. It lets the application focus on the computer vision output of the modules, or the device output data. The pipeline can manage computer vision modules, which are implemented as a processing block. The pipeline is the consumer of the processing block interface, while the application consumes the computer vision interface.
scale	Scaling factor.
framesToSkip	Frames' number to skip in order to ignore a certain amount of frames and extend the recording interval. A '0' value means 'capture each frame'.

See also

https://dev.intelrealsense.com/docs/docs-get-started

getVideoFrames(unsigned int user_nFrame, rs2::pipeline & pipelineStream, float scale)

getVideoFramesRS(unsigned int user_nFrame, rs2::pipeline & pipelineStream, rs2::depth_frame & depthFrame, rs2::frame & colorFrame, rs2::frame & colorizedDepthFrame)

Definition at line 37 of file OpenCV_Manager.cpp.

                                                                                                                                                 {
    UsageManager * usageManagerInstance = UsageManager::getInstance();
    if (usageManagerInstance == nullptr) CV_Error(USAGE_MANAGER_NULLPTR_ERROR, USAGE_MANAGER_NULLPTR_SCOPE);
    char ** argv = * usageManagerInstance->get_argv();
    const char * imagesFolder = argv[imagesFolderOffset];
    FacadeSingleton * facadeSingletonInstance = FacadeSingleton::getInstance();
    if (facadeSingletonInstance == nullptr) CV_Error(FACADE_SINGLETON_NULLPTR_ERROR, FACADE_SINGLETON_NULLPTR_SCOPE);
    RealSenseManager * cameraManager = facadeSingletonInstance->getCameraManager();
    ImageManager * imageManager = facadeSingletonInstance->getImageManager();
    
    // Do this until we don't capture, convert, scale, compute and save user_nFrame frame
    for (unsigned int nFrame = 0; nFrame < user_nFrame; nFrame++) {
        // Capture
        unsigned int frameID = cameraManager->getFrameID();
        rs2::depth_frame depthFrame = rs2::depth_frame(rs2::frame());
        rs2::frame colorFrame, colorizedDepthFrame;
        cameraManager->getVideoFramesRS(user_nFrame, pipelineStream, depthFrame, colorFrame, colorizedDepthFrame, framesToSkip);
        int cols = depthFrame.get_width(), rows = depthFrame.get_height();
        // Convert
        cv::Mat colorImage = realsenseFrameToMat(colorFrame), distanceImage = cv::Mat::zeros(rows, cols, CV_32FC1);
        cv::Mat depthImage = realsenseFrameToMat(depthFrame), colorizedDepthImage = realsenseFrameToMat(colorizedDepthFrame);
        // Scale
        depthImage *= 1000.0 * scale;
        // Compute distance
        for (int i = 0; i < cols; i++) for (int j = 0; j < rows; j++) distanceImage.at <float> (j, i) = (float) depthFrame.get_distance(i, j);
 
        std::stringstream colorImagePath, distanceImagePath, colorizedDepthImagePath;
        colorImagePath <<           imagesFolder << "rgb/" <<   frameID << "_Color.png";
        distanceImagePath <<        imagesFolder << "d/" <<     frameID << "_Distance.exr";
        colorizedDepthImagePath <<  imagesFolder << "depth/" << frameID << "_Depth.png";
        // Save
        imageManager->saveImages( { colorImage, distanceImage, colorizedDepthImage },
                                 { colorImagePath.str(), distanceImagePath.str(), colorizedDepthImagePath.str() } );
        imageManager->releaseImages( { colorImage, depthImage, distanceImage, colorizedDepthImage } );
    }
}

References FACADE_SINGLETON_NULLPTR_ERROR, FACADE_SINGLETON_NULLPTR_SCOPE, UsageManager::get_argv(), FacadeSingleton::getCameraManager(), RealSenseManager::getFrameID(), FacadeSingleton::getImageManager(), FacadeSingleton::getInstance(), UsageManager::getInstance(), RealSenseManager::getVideoFramesRS(), imagesFolderOffset, realsenseFrameToMat(), ImageManager::releaseImages(), ImageManager::saveImages(), USAGE_MANAGER_NULLPTR_ERROR, and USAGE_MANAGER_NULLPTR_SCOPE.

Referenced by FacadeSingleton::getVideoFrames().

realsenseFrameToMat()

cv::Mat OpenCV_Manager::realsenseFrameToMat

(

const rs2::frame &

singleFrame

)

This method converts a frame of type rs2::frame to cv::Mat.

Parameters

singleFrame

A frame of type rs2::frame.

Returns

cv::Mat

Definition at line 15 of file OpenCV_Manager.cpp.

                                                                       {
    rs2::video_frame videoFrame = singleFrame.as<rs2::video_frame>();
    const int frameWidth = videoFrame.get_width(), frameHeight = videoFrame.get_height();
    
    switch (singleFrame.get_profile().format()) {
        case RS2_FORMAT_BGR8:
            return cv::Mat(cv::Size(frameWidth, frameHeight), CV_8UC3, (void *) singleFrame.get_data(), cv::Mat::AUTO_STEP);
        case RS2_FORMAT_RGB8: {
            cv::Mat rgbFrame = cv::Mat(cv::Size(frameWidth, frameHeight), CV_8UC3, (void *) singleFrame.get_data(), cv::Mat::AUTO_STEP);
            cv::cvtColor(rgbFrame, rgbFrame, cv::COLOR_RGB2BGR);
            return rgbFrame;
        }
        case RS2_FORMAT_Z16:
            return cv::Mat(cv::Size(frameWidth, frameHeight), CV_16UC1, (void *) singleFrame.get_data(), cv::Mat::AUTO_STEP);
        case RS2_FORMAT_Y8:
            return cv::Mat(cv::Size(frameWidth, frameHeight), CV_8UC1, (void *) singleFrame.get_data(), cv::Mat::AUTO_STEP);
        default:
            throw std::runtime_error("Frame format is not supported yet!");
            break;
    }
}

Referenced by getVideoFramesCV().

showSkeletonsCV()

void OpenCV_Manager::showSkeletonsCV	(	unsigned int	user_nFrame,
		const float	skeletonThreshold
	)

Retrieve OpenPose's output, convert it to RealSense coordinate's space, show results and save them.

Parameters

user_nFrame	frame's number to capture according to user choice.
skeletonThreshold	The magnitude or intensity that must be exceeded for a specific reaction, phenomenon, result, or condition to occur or be manifested. In this case, if the skeleton's confidence mean value is greater than "skeletonThreshold", then the skeleton will be considered a meaningful skeleton.

See also

showSkeletons(unsigned int user_nFrame)

Definition at line 74 of file OpenCV_Manager.cpp.

                                                                                             {
    FacadeSingleton * facadeSingletonInstance = FacadeSingleton::getInstance();
    if (facadeSingletonInstance == nullptr) CV_Error(FACADE_SINGLETON_NULLPTR_ERROR, FACADE_SINGLETON_NULLPTR_SCOPE);
    RealSenseManager * cameraManager = facadeSingletonInstance->getCameraManager();
    OutputManagerJSON * outputManagerJSON = (OutputManagerJSON *) facadeSingletonInstance->getOutputManager();
    UsageManager * usageManagerInstance = facadeSingletonInstance->getUsageManager();
    if (usageManagerInstance == nullptr) CV_Error(USAGE_MANAGER_NULLPTR_ERROR, USAGE_MANAGER_NULLPTR_SCOPE);
    ImageManager * imageManager = facadeSingletonInstance->getImageManager();
    char ** argv = * usageManagerInstance->get_argv(), * imagesFolder = argv[imagesFolderOffset], * outputFolder = argv[outputFolderOffset];
    unsigned int frameID = cameraManager->getFrameID(), currentImageID;
    
    Json::Value currentJSON;
    
    // Do this until we don't iterate on user_nFrame frame
    for (unsigned int nFrame = 0; nFrame < user_nFrame; nFrame++) {
        std::stringstream inputJsonFilePath, skeletonImagePath, colorImagePath, distanceImagePath, colorizedDepthImagePath, skeletonOnlyImagePath, outputJsonFilePath;
        currentImageID = frameID - user_nFrame + nFrame;
        
        // Internal Approach OpenPose
//        inputJsonFilePath << outputFolder << "op/" << nFrame << "_keypoints.json";
        // External Approach OpenPose
        inputJsonFilePath << outputFolder << "op/" << currentImageID << "_Color_keypoints.json";
        
        // Try to load OpenPose's output JSON file
        if (outputManagerJSON->loadJSON(inputJsonFilePath.str(), currentJSON)) {
            Json::Value people = outputManagerJSON->getValueAt("people", currentJSON);
            colorImagePath << imagesFolder << "rgb/" << currentImageID << "_Color.png";
            distanceImagePath << imagesFolder << "d/" << currentImageID << "_Distance.exr";
            colorizedDepthImagePath << imagesFolder << "depth/" << currentImageID << "_Depth.png";
            
            // Load images captured previously from camera
            cv::Mat colorImage, colorizedDepthImage;
            imageManager->loadImage(colorImagePath.str(), cv::IMREAD_COLOR, colorImage);
            cv::Mat distanceImage = cv::Mat(colorImage.rows, colorImage.cols, CV_32FC1);
            imageManager->loadImage(distanceImagePath.str(), cv::IMREAD_ANYCOLOR | cv::IMREAD_ANYDEPTH, distanceImage);
            imageManager->loadImage(colorizedDepthImagePath.str(), cv::IMREAD_COLOR, colorizedDepthImage);
            // Note: Comment following line for better performance
            imageManager->showImages( { colorImage, colorizedDepthImage }, { "Frame No Skeleton", "Frame Colorized Depth" } );
            cv::Mat skeletonOnlyImage = cv::Mat::zeros(colorImage.rows, colorImage.cols, colorImage.type());
            
            // Produce a JSON output file for people present in this frame
            outputManagerJSON->createJSON(people, colorImage, distanceImage, skeletonOnlyImage, currentImageID, outputFolder, skeletonThreshold);
            
            // Show images and save them. Note: Comment following line for better performance
            imageManager->showImages( { skeletonOnlyImage, colorImage }, { "Frame Skeleton Background Cut", "Frame Skeleton" } );
            skeletonOnlyImagePath << imagesFolder << "sk/" << currentImageID << "_sk.png";
            skeletonImagePath << imagesFolder << "skeleton/" << currentImageID << "_Skeleton.png";
            imageManager->saveImages( { colorImage, skeletonOnlyImage }, { skeletonImagePath.str(), skeletonOnlyImagePath.str() } );
            imageManager->releaseImages( { colorImage, distanceImage, colorizedDepthImage, skeletonOnlyImage } );
        }
    }
}

References OutputManagerJSON::createJSON(), FACADE_SINGLETON_NULLPTR_ERROR, FACADE_SINGLETON_NULLPTR_SCOPE, UsageManager::get_argv(), FacadeSingleton::getCameraManager(), RealSenseManager::getFrameID(), FacadeSingleton::getImageManager(), FacadeSingleton::getInstance(), FacadeSingleton::getOutputManager(), FacadeSingleton::getUsageManager(), OutputManagerJSON::getValueAt(), imagesFolderOffset, ImageManager::loadImage(), OutputManagerJSON::loadJSON(), outputFolderOffset, ImageManager::releaseImages(), ImageManager::saveImages(), ImageManager::showImages(), USAGE_MANAGER_NULLPTR_ERROR, and USAGE_MANAGER_NULLPTR_SCOPE.

Referenced by FacadeSingleton::showSkeletons().

---

The documentation for this class was generated from the following files:

• Managers/OpenCV_Manager.hpp
• Managers/OpenCV_Manager.cpp