How do I apply image effects like the stone detection stream in a Windows 8 app?

I just played around a bit in this area last week and even thought about writing a blog post about it. I think this answer could be just as good.

You can go for the MFT path that needs to be done in C ++, but what you would need to write won't be much different between C # and C ++. The only thing to note is that I think MFT works in YUV color space, so your typical convolution filters / effects may behave differently or need to be converted to RGB. If you decide to go this route. On the C # application side, the only thing you need to do is call MediaCapture.AddEffectAsync (). Well, and you need to edit your Package.appxmanifest etc, but let's go first things first.

If you watch media capture using the webcam example - it already does what you need it to do. It applies a grayscale effect to the camera channel. It includes a C ++ MFT project that is used in an application available in the C # version. I had to apply an effect to the MediaElement, which may not be what you want, but just as simple - call MediaElement.AddVideoEffect () and now your video playback will apply the grayscale effect. To be able to use MFT - you just need to add a link to the GrayscaleTransform project and add the following lines to your appxmanifest:

<Extensions>
  <Extension Category="windows.activatableClass.inProcessServer">
    <InProcessServer>
      <Path>GrayscaleTransform.dll</Path>
      <ActivatableClass ActivatableClassId="GrayscaleTransform.GrayscaleEffect" ThreadingModel="both" />
    </InProcessServer>
  </Extension>
</Extensions>

      

        
        
        
      

    

How the MFT code works:

The following lines create a pixel color transformation matrix

float scale = (float)MFGetAttributeDouble(m_pAttributes, MFT_GRAYSCALE_SATURATION, 0.0f);
float angle = (float)MFGetAttributeDouble(m_pAttributes, MFT_GRAYSCALE_CHROMA_ROTATION, 0.0f);
m_transform = D2D1::Matrix3x2F::Scale(scale, scale) * D2D1::Matrix3x2F::Rotation(angle);

      

        
        
        
      

    

Depending on the pixel format of the video feed, a different conversion method is selected for scanning pixels. Find these lines:

m_pTransformFn = TransformImage_YUY2;
m_pTransformFn = TransformImage_UYVY;
m_pTransformFn = TransformImage_NV12;

      

        
        
        
      

    

For my sample m4v file - the format is defined as NV12, so it calls TransformImage_NV12.

For pixels in the specified range (m_rcDest) or for the entire screen if no range is specified, the TransformImage_ ~ methods call TransformChroma (mat, & u, & v). For other pixels, the values ​​from the original frame are copied.

TransformChroma transforms pixels using m_transform. If you want to change the effect - you can simply change the m_transform matrix or if you need access to neighboring pixels, as in the edge detection filter - change the TransformImage_ methods to handle those pixels.

This is one way to do it. I think this is quite CPU intensive, so I prefer writing pixel shaders for operations like this. How do you apply a pixel shader to a video stream? Well, I'm not quite there yet, but I believe you can transfer video snippets to the DirectX surface quite easily and call them a pixel shader later.I've managed to transfer video snippets so far and I hope to apply the shaders next week. I could write a blog post about this. I took the meplayer class from the original C ++ playback sample for the media environmentand moved it to a C ++ DirectX project converted to WinRTComponent library and then used it with a C # / XAML application, linking the swapchain generated by the meplayer class to the SwapChainBackgroundPanel that I use in the C # project to display the video. I had to make a few changes to the meplayer class. First, I had to move it to a public namespace which would make it available to another assembly. Then I had to modify the swapchain it creates into the format I would use with SwapChainBackgroundPanel:

        DXGI_SWAP_CHAIN_DESC1 swapChainDesc = {0};
        swapChainDesc.Width = m_rcTarget.right;
        swapChainDesc.Height = m_rcTarget.bottom;
        // Most common swapchain format is DXGI_FORMAT_R8G8B8A8-UNORM
        swapChainDesc.Format = m_d3dFormat;
        swapChainDesc.Stereo = false;

        // Don't use Multi-sampling
        swapChainDesc.SampleDesc.Count = 1;
        swapChainDesc.SampleDesc.Quality = 0;

        //swapChainDesc.BufferUsage = DXGI_USAGE_BACK_BUFFER | DXGI_USAGE_RENDER_TARGET_OUTPUT;
        swapChainDesc.BufferUsage = DXGI_USAGE_RENDER_TARGET_OUTPUT; // Allow it to be used as a render target.
        // Use more than 1 buffer to enable Flip effect.
        //swapChainDesc.BufferCount = 4;
        swapChainDesc.BufferCount = 2;
        //swapChainDesc.Scaling = DXGI_SCALING_NONE;
        swapChainDesc.Scaling = DXGI_SCALING_STRETCH;
        swapChainDesc.SwapEffect = DXGI_SWAP_EFFECT_FLIP_SEQUENTIAL;
        swapChainDesc.Flags = 0;

      

        
        
        
      

    

Finally - instead of calling CreateSwapChainForCoreWindow - I call CreateSwapChainForComposition and link the swapchain to my SwapChainBackgroundPanel:

        // Create the swap chain and then associate it with the SwapChainBackgroundPanel.
        DX::ThrowIfFailed(
            spDXGIFactory.Get()->CreateSwapChainForComposition(
                spDevice.Get(),
                &swapChainDesc,
                nullptr,                                // allow on all displays
                &m_spDX11SwapChain)
            );

        ComPtr<ISwapChainBackgroundPanelNative> dxRootPanelAsSwapChainBackgroundPanel;

        // Set the swap chain on the SwapChainBackgroundPanel.
        reinterpret_cast<IUnknown*>(m_swapChainPanel)->QueryInterface(
            IID_PPV_ARGS(&dxRootPanelAsSwapChainBackgroundPanel)
            );

        DX::ThrowIfFailed(
            dxRootPanelAsSwapChainBackgroundPanel->SetSwapChain(m_spDX11SwapChain.Get())
            );

      

        
        
        
      

    

* EDIT follows

Forgot one more thing. If your goal is to stay pure C # - if you figure out how to capture frames in a WriteableBitmap (perhaps by calling MediaCapture.CapturePhotoToStreamAsync () with a MemoryStream and then calling WriteableBitmap. SetSource () on the stream) - you can use WriteableBitmapEx to handle your images. This may not be the best performance, but if your resolution is not too high or your frame rate requirements are low, it may be enough. The CodePlex project does not officially support WinRT, but I have a version that should work and you can try it here (Dropbox) .