WinRT Extract frames from video stream
I decode barcodes with the built-in camera, I do it using capElement.Source.CapturePhotoToStreamAsync
to capture photos from preview. it works but freezes the app for a short moment which seems very clumsy and buggy.
As such, I would like it to be in the background, although at least keeping a responsive interface when processing photos.
So far, I have come up with this for capturing a video stream:
private async void ScanInBackground()
{
bool failedScan = true;
var stream = new InMemoryRandomAccessStream();
await capElement.Source.StartRecordToStreamAsync(MediaEncodingProfile.CreateWmv(VideoEncodingQuality.HD1080p), stream);
while(failedScan)
{
Byte[] bytes = await GetBytesFromStream(stream);
//How to split the bytes into frames?
Task.Delay(50);
}
Dispatcher.RunAsync(CoreDispatcherPriority.Low,() => StopCap());
}
and this method to get bytes from stream:
public static async Task<byte[]> GetBytesFromStream(IRandomAccessStream randomStream)
{
var reader = new DataReader(randomStream.GetInputStreamAt(0));
var bytes = new byte[randomStream.Size];
try
{
await reader.LoadAsync((uint)randomStream.Size); reader.ReadBytes(bytes);
}
catch(Exception ex)
{
Logger.LogExceptionAsync(ex, "GetBytesFromStream");
}
return bytes;
}
From the comment in ScanInBackground
you can see that I don't know how to split the stream into photos / frames.
source to share
There is an example on the Microsoft github page that is relevant, although they target Windows 10. You might be interested in porting your project to get this functionality.
GetPreviewFrame : This sample will capture preview frames rather than full-blown photos. When he has a preview frame, he can read and edit the pixels on it.
Here's the relevant part:
private async Task GetPreviewFrameAsSoftwareBitmapAsync()
{
// Get information about the preview
var previewProperties = _mediaCapture.VideoDeviceController.GetMediaStreamProperties(MediaStreamType.VideoPreview) as VideoEncodingProperties;
// Create the video frame to request a SoftwareBitmap preview frame
var videoFrame = new VideoFrame(BitmapPixelFormat.Bgra8, (int)previewProperties.Width, (int)previewProperties.Height);
// Capture the preview frame
using (var currentFrame = await _mediaCapture.GetPreviewFrameAsync(videoFrame))
{
// Collect the resulting frame
SoftwareBitmap previewFrame = currentFrame.SoftwareBitmap;
// Add a simple green filter effect to the SoftwareBitmap
EditPixels(previewFrame);
}
}
private unsafe void EditPixels(SoftwareBitmap bitmap)
{
// Effect is hard-coded to operate on BGRA8 format only
if (bitmap.BitmapPixelFormat == BitmapPixelFormat.Bgra8)
{
// In BGRA8 format, each pixel is defined by 4 bytes
const int BYTES_PER_PIXEL = 4;
using (var buffer = bitmap.LockBuffer(BitmapBufferAccessMode.ReadWrite))
using (var reference = buffer.CreateReference())
{
// Get a pointer to the pixel buffer
byte* data;
uint capacity;
((IMemoryBufferByteAccess)reference).GetBuffer(out data, out capacity);
// Get information about the BitmapBuffer
var desc = buffer.GetPlaneDescription(0);
// Iterate over all pixels
for (uint row = 0; row < desc.Height; row++)
{
for (uint col = 0; col < desc.Width; col++)
{
// Index of the current pixel in the buffer (defined by the next 4 bytes, BGRA8)
var currPixel = desc.StartIndex + desc.Stride * row + BYTES_PER_PIXEL * col;
// Read the current pixel information into b,g,r channels (leave out alpha channel)
var b = data[currPixel + 0]; // Blue
var g = data[currPixel + 1]; // Green
var r = data[currPixel + 2]; // Red
// Boost the green channel, leave the other two untouched
data[currPixel + 0] = b;
data[currPixel + 1] = (byte)Math.Min(g + 80, 255);
data[currPixel + 2] = r;
}
}
}
}
}
And declare this outside of your class:
[ComImport]
[Guid("5b0d3235-4dba-4d44-865e-8f1d0e4fd04d")]
[InterfaceType(ComInterfaceType.InterfaceIsIUnknown)]
unsafe interface IMemoryBufferByteAccess
{
void GetBuffer(out byte* buffer, out uint capacity);
}
And, of course, your project will have to allow unsafe code for all of this.
Take a closer look at the sample to find out how to get all the details. Or, for a walkthrough, you can watch a camera session from a recent // build / conference, which includes a small walkthrough of some camera samples.
source to share
I believe it is necessary to show a media preview and handle various possible exceptions, here is a simple example on how to do this,
let's say you have the following UI, with CaptureElement
to show a preview and a control Image
to show the captured picture,
mc:Ignorable="d" Loaded="MainPage_OnLoaded">
<Grid Background="{StaticResource ApplicationPageBackgroundThemeBrush}" VerticalAlignment="Stretch" HorizontalAlignment="Stretch">
<Grid.ColumnDefinitions>
<ColumnDefinition Width="auto"/>
<ColumnDefinition Width="auto"/>
</Grid.ColumnDefinitions>
<Grid.RowDefinitions>
<RowDefinition Height="auto"/>
<RowDefinition Height="auto"/>
</Grid.RowDefinitions>
<CaptureElement x:Name="PreviewElement" Width="400" Height="400" Grid.Column="0" Grid.Row="0"/>
<Image x:Name="ImageElement" Width="400" Height="400" Grid.Column="1" Grid.Row="0"/>
<Button Click="TakePhoto_Click" Content="Take Photo" Grid.Column="0" Grid.Row="1" Grid.ColumnSpan="2" HorizontalAlignment="Stretch" />
</Grid>
In the code behind, declare the mediaCapture field,
private MediaCapture _mediaCapture;
then in the page loaded event handler you need
- start the media recorder,
- install multiple possible exception handlers,
-
and start the camera preview
private async void MainPage_OnLoaded(object sender, RoutedEventArgs e) { //Start the device try { _mediaCapture = new MediaCapture(); _mediaCapture.RecordLimitationExceeded += MediaCapture_RecordLimitationExceeded; _mediaCapture.Failed += MediaCapture_Failed; await _mediaCapture.InitializeAsync(); } catch (UnauthorizedAccessException ex) { (new MessageDialog("Set the permission to use the webcam")).ShowAsync(); } catch (Exception ex) { (new MessageDialog("Can't initialize the webcam !")).ShowAsync(); } //Start the preview if (_mediaCapture != null) { try { PreviewElement.Source = _mediaCapture; await _mediaCapture.StartPreviewAsync(); } catch (Exception ex) { (new MessageDialog("Something went wrong !")).ShowAsync(); } } } private async void MediaCapture_Failed(MediaCapture sender, MediaCaptureFailedEventArgs errorEventArgs) { await Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, () => (new MessageDialog("Media capture failed")).ShowAsync()); } private async void MediaCapture_RecordLimitationExceeded(MediaCapture sender) { await _mediaCapture.StopRecordAsync(); await Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, () => (new MessageDialog("Record limitation exceeded")).ShowAsync()); }
And finally here's how to take the shot correctly, each thing is asynchronous so there is no lag or at all
private async void TakePhoto_Click(object sender, RoutedEventArgs e)
{
if (_mediaCapture != null)
{
try
{
ImageEncodingProperties encodingProperties = ImageEncodingProperties.CreateJpeg();
WriteableBitmap bitmap = new WriteableBitmap((int)ImageElement.Width, (int)ImageElement.Height);
using (var imageStream = new InMemoryRandomAccessStream())
{
await this._mediaCapture.CapturePhotoToStreamAsync(encodingProperties, imageStream);
await imageStream.FlushAsync();
imageStream.Seek(0);
bitmap.SetSource(imageStream);
await Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal,
() =>
{
ImageElement.Source = bitmap;
});
}
}
catch (Exception ex)
{
(new MessageDialog("Something went wrong !")).ShowAsync();
}
}
}
source to share