Generate mipmaped MTLTextures from camera AVFoundation stream
I am using AVFoundation to get a camera stream. I am using this code to get MTLTextures:
- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
{
CVPixelBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
id<MTLTexture> texture = nil;
{
size_t width = CVPixelBufferGetWidth(pixelBuffer);
size_t height = CVPixelBufferGetHeight(pixelBuffer);
MTLPixelFormat pixelFormat = MTLPixelFormatBGRA8Unorm;
CVMetalTextureRef metalTextureRef = NULL;
CVReturn status = CVMetalTextureCacheCreateTextureFromImage(NULL, _textureCache, pixelBuffer, NULL, pixelFormat, width, height, 0, &metalTextureRef);
if(status == kCVReturnSuccess)
{
texture = CVMetalTextureGetTexture(metalTextureRef);
if (self.delegate){
[self.delegate textureUpdated:texture];
}
CFRelease(metalTextureRef);
}
}
}
It works great, except that the generated MTLTexture object is not mipmaped (only has one mip level). In this call:
CVMetalTextureCacheCreateTextureFromImage(NULL, _textureCache, pixelBuffer, NULL, pixelFormat, width, height, 0, &metalTextureRef);
There is a third parameter called " textureAtributes ", I think it can be specified that I want a mipmaped texture, but I couldn't find a word in the documentation as to what exactly is going on there. I also haven't found any source code that passes something else instead of NULL.
There is a similar method in OpenGLES for iOS with the same parameter and also without words in the documentation.
source to share
Just got a response from the Metal Engineer here . Here's a quote:
No, it is not possible to create a mipmapped texture from CVPixelBuffer directly.
CVPixelBuffer images usually have a linear / strategic layout as non-GPU hardware units can interact with them, and most GPU hardware only supports mipmapping from textured textures. You will need to issue blit to copy from linear MTLTexture to private MTLTexture of your own creation, then create mipmaps.
As for textureAttributes , only one key is supported:kCVMetalTextureCacheMaximumTextureAgeKey
source to share
There is no way to get the texture mipmapped directly, but you can create it yourself quite easily.
First, use your metal device to create an empty metal texture that is the same size and format as the existing texture, but has a full mipmap chain. See the newTexture documentation
Use the MTLCommandBuffer object to create the blitEncoder object. See blitCommandEncoder documentation
Use blitEncoder to copy from your camera texture to an empty texture. destinationLevel should be zero as you are only copying the top level mipmap. See copyFromTexture documentation
Finally, use blitEncoder to generate all mip levels by calling generateMipmapsForTexture. See generateMipMapsForTexture documentation
At the end of this you have a metal texture from the camera with a full mip chain.
source to share