What is the best way to record video with augmented reality

What's the best way to record augmented reality videos? (add text, image logos to frames from iPhone / iPad)

I used to try to figure out how to draw CIImage

( How do I draw text in CIImage? ) And convert CIImage

back to CMSampleBuffer

( CIImage back to CMSampleBuffer )

I did almost everything, only the problem with recording video using new CMSampleBuffer

inAVAssetWriterInput

But this solution is not very good anyway, it eats a lot of CPU when converting CIImage

to CVPixelBuffer

( ciContext.render(ciImage!, to: aBuffer)

)

So, I want to stop here and find other ways to record video with augmented reality (for example, dynamically add (draw) text inside frames when encoding video to mp4 file)

Here's what I tried and don't want to use anymore ...

// convert original CMSampleBuffer to CIImage, 
// combine multiple `CIImage`s into one (adding augmented reality -  
// text or some additional images)
let pixelBuffer: CVPixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)!
let ciimage : CIImage = CIImage(cvPixelBuffer: pixelBuffer)
var outputImage: CIImage?
let images : Array<CIImage> = [ciimage, ciimageSec!] // add all your CIImages that you'd like to combine
for image in images {
    outputImage = outputImage == nil ? image : image.composited(over: outputImage!)
}

// allocate this class variable once         
if pixelBufferNew == nil {
    CVPixelBufferCreate(kCFAllocatorSystemDefault, CVPixelBufferGetWidth(pixelBuffer),  CVPixelBufferGetHeight(pixelBuffer), kCVPixelFormatType_32BGRA, nil, &pixelBufferNew)
}

// convert CIImage to CVPixelBuffer
let ciContext = CIContext(options: nil)
if let aBuffer = pixelBufferNew {
    ciContext.render(outputImage!, to: aBuffer) // >>> IT EATS A LOT OF <<< CPU
}

// convert new CVPixelBuffer to new CMSampleBuffer
var sampleTime = CMSampleTimingInfo()
sampleTime.duration = CMSampleBufferGetDuration(sampleBuffer)
sampleTime.presentationTimeStamp = CMSampleBufferGetPresentationTimeStamp(sampleBuffer)
sampleTime.decodeTimeStamp = CMSampleBufferGetDecodeTimeStamp(sampleBuffer)
var videoInfo: CMVideoFormatDescription? = nil
CMVideoFormatDescriptionCreateForImageBuffer(kCFAllocatorDefault, pixelBufferNew!, &videoInfo)
var oBuf: CMSampleBuffer?
CMSampleBufferCreateForImageBuffer(kCFAllocatorDefault, pixelBufferNew!, true, nil, nil, videoInfo!, &sampleTime, &oBuf)

/*
try to append new CMSampleBuffer into a file (.mp4) using 
AVAssetWriter & AVAssetWriterInput... (I met errors with it, original buffer works ok 
- "from func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection)")
*/*

      

Is there a better solution?

+2


source to share


1 answer


now i answer my own question

the best would be to use Objective-C++

class ( .mm

) where we can use OpenCV and convert easily / quickly from CMSampleBuffer

to cv::Mat

and back to CMSampleBuffer

after processing



we can easily call Objective-C ++ functions from Swift

0


source







All Articles