Is v4l2 camera capture with MMAP ring buffer to track application

Question

Is v4l2 camera capture with MMAP ring buffer to track application

I am working on API v4l2 to capture images from a raw sensor on an embedded platform. My capture procedure is linked to example [1] . The suggested streaming method is to use mmaped buffers as a ring buffer.
For initialization, buffers (default = 4 buffers) are requested using an ioctl with the VIDIOC_REQBUFS identifier. They are subsequently queued using VIDIOC_QBUF. The entire streaming procedure is described here [2] . As soon as streaming starts, the driver fills the buffer queues with data. The timestamp of the v4l2_buffer structure indicates the time of the first byte, which in my case results in a time interval of approximately 8.3ms (= 120fps) between buffers. So far so good.
Now, what I expect from a circular buffer is that new captures will automatically overwrite older ones in a circular fashion. But that's not what is happening. Only when the buffer is queued again (VIDIOC_QBUF) after it has been deleted (VIDIOC_DQBUF) and processed (demosaicing, tracking step, ..), a buffer is assigned to the new frame. If I meet the sync condition (processing <8.3ms), I don't get the last captured frame on deletion, but the oldest captured (according to the FIFO), so one of 3x8.3ms before the current one. If the sync condition is not met, the time interval becomes even longer because the buffers are not overwritten.

So, I have a few questions:
1. Does it make sense in this tracking application to have a circular buffer since I don't need a frame history? I doubt it of course, but using the suggested mmap method, in most cases a minimum number of buffers are required per request.
2. Should a separate thread continuously DQBUF and QBUF overwrite the buffer? How can I do that?
3. As a workaround, it would seem to be possible to delete and reload all buffers on every capture, but that doesn't sound like that. Is there someone with a lot of live and streaming experience that can point you to the "right" way? 4. Also I am currently doing a demosaicing step between DQBUF and QBUF and a subsequent tracking step. Should the tracking step also be done before calling QBUF again?

So the main code basically does Capture () and Track () afterwards in a while loop. The Capture procedure looks like this:

cv::Mat v4l2Camera::Capture( size_t timeout ) { 

    fd_set fds;
    FD_ZERO(&fds);
    FD_SET(mFD, &fds);

    struct timeval tv;

    tv.tv_sec  = 0;
    tv.tv_usec = 0;

    const bool threaded = true; //false;

    // proper register settings
    this->format2registerSetting();
    //

    if( timeout > 0 )
    {
        tv.tv_sec  = timeout / 1000;
        tv.tv_usec = (timeout - (tv.tv_sec * 1000)) * 1000;
    }

    //
    const int result = select(mFD + 1, &fds, NULL, NULL, &tv);


    if( result == -1 ) 
    {
        //if (EINTR == errno)
        printf("v4l2 -- select() failed (errno=%i) (%s)\n", errno, strerror(errno));
        return cv::Mat();
    }
    else if( result == 0 )
    {
    if( timeout > 0 )
        printf("v4l2 -- select() timed out...\n");  
        return cv::Mat(); // timeout, not necessarily an error      (TRY_AGAIN)
    }

    // dequeue input buffer from V4L2
    struct v4l2_buffer buf;
    memset(&buf, 0, sizeof(v4l2_buffer));

    buf.type   = V4L2_BUF_TYPE_VIDEO_CAPTURE;
    buf.memory = V4L2_MEMORY_MMAP;  //V4L2_MEMORY_USERPTR;

    if( xioctl(mFD, VIDIOC_DQBUF, &buf) < 0 )
    {
        printf("v4l2 -- ioctl(VIDIOC_DQBUF) failed (errno=%i) (%s)\n", errno, strerror(errno));
        return cv::Mat();
    }

    if( buf.index >= mBufferCountMMap )
    {
        printf("v4l2 -- invalid mmap buffer index (%u)\n", buf.index);
        return cv::Mat();
    }

    // emit ringbuffer entry
    printf("v4l2 -- recieved %ux%u video frame (index=%u)\n", mWidth, mHeight, (uint32_t)buf.index);

    void* image_ptr = mBuffersMMap[buf.index].ptr;

    // frame processing (& tracking step)
    cv::Mat demosaic_mat = demosaic(image_ptr,mSize,mDepth,1);

    // re-queue buffer to V4L2
    if( xioctl(mFD, VIDIOC_QBUF, &buf) < 0 )
        printf("v4l2 -- ioctl(VIDIOC_QBUF) failed (errno=%i) (%s)\n", errno, strerror(errno));

    return demosaic_mat;
}