Consumer notice when the manufacturer

I am reading a lot of data from ldap that needs to be compared against corresponding entries in the database. To keep the number of SQL queries to a minimum, I want to combine multiple ldap records into one query.

It's all pretty simple: a thread to create ldap results and a thread to use those results and run a SQL query.

ldap_results = Queue.Queue (10)
def producer ():
  for result in ldap_results ():
    ldap_results.put (result)

def consumer ():
  buffer = []
  buffer_size = 5
  while True:
    record = ldap_results.get ()
    buffer.append (record)
    if len (buffer)> = buffer_size:
      do_sql (buffer)
      buffer = []

The problem is, if ldap only returns, say, 3 results, and buffer_size

equals 5, this ends up being blocked forever. I understand that I can add a special character to the buffer, like None

or "EOF"

, but that sounds like bad design: "iterate until you finish, oh, if you don't see this special value, done too."

I came up with two alternative ideas. The first is to have a shared variable eof

, but I don't know how to synchronize it correctly.

def producer ():
  while data:
    buffer.put ()
  eof = True

def consumer ():
  while not eof:
    buffer.get ()

Second, there is a method for the producer ProduceChunks(chunk_size)

and it will handle the reworking of the results, but I don't like that because the producer is supposed to know how best to buffer the results when, in fact, I think it is the responsibility of the consumer.

Does anyone have directions?