Consumer notice when the manufacturer

I am reading a lot of data from ldap that needs to be compared against corresponding entries in the database. To keep the number of SQL queries to a minimum, I want to combine multiple ldap records into one query.

It's all pretty simple: a thread to create ldap results and a thread to use those results and run a SQL query.

ldap_results = Queue.Queue (10)
def producer ():
  for result in ldap_results ():
    ldap_results.put (result)

def consumer ():
  buffer = []
  buffer_size = 5
  while True:
    record = ldap_results.get ()
    buffer.append (record)
    if len (buffer)> = buffer_size:
      do_sql (buffer)
      buffer = []

The problem is, if ldap only returns, say, 3 results, and buffer_size

equals 5, this ends up being blocked forever. I understand that I can add a special character to the buffer, like None

or "EOF"

, but that sounds like bad design: "iterate until you finish, oh, if you don't see this special value, done too."

I came up with two alternative ideas. The first is to have a shared variable eof

, but I don't know how to synchronize it correctly.

def producer ():
  while data:
    buffer.put ()
  eof = True

def consumer ():
  while not eof:
    buffer.get ()

Second, there is a method for the producer ProduceChunks(chunk_size)

and it will handle the reworking of the results, but I don't like that because the producer is supposed to know how best to buffer the results when, in fact, I think it is the responsibility of the consumer.

Does anyone have directions?

+1


source to share


2 answers


I would like to follow the pattern "Run it, get it right, make it fast, make it simple."



Can you implement this correctly without the special "EOF" token? If not, then you just need to use the EOF token, don't sweat it. Yes, the termination condition is more complex, but now it is "correct".

+1


source


The "EOF" approach is quite respectable. Let's take a look at the microcosm of an ANSI string. Zero is EOF. What's wrong with that?

Alternatively, consider the BSTR microcosm. Instead of a trailing zero, the first bytes tell you how the bytes are going.



It's okay anyway.

No matter.

0


source







All Articles