Clever Geek Handbook

How to run the program and also execute the code when data is received from the network?

I wrote a small Python program that basically does the following:

  • Receives a hot word as input from the user. If it matches the set keyword, it continues.
  • After entering the correct hot word, the user is prompted to enter a command.
  • After reading the command, the program checks the command file to see if there is a command that matches that input
  • If a command is found, follow the steps for that command.

I would like to add the ability to execute commands over the network like this (and learn how to use Twisted on the way):

  • Customer # 1 enters a command targeting Customer # 2.
  • The command is sent to the server which sends it to client # 2.
  • Client # 2 receives the command and executes it if it is valid.

Note. Entering commands locally (using the code below) and remotely should be possible.

After some thought, I couldn't find any other way to implement this other than:

  • Execute the above program as process # 1 (a program executed locally, as I wrote at the beginning).
  • The twisted client will run as process # 2 and receive commands from remote clients. Whenever a command is received, the Twisted client will initialize a thread that will parse the command, check its validity, and execute it if it is valid.

Since I don't have much experience with threads and I don't have network programming, I couldn't think of any other scheme that makes sense to me.

Is the above diagram overly complex? I would appreciate your understanding before trying to implement it this way.

Code for python program (no client):

The main one (this is the start () method):

class Controller:
    def __init__(self,listener, executor):
        self.listener = listener
        self.executor = executor


    def start(self):
        while True:
            text = self.listener.listen_for_hotword()

            if self.executor.is_hotword(text):
                text = self.listener.listen_for_command()
                if self.executor.has_matching_command(text):
                    self.executor.execute_command(text)
                else:
                    tts.say("No command found. Please try again")

Listener (receives input from user):

class TextListener(Listener):
    def listen_for_hotword(self):
        text = raw_input("Hotword: ")
        text =' '.join(text.split()).lower()
        return text

    def listen_for_command(self):
        text = raw_input("What would you like me to do: ")
        text = ' '.join(text.split()).lower()
        return text

Executor (class executing the given command):

class Executor:
    #TODO: Define default path
    def __init__(self,parser, audio_path='../Misc/audio.wav'):
        self.command_parser = parser
        self.audio_path = audio_path

    def is_hotword(self,hotword):
        return self.command_parser.is_hotword(hotword)

    def has_matching_command(self,command):
        return self.command_parser.has_matching_command(command)

    def execute_command(self,command):
        val = self.command_parser.getCommand(command)
        print val
        val = os.system(val) #So we don't see the return value of the command

Batch File Parser:

class KeyNotFoundException(Exception):
    pass


class YAMLParser:
    THRESHOLD = 0.6

    def __init__(self,path='Configurations/commands.yaml'):
        with open(path,'r') as f:
            self.parsed_yaml = yaml.load(f)

    def getCommand(self,key):
        try:
            matching_command = self.find_matching_command(key)
            return self.parsed_yaml["Commands"][matching_command]
        except KeyError:
            raise KeyNotFoundException("No key matching {}".format(key))

    def has_matching_command(self,key):
        try:
            for command in self.parsed_yaml["Commands"]:
                if jellyfish.jaro_distance(command,key) >=self.THRESHOLD:
                    return True
        except KeyError:
            return False

    def find_matching_command(self,key):
        for command in self.parsed_yaml["Commands"]:
            if jellyfish.jaro_distance(command,key) >=0.5:
                return command

    def is_hotword(self,hotword):
        return jellyfish.jaro_distance(self.parsed_yaml["Hotword"],hotword)>=self.THRESHOLD

Sample config file:

Commands:
  echo : echo hello


Hotword: start
0


source to share


1 answer


It is very difficult for me to keep track of the background in your questions, but I will take on the questions myself.

How to run the program and also execute the code when data is received from the network?

As you noted in your question, the typical way to write a walk-and-gum-style application is to code in a thread or style loop.

Considering you're talking about streaming and Twisted (which is the style of the event loop), I'm worried that you might be thinking about mixing the two.

I see them as fundamentally different styles of programming (each with places they excel), and that mixing them up is usually a path to hell.

Let me give you some examples:

Background: Threading vs Event programming

Threading

  • How to think about the concept:

    I have several things that I need to do at the same time, and I want my operating system to determine how and when to perform these individual tasks.

  • Advantages:

    • 'A way to allow one program to use multiple processor cores at the same time

      In the posix world, the only way to allow one process to run on multiple processor cores at the same time is through threads (while the typical ideal number of threads is no more than the cores in a given machine)

    • It's easier to start with

      The same code you used in line can be pushed onto the stream, usually without the need for a redesign (without the GIL, some locking would be required, but more on that later)

    • Much easier to use with tasks that will use up all the CPU you can give them.

      those. in most cases math is easier to deal with using streaming solutions and then using event / async frameworks

  • Disadvantages:

    • Python has a special problem with threads

      In CPython, locking the global interpreter (GIL) can deny the ability of threads to multitask (making threads nearly useless). Avoiding the GIL is messy and can reverse all of the ease of use in threads

    • As you add threads (and locks) things get tricky quickly, see this SO: Threading Guidelines

    • Rarely optimal for IO / user-related tasks

      While threads are very good at handling a small number of tasks that want to use a lot of CPU (ideally one thread per core), they are much less optimal when dealing with a large number of tasks that spend most of the waiting time.

  • Best use:

    Computationally expensive things.

    If you have large chucks of math that you want to run concurrently, it is very unlikely that you will be able to plan your CPU usage more intelligently than your operating system.

    (Given the GIL CPythons issue, streams should not be used manually for math, instead, a library that uses internal streams inside should be used (e.g. NumPy))

Event-loop / asynchronous programming

  • How to think about the concept:

    I have several things that I need to do at the same time, but I (the programmer) want direct control / direct implications as to how and when my subtasks are executed

  • How you should think about your code:

    Think of all your subtasks in one big intertwined whole, your mind should always think about "will this code run fast enough that it won't do the other subtasks I'm managing"

  • Advantages:

    • Extremely efficient with network / IO / UI connectivity, including high connectivity

      Event loop style programs are one of the key technologies that c10k has solved . Structures like Twisted can literally handle tens of thousands of connections in a single python process running on a small machine.

    • Predictable (slight) increase in complexity as more connections / tasks are added

      While there is a pretty steep learning curve (especially the twisted one), once you understand the basics, new connection types can be added to projects with minimal increase in overall complexity. Moving from a program that offers a keyboard interface to one that offers keyboard / telnet / web / ssl / ssh connections can be just a few lines of code for each interface (... this is highly framework dependent, but no matter the Complexity the event-event loop is more predictable than threads as the number of connections increases.

  • Disadvantages:

    • Harder to get started.

      Event / asynchronous programming requires a different design style from the first line of code (although you will find that the design style carries over somewhat from language to language)

    • One event loop, one core

      While event loops can handle an impressive number of IO connections at the same time, they cannot run on multiple cores at the same time. The usual way to deal with this is to write programs in such a way that you can run multiple instances of the program at the same time, one for each core (this method bypasses the GIL issue).

    • Multiplexing large CPU tasks can be difficult

      Event programming requires carving large CPU tasks into chunks so that each chunk takes up (ideally predictably) a small amount of CPU, otherwise the event system stops multitasking whenever a high CPU task is running.

  • Best use:

    IO-based stuff

TL; DR - Considering your question, look at twisted and non-nonsense threads



While your application does not seem to be purely I / O based, none of them appear to be processor based (it looks like your playing sound is currently using a call to system

system

shut down an independent process every time its like this that its running does not burn your processes.CPU - although system

it does block, so its no-no twisted - you need to use different calls twisted.)

Nor does your question suggest that you are interested in making the most of multiple cores.

So given that you specifically talked about Twisted, and the event loop solution seems to be the best for your application, I would recommend looking at Twisted and -not-threads.

Given the "Best Uses" listed above, you might be tempted to think mixing curls and streams is the way to get it, but if you are even slightly wrong, you disable the benefits of both -loop events (you block) and threads (the GIL will not allow thread multitasking) and has something super-complicated that doesn't give you any advantage.

Is the above circuit overly complex? I would appreciate your understanding before trying to implement it this way.

The "schema" you gave:

After some thought, I couldn't find any other way to implement this other than:

  • Execute the above program as process # 1 (a program executed locally, as I wrote at the beginning).
  • The twisted client will run as process # 2 and receive commands from remote clients. Whenever a command is received, the Twisted client will initialize a thread that will parse the command, check its validity, and execute it if it is valid.

Since I don't have that much experience with threads, and I don't have network programming, I couldn't think of any other scheme that makes sense to me.

In response to "Is the circuit ... too complex," I would almost certainly say because you're talking about twisted and threading. (see above tr; dr)

Given my, of course, incomplete (and confusing) understanding of what you want to build, I would guess that the solution would be twisted for you:

  • Take a close look at the krondo Twisted Introduction (it really does require keeping track of a line of code for a line, but if you run its AWESOME a learning tool for twisting and programming events in general)
  • Right from the start, you rewrite your hot word in twisted, using what you learned in the krondo manual, starting with providing whatever interface you currently have (keyboard?).
  • Add other communication interfaces to this code (telnet, web, etc.) that will allow you to access the processing code you wrote for the keyboard interface (?).

If, as you state in your question, you really need a server, you can write a second twisted program to provide this (you will see examples of everything in the krondo manual). Although I'm guessing, when you understand the twisted library support, you will realize that you don't need to create any additional servers that you can simply include depending on what protocols you need in your base code.

+4


source






More articles:


All Articles