Clever Geek Handbook

How do I write a non-stop crawler using python and run it on the server?

I want to write a scraper with python that scans some urls and cleans and saves data. I know how to write it as a simple program. I am looking for a way to deploy it on my virtual server (running ubuntu) as a service to make it scan non-stop. Can anyone tell me how I can do this?

+3


source to share


1 answer


What you want to do is demo the process. This will be useful when creating a daemon.

Demonstrating the process will allow it to run in the background, so as long as the server is running (even if the user is not logged in) it will continue to run.



Here is an example daemon that writes time to a file.

import daemon
import time

def do_something():
    while True:
        with open("/tmp/current_time.txt", "w") as f:
            f.write("The time is now " + time.ctime())
        time.sleep(5)

def run():
    with daemon.DaemonContext():
        do_something()

if __name__ == "__main__":
    run()
+3


source






More articles:


All Articles