What's the fastest way to run multiple PHP-CLIs?

I need to move many mysql rows to couchbase server. The trick is that I need to use a PHP class to do the job (the class has business logic)

I created a PHP ClI script and ran 6 of them at once. It's faster than running a single CLI script, but not enough. It took me 2 hours to transfer everything.

Is there a better way?

Updated:

What PHP code does with Mysql

  • select * from limit $ limit table

What about it. Nothing unusual.

+3


source to share


1 answer


Is there a better way?

Yes. Most likely there is.

You need to identify the bottleneck. From what you are describing, it seems the bottleneck is the number of jobs running in parallel. Therefore, you must increase this until you find the maximum performance. GNU Parallel helps you with this often.

When you have done this, the bottleneck is somewhere else. But since your description has very few details, it's impossible to tell where.

Therefore, you will need to find a new bottleneck. The bottleneck is typically disk I / O, network I / O, or processor, but can also be shared lock or other source.



To find the CPU bottleneck top

. If you see that the process is working 100%, and you cannot parallelize this process, then you have found your bottleneck.

To find the disk I / O bottleneck iostat -dkx 1

. If the last column hits 100% more than 50% of the time, you've found your bottleneck.

Finding a network I / O node iotop

. If the used bandwidth is> 70% of the available network bandwidth, you have found your bottleneck.

DNS is a particular network problem: it can often be seen as a process that has been stuck for 5-10 seconds without a valid reason, but otherwise it works fine. Use tcpdump -s1000 -n port 53

and see if the questions are answered quickly. Do this on all machines participating in the mission.

Finding a shared lock is more difficult. First, you will need to find suspicious processes, and then you have to find strace

them.

+1


source







All Articles