Ytukyg

I am looking to run multiple instances of a command line script at the same time. I am new to this concept of "multi-threading" so am at bit of a loss as to why I am seeing the things that I am seeing.

I have tried to execute the sub-processing in two different ways:

1 - Using multiple calls of Popen without a communicate until the end:

command = 'raster2pgsql -I -C -e -s 26911 %s -t 100x100 -F p839.%s_image_sum_sum1 | psql -U david -d projects -h pg3' % (workspace + '\r_sumsum1{}'.format(i), str(i))



process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)



command = 'raster2pgsql -I -C -e -s 26911 %s -t 100x100 -F p839.%s_image_sum_sum2 | psql -U david -d projects -h pg3' % (workspace + '\r_sumsum2{}'.format(i), str(i))



process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)



command = 'raster2pgsql -I -C -e -s 26911 %s -t 100x100 -F p839.%s_image_sum_sum3 | psql -U david -d projects -h pg3' % (workspace + '\r_sumsum3{}'.format(i), str(i))



process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)



(stdoutdata, stderrdata) = process.communicate()

this starts up each of the command line item but only completes the last entry leaving the other 2 hanging.

2 - Attempting to implement an example from Python threading multiple bash subprocesses? but nothing happens except for a printout of the commands (program hangs with no command line arguments running as observed in windows task manager:

import threading

import Queue

import commands

import time



workspace = r'F:ProcessingSM'

image = 't08r_e'

image_name = (image.split('.'))[0]

i = 0

process_image_tif = workspace + '\{}{}.tif'.format((image.split('r'))[0], str(i))



# thread class to run a command

class ExampleThread(threading.Thread):

    def __init__(self, cmd, queue):

        threading.Thread.__init__(self)

        self.cmd = cmd

        self.queue = queue



    def run(self):

        # execute the command, queue the result

        (status, output) = commands.getstatusoutput(self.cmd)

        self.queue.put((self.cmd, output, status))



# queue where results are placed

result_queue = Queue.Queue()



# define the commands to be run in parallel, run them

cmds = ['raster2pgsql -I -C -e -s 26911 %s -t 100x100 -F p839.%s_image_sum_sum1 | psql -U david -d projects -h pg3' % (workspace + '\r_sumsum1{}'.format(i), str(i)),

        'raster2pgsql -I -C -e -s 26911 %s -t 100x100 -F p839.%s_image_sum_sum2 | psql -U david -d projects -h pg3' % (workspace + '\r_sumsum2{}'.format(i), str(i)),

        'raster2pgsql -I -C -e -s 26911 %s -t 100x100 -F p839.%s_image_sum_sum3 | psql -U david -d projects -h pg3' % (workspace + '\r_sumsum3{}'.format(i), str(i)),



       ]

for cmd in cmds:

    thread = ExampleThread(cmd, result_queue)

    thread.start()



# print results as we get them

while threading.active_count() > 1 or not result_queue.empty():

    while not result_queue.empty():

        (cmd, output, status) = result_queue.get()

        print(cmd)

        print(output)

How can I run all of these commands at the same time achieving a result at the end? I am running in windows, pyhton 2.7.

My first try didn't work because of the repeated definitions of stdout and sterror. Removing these definitions causes expected behavior.