Ytukyg

I assume this is an obvious one, but I am making a script which uses threading to tests proxies, and should stop once it finds a certain number of proxies. When I run it, threads stop generating output if the condition is met, but the program does not close. I've looked at other similar issues but can't seem to implement any successfully. Would appreciate any pointers.

import queue

import threading



import time   

import urllib.request



class ThreadUrl(threading.Thread):



    def __init__(self, queue, working_proxies):

        threading.Thread.__init__(self)

        self.queue = queue

        self.working_proxies = working_proxies



    def run(self):

        while len(self.working_proxies)<5:



            proxy = self.queue.get()



            try:

                proxy_handler = urllib.request.ProxyHandler({'http': proxy})

                opener = urllib.request.build_opener(proxy_handler)

                opener.addheaders = [('User-agent',

                                      'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36')]

                urllib.request.install_opener(opener)

                req = urllib.request.Request('http://www.wikipedia.org')

                sock=urllib.request.urlopen(req)

                print(f'{proxy} works')

                with appending_lock:

                    self.working_proxies.append(proxy)



            except urllib.request.HTTPError as e:

                print('Error code: ', e.code)



            except Exception as detail:

                print("ERROR:", detail)



            self.queue.task_done()





def main(proxies, working_proxies):

    for i in range(5):

        t = ThreadUrl(queue,working_proxies)

        t.daemon = True

        t.start()





        for proxy in proxies:

            queue.put(proxy)



    queue.join()





if __name__ == '__main__':

    start = time.time()



    appending_lock = threading.Lock()

    proxies = [...list of proxies...]

    working_proxies = 



    queue = queue.Queue()



    main(proxies, working_proxies)

    print("Elapsed Time: %s" % (time.time() - start))

Am I using the daemon attribute wrong, or are there other parameters I should set up in order to make sure threading stops?

After going through the docs and a few other resources, I found that the threads did close properly, but queue.join() function was blocking until the queue was emptied. Since that was not going to happen if the threads were killed before finishing all items from the queue, the script continued to run.

So, I have overridden the queue.join() to look like this:

queue = queue.Queue()



def waiter(queue):

    while not queue.empty() and dead == False:

        pass



queue.join = waiter

The 'dead' variable is accessed within a thread if it establishes that a closing condition is fulfilled. In the code above, 'dead' would be put instead of 'len(self.working_proxies)<5' in the while loop of the run() function. Each time a new item is appended to self.working_proxies, the script should check if the condition for terminating has been met, and, if so, set 'dead' to True.

Also, 'dead' is a global variable, so it does not need to be passed into the new waiter() function.

I'm sure there is a more elegant way to approach the issue, but this one should do the trick in the meanwhile.