Blog > Threading with python


Threading with python

how to spawn exact number of processes in parallel

# Import necessary modules

from multiprocessing.pool import ThreadPool import subprocess

# Define work function

def work(sample): my_tool_subprocess = subprocess.Popen('mytool'.format(sample), shell=True, stdout=subprocess.PIPE) line = True
while line: myline = my_tool_subprocess.stdout.readline()

# here I parse stdout..

# Set the number of workers (defaults to CPU count)

num = None tp = ThreadPool(num)

# Apply work function asynchronously to each sample

for sample in all_samples: tp.apply_async(work, (sample,))

# Close and join ThreadPool

tp.close() tp.join()

Threading Advanages:

  • - Threading allows multiple tasks to be executed concurrently.
  • - Threading can enhance responsiveness, especially in tasks involving waiting for external resources.
  • - Threading is beneficial for tasks that can run independently, making efficient use of system resources.


    • is used to run a command in a subprocess and wait for it to complete.
    • It is a blocking call, meaning the program waits until the subprocess finishes its execution before proceeding.
    • It returns the return code of the subprocess.
  2. subprocess.Popen:
    • subprocess.Popen is used for more advanced process handling, allowing greater control over the input, output, and error streams of the subprocess.
    • It runs the command in a subprocess, but the Python program doesn't wait for the subprocess to finish.
    • You can communicate with the subprocess using its input and output streams.

If you're considering threading in Python, the subprocess module itself is not directly related to threading. If you want to run multiple subprocesses concurrently, you might want to explore the concurrent.futures module, which provides a high-level interface for asynchronously executing functions in separate threads or processes.

If you have a specific use case related to threading and subprocesses, providing more details would help in giving a more targeted recommendation.