Bernhard's Homepage

Concurrent execution of tasks within Python

Concurrent execution of tasks within Python

The concurrent.futures module in Python 3 allows parallel execution of tasks. For Python 2 a backport is available by the futures module.

I'm using this module to parallelize NAPALM requests. The same topology as in the "Use NAPALM to configure Cisco and Juniper router" post will be used here.

In this example, I'm implementing a get_conf_all program, that gets the configuration of the all devices, up to 5 configurations are loaded in parallel. It closely follows the ThreadPoolExecutor example of the documentation.

# load device configuration
import sys
import json
from napalm import get_network_driver
from concurrent.futures import ThreadPoolExecutor, as_completed

def die(*msg_list):
    """ abort program with error message """
    error_msg = ' '.join(str(x) for x in msg_list)

def save_conf(dev_name, dev_param):
    """ load config from device and save it into file """
    driver = get_network_driver(dev_param['type'])
    with driver(dev_param['IP'], dev_param['user'], dev_param['password']) as device:
        conf = device.get_config()
        with open(dev_name + ".conf", "w") as conf_file:
    return dev_name + " saved"

# load device parameter database
    with open("devices", "r") as f:
        dev_db = json.load(f)
except (ValueError, IOError, OSError) as err:
    die("Could not read the 'devices' file:", err)

# concurrent execution of max. 5 threads
with ThreadPoolExecutor(5) as executor:
    # start threads
    futures = {}
    for dev in dev_db:
        futures[executor.submit(save_conf, dev, dev_db[dev])] = dev

    # wait for thread termination
    for proc in as_completed(futures):
        dev = futures[proc]
            data = proc.result()
        except Exception as exc:
            print('{}: {}'.format(dev, exc))