1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Python: why does multiprocessing apply threading instead of multiple processors?

Discussion in 'Programming/Internet' started by apfz, Sep 12, 2018.

  1. apfz

    apfz Guest

    I am using the inspyred library for genetic algorithms. I have the following configuration:

    def main(debugMode):
    start = default_timer()

    data = DataProblem(debugMode)

    prng = Random()
    prng.seed(time())

    problem = inspyred.benchmarks.TSP(data.distances)
    ea = inspyred.ec.EvolutionaryComputation(prng)
    ea.selector = inspyred.ec.selectors.tournament_selection
    ea.variator = [custom_partially_matched_crossover, custom_inversion_mutation]
    ea.replacer = inspyred.ec.replacers.generational_replacement
    ea.terminator = inspyred.ec.terminators.generation_termination
    final_pop = ea.evolve(generator=generator,
    bounder=problem.bounder,
    maximize=False,
    evaluator=inspyred.ec.evaluators.parallel_evaluation_mp,
    mp_evaluator=evaluator,
    pop_size=100,
    data=data,
    max_generations=100000,
    tournament_size=5,
    num_selected=100,
    num_elites=1)

    best = max(ea.population)
    print('Best Solution: {0}: {1}'.format(str(best.candidate), best.fitness))
    duration = default_timer() - start
    print(duration / 60)
    return ea


    if __name__ == '__main__':

    try:
    fileLocation = sys.argv[1]
    timeOfDay = sys.argv[2]
    debugMode = False
    except:
    debugMode = True
    timeOfDay = 'morning'

    main(debugMode)


    The mp_evaluator enables multiprocessing and uses the following method:

    import time
    import multiprocessing
    logger = args['_ec'].logger

    try:
    evaluator = args['mp_evaluator']
    except KeyError:
    logger.error('parallel_evaluation_mp requires \'mp_evaluator\' be defined in the keyword arguments list')
    raise
    try:
    nprocs = args['mp_nprocs']
    except KeyError:
    nprocs = multiprocessing.cpu_count()

    pickled_args = {}
    for key in args:
    try:
    pickle.dumps(args[key])
    pickled_args[key] = args[key]
    except (TypeError, pickle.PickleError, pickle.PicklingError):
    logger.debug('unable to pickle args parameter {0} in parallel_evaluation_mp'.format(key))
    pass

    start = time.time()
    try:
    pool = multiprocessing.Pool(processes=nprocs)
    results = [pool.apply_async(evaluator, ([c], pickled_args)) for c in candidates]
    pool.close()
    pool.join()
    return [r.get()[0] for r in results]
    except (OSError, RuntimeError) as e:
    logger.error('failed parallel_evaluation_mp: {0}'.format(str(e)))
    raise
    else:
    end = time.time()
    logger.debug('completed parallel_evaluation_mp in {0} seconds'.format(end - start))


    However when I look in my activity monitor I only see about 85% of 1 processor being used with 4 threads. The code executes actually faster on 1 processor without the multiprocessor enabled.

    But why is it using 4 threads instead of 4 processors in case of a quad-core machine?

    Login To add answer/comment
     

Share This Page