本篇文章给大家带来的内容是关于python中并发future模块的介绍(代码) ,有一定的参考价值,有需要的朋友可以参考一下,希望对你有所帮助。
concurrent.futures模块该模块主要特色在于threadpoolexecutor 和 processpoolexecutor 类,这两个类都继承自concurrent.futures._base.executor类,它们实现的接口能分别在不同的线程或进程中执行可调用的对象,它们都在内部维护着一个工作线程或者进程池。
threadpoolexecutor 和 processpoolexecutor 类是高级类,大部分情况下只要学会使用即可,无需关注其实现细节。
####processpoolexecutor 类
>class threadpoolexecutor(concurrent.futures._base.executor)>| this is an abstract base class for concrete asynchronous executors.>| method resolution order:>| threadpoolexecutor | concurrent.futures._base.executor | builtins.object | | methods defined here: | | init(self, max_workers=none, thread_name_prefix='') | initializes a new threadpoolexecutor instance. | | args: | max_workers: the maximum number of threads that can be used to | execute the given calls. | thread_name_prefix: an optional name prefix to give our threads. | | shutdown(self, wait=true) | clean-up the resources associated with the executor. | | it is safe to call this method several times. otherwise, no other | methods can be called after this one. | | args: | wait: if true then shutdown will not return until all running | futures have finished executing and the resources used by the | executor have been reclaimed. | | submit(self, fn, *args, **kwargs) | submits a callable to be executed with the given arguments. | | schedules the callable to be executed as fn(*args, **kwargs) and returns | a future instance representing the execution of the callable. | | returns: | a future representing the given call. | | ---------------------------------------------------------------------- | methods inherited from concurrent.futures._base.executor: | | enter(self) | | exit(self, exc_type, exc_val, exc_tb) | | map(self, fn, *iterables, timeout=none, chunksize=1) | returns an iterator equivalent to map(fn, iter). | | args: | fn: a callable that will take as many arguments as there are | passed iterables. | timeout: the maximum number of seconds to wait. if none, then there | is no limit on the wait time. | chunksize: the size of the chunks the iterable will be broken into | before being passed to a child process. this argument is only | used by processpoolexecutor; it is ignored by | threadpoolexecutor. | | returns: | an iterator equivalent to: map(func, *iterables) but the calls may | be evaluated out-of-order. | | raises: | timeouterror: if the entire result iterator could not be generated | before the given timeout. | exception: if fn(*args) raises for any values.
初始化可以指定一个最大进程数作为其参数 max_workers 的值,该值一般无需指定,默认为当前运行机器的核心数,可以由os.cpu_count()获取;类中含有方法:
map()方法,与python内置方法map() 功能类似,也就是映射,参数为:
一个可调用函数 fn
一个迭代器 iterables
超时时长 timeout
块数chuncksize 如果大于1, 迭代器会被分块处理
---->> 该函数有一个特性:其返回结果与调用开始的顺序是一致的;在调用过程中不会产生阻塞,也就是说可能前者被调用执行结束之前,后者被调用已经执行结束了。
如果一定要获取到所有结果后再处理,可以选择采用submit()方法和futures.as_completed函数结合使用。
shutdown()方法,清理所有与当前执行器(executor)相关的资源
submit() 方法,提交一个可调用对象给fn使用
从concurrent.futures._base.executor继承了__enter__() 和 __exit__()方法,这意味着processpoolexecutor 对象可以用于with 语句。
from concurrent import futureswith futures.processpoolexecutor(max_works=3) as executor: executor.map()
threadpoolexecutor类class threadpoolexecutor(concurrent.futures._base.executor) | this is an abstract base class for concrete asynchronous executors. | | method resolution order: | threadpoolexecutor | concurrent.futures._base.executor | builtins.object | | methods defined here: | | init(self, max_workers=none, thread_name_prefix='') | initializes a new threadpoolexecutor instance. | | args: | max_workers: the maximum number of threads that can be used to | execute the given calls. | thread_name_prefix: an optional name prefix to give our threads. | | shutdown(self, wait=true) | clean-up the resources associated with the executor. | | it is safe to call this method several times. otherwise, no other | methods can be called after this one. | | args: | wait: if true then shutdown will not return until all running | futures have finished executing and the resources used by the | executor have been reclaimed. | | submit(self, fn, *args, **kwargs) | submits a callable to be executed with the given arguments. | | schedules the callable to be executed as fn(*args, **kwargs) and returns | a future instance representing the execution of the callable. | | returns: | a future representing the given call. | | ---------------------------------------------------------------------- | methods inherited from concurrent.futures._base.executor: | | enter(self) | | exit(self, exc_type, exc_val, exc_tb) | | map(self, fn, *iterables, timeout=none, chunksize=1) | returns an iterator equivalent to map(fn, iter). | | args: | fn: a callable that will take as many arguments as there are | passed iterables. | timeout: the maximum number of seconds to wait. if none, then there | is no limit on the wait time. | chunksize: the size of the chunks the iterable will be broken into | before being passed to a child process. this argument is only | used by processpoolexecutor; it is ignored by | threadpoolexecutor. | | returns: | an iterator equivalent to: map(func, *iterables) but the calls may | be evaluated out-of-order. | | raises: | timeouterror: if the entire result iterator could not be generated | before the given timeout. | exception: if fn(*args) raises for any values.
与processpoolexecutor 类十分相似,只不过一个是处理进程,一个是处理线程,可根据实际需要选择。
示例from time import sleep, strftimefrom concurrent import futuresdef display(*args): print(strftime('[%h:%m:%s]'), end=) print(*args)def loiter(n): msg = '{}loiter({}): doing nothing for {}s' display(msg.format('\t'*n, n, n)) sleep(n) msg = '{}loiter({}): done.' display(msg.format('\t'*n, n)) return n*10def main(): display('script starting') executor = futures.threadpoolexecutor(max_workers=3) results = executor.map(loiter, range(5)) display('results:', results) display('waiting for inpidual results:') for i, result in enumerate(results): display('result {} : {}'.format(i, result))if __name__ == '__main__': main()
运行结果:
[20:32:12]script starting[20:32:12]loiter(0): doing nothing for 0s[20:32:12]loiter(0): done.[20:32:12] loiter(1): doing nothing for 1s[20:32:12] loiter(2): doing nothing for 2s[20:32:12]results: <generator object executor.map.<locals>.result_iterator at 0x00000246db21bc50>[20:32:12]waiting for inpidual results:[20:32:12] loiter(3): doing nothing for 3s[20:32:12]result 0 : 0[20:32:13] loiter(1): done.[20:32:13] loiter(4): doing nothing for 4s[20:32:13]result 1 : 10[20:32:14] loiter(2): done.[20:32:14]result 2 : 20[20:32:15] loiter(3): done.[20:32:15]result 3 : 30[20:32:17] loiter(4): done.[20:32:17]result 4 : 40
不同机器运行结果可能不同。
示例中设置max_workers=3,所以代码一开始运行,则有三个对象(0,1,2)被执行loiter() 操作; 三秒后,对象0运行结束,得到结果result 0之后,对象3才开始被执行,同理,对象4的执行时间在对象1执行结果result 1打印结束之后。
相关推荐:
python如何通过future处理并发问题的实例详解
以上就是python中并发future模块的介绍(代码)的详细内容。