Shulou(Shulou.com)06/02 Report--

This article mainly explains the "python multi-process and VNPY multi-process parameter optimization example analysis", the content of the article is simple and clear, easy to learn and understand, the following please follow the editor's ideas slowly in-depth, together to study and learn "python multi-process and VNPY multi-process parameter optimization example analysis"!

First of all, due to the problem of GIL (Global interpreted Lock), global objects can only be called by one process, and python multithreading can not make full use of multi-core processors. For example, sometimes using pandas to run large data analysis, it is found that only one core is working hard. If you want to make full use of the resources of multicore CPU, you need to use multiple processes in most cases in python. Multiprocessing can assign a separate Python interpreter to each process, thus avoiding the problems caused by global interpretation of locks. It can be understood that multicore CPU assigns a work task, which includes work method and work content.

In fact, python multithreading is very simple, compared to other languages. In fact, it is simply for the method func (a) that requires multithreading, where an is the parameter. Equivalent to the work content; using Multiprocessing.Process (target = func, args = (a,)), create a Prcoess object, the work task, and then start the object, so that a multi-process task is complete. When CPU assigns an independent core to work, func (a) starts. The only thing to note here is that args is the default input meta-ancestor parameter.

P = Multiprocessing.Process (target = func, args = (a,) P.start ()

Multiprocessing provides a more concise pool as a process pool, which is actually more appropriate to call it a task pool. Pack up the tasks that need to be done and put them in this pool, so that the free core will pick up the tasks of pool.

The common use of pool is as follows, where prcesses = 4 defines the task pool size, which is not necessarily less than or equal to the number of cpu cores, but can be greater than the number of cpu cores, but there are several tasks hanging empty and taking up memory.

Then use the pool method apply_async (task, args= (x,)) to insert the packaged task into the pool. Apply_asyncs is asynchronous with a return value. It's OK to use apply, but there is no return value, so I won't take a closer look at it here.

After that, close () closes the task pool and no longer accepts new tasks; but there are still some existing tasks running, so use pool.join () to hang the main program until all the tasks are completed before moving on to the next step.

If _ _ name__ ='_ main__': Multiprocessing.pool = Pool (processes=4) for x in range (10): pool.apply_async (task, args= (x,) pool.close () pool.join ()

Let's take a look at the VNPY multi-process optimization method. It's easy to understand that runParallelOptimization is a BacktestingEngine-like method.

The passed parameter strategyClass is this policy class. Setting is to optimize the parameter range, and then the policy parameter queue is generated through optimizationSetting.generateSetting () as the task content; optimizationSetting.optimizeTarget is the return value. As for the return test variety, the return test time period, and the transaction cost, it was maintained when the BacktestingEngine was created.

Then create a task pool pool, which is exactly the number of cpu cores, which is also a relatively safe setting.

Then make an l queue to put the return value.

Then the policy class, backtest parameters and policy parameters are packaged as task contents, and combined with the task method optimize to form a work task. Then insert the task pool to the cpu core to run. At this time, the system monitor can see the operation of the python virtual environment with the same number of cores.

Then sort the return values. I'll talk about it in more detail later.

Df = engine.runParallelOptimization (AtrRsiStrategy, setting) def runParallelOptimization (self, strategyClass, optimizationSetting): "" parallel optimization parameters "" # get optimization settings settingList = optimizationSetting.generateSetting () targetName = optimizationSetting.optimizeTarget # check parameter setting problems if not settingList or not targetName: self.output (there is a problem with u' optimization settings, please check') # multiprocess optimization Start a process pool corresponding to the number of CPU cores pool = multiprocessing.Pool (multiprocessing.cpu_count ()) l = [] for setting in settingList: l.append (pool.apply_async (optimize, (strategyClass, setting, targetName, self.mode, self.startDate, self.initDays, self.endDate) Self.slippage, self.rate, self.size, self.priceTick, self.dbName, self.symbol)) pool.close () pool.join () # shows the result resultList = [res.get () for res in l] resultList.sort (reverse=True, key=lambda result:result [1]) return resultList

For example, there are now four python environments running tasks in dual-core quad threads.

Here, we will find that we use the static method optimize. If it is more concise to call the back test method of BacktestingEngine directly, why not? this is a limitation of python2.7 's Multiprocessing. You can only package the static method as the working method. If you package the method in the class, you will get an error.

CPickle.PicklingError: Can't pickle: attribute lookup builtin.instanceme

If VNPY2.0 is based on the python3.6 version, it should be a little more simplified.

Let's take a look at the static method optimize. In fact, there is nothing to say, that is, to create a new backtest engine BacktestingEngine object, run the retest according to the parameters, and return a meta-ancestor, including the parameters of the retest, the value of the target, and a dictionary containing the results of the test. This dictionary includes a bunch of test results, such as annual income, sharpe and so on.

Then all the meta-ancestors of the test results form a queue of test results, which is sorted in reverse according to targetValue, with the largest in the first place.

Because there are too many, I usually output to excel, and I said how to achieve it before.

#-def optimize (strategyClass, setting, targetName, mode, startDate, initDays, endDate, slippage, rate, size, priceTick, dbName Symbol): "engine = BacktestingEngine () engine.setBacktestingMode (mode) engine.setStartDate (startDate, initDays) engine.setEndDate (endDate) engine.setSlippage (slippage) engine.setRate (rate) engine.setSize (size) engine.setPriceTick (priceTick) engine.setDatabase (dbName, symbol) engine.initStrategy (strategyClass, setting) engine.runBacktesting () engine.calculateDailyResult () d Result = engine.calculateDailyStatistics () try: targetValue = result [targetName] except KeyError: targetValue = 0 return (str (setting), targetValue, result)

In fact, python's multi-process library Multiprocessing is not complex, but it works well in back testing; now with genetic algorithm, it is more convenient to optimize the strategy.

Thank you for your reading, the above is the content of "python multi-process and VNPY multi-process parameter optimization example analysis". After the study of this article, I believe you have a deeper understanding of the python multi-process and VNPY multi-process parameter optimization example analysis of this problem, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.