Shulou(Shulou.com)06/01 Report--

This article will explain in detail how to use the parallel software ParaFly on linux. The content of the article is of high quality, so the editor shares it for you as a reference. I hope you will have some understanding of the relevant knowledge after reading this article.

Most of the time, Shengxin workers deal with multiple files and do the same for each file. At this time, it is possible to write a for loop serial processing. But sometimes serial operations take a long time, such as doing a bwa mem comparison, if a file is processed, the time is cumulative.

Generally speaking, we all run tasks on the server, and the server is multi-core and multi-threaded, and then it is our own multi-core and multi-threaded notebook, which actually allows tasks to run in parallel.

For example, using htop to view, the following is a 40-core server. It is better to use idle cores when needed, saving time and effort:

Today, I would like to introduce to you a very easy to use gadget ParaFly, which only needs a very simple operation to parallelize tasks on linux.

First install ParaFly using anconda3

$conda install-c bioconda parafly

Check to see how it works:

$ParaFly-h

# Usage: ParaFly (opts) # # Required: #-c: filename containing list of bash-style commands to execute.#-CPU: number_of_threads## Optional:#- Shuffle: randomly shuffles the command order. #-failed_cmds: filename to capture failed commands. Default ("FailedCommands") #-v: simple progress monitoring.#-vv: increased verbosity in progress monitoring.###

Note: This process creates a file named based > with a. Extension.This enables a resume functionality, where if rerun, only those commands not completed successfully will be reprocessed.

You can see that there are only two required parameters (Required), indicating that the tool is very easy to use. In fact, you only need to write the commands you need to operate in a file, specify the file with-c, and specify the total number of cores in parallel with-CPU (you need to execute several commands at the same time).

If the command is executed successfully by default, the corresponding command line will be output to a file with a filename followed by the suffix. Commands that are not successfully executed will be output to a FailedCommands file.

ParaFly: Simple parallel unix command processing using OpenMPParaFly provides a simple mechanism for running a predefined list of unix commands in parallel using multithreading. Failed processes are captured and reported. Successfully executed processes are noted. If the process is rerun, > or failed processes will be executed.

Take a simple example:

Fastq-dump splits three SRA files, and you can write these three commands into the fastq_dump.txt file:

$cat fastq_dump.txtfastq-dump-split-3 SRR1510125.1fastq-dump-split-3 SRR1004284.1fastq-dump-split-3 SRR1004285.1

Then use ParaFly parallelization:

$ParaFly-c fastq_dump.txt-CPU 3

The server runs the three commands, the ParaFly result file, with three cores at the same time:

$cat fastq_dump.txt.completedfastq-dump-split-3 SRR1004285.1fastq-dump-split-3 SRR1004284.1fastq-dump-split-3 SRR1510125.1

Generally speaking, ParaFly is easy to learn and easier to use than the parallel that comes with linux. When you do multi-task processing on linux, you can consider adding it, which will greatly shorten the data processing time.

About how to use the parallel software ParaFly on linux to share here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.