GNU Parallel
GNU parallel is a shell tool for executing jobs in parallel locally or using remote computers. A job is typically a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. If you use xargs today you will find GNU parallel very easy to use, as GNU parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. If you use ppss or pexec you will find GNU parallel will often make the command easier to read. GNU parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU parallel as input for other programs.
| Tags | Text Processing parallel Parallel processing Multicore Clustering/Distributed Networks Command Line Tools Filters System Administration |
|---|---|
| Licenses | GPLv3 |
| Operating Systems | POSIX Linux Solaris Unix Darwin FreeBSD |
| Implementation | Perl |
Recent releases
- 23 Jan 2012 01:24
Release Notes: --header now uses the first input line as column names, and you can then use {colname} as a replacement string in the command. (This also works with multiple ":::"s.) --header <regexp> matches a header as a regular expression and repeats the header for each block with --pipe. --resume resumes from the last unfinished job. (This is useful if you stop GNU Parallel and restart it later with the same arguments.) niceload now propagates the exit status correctly. The options -g, -B, -T, -U, -W, and -Y have been retired, as warned six months ago.
- 22 Dec 2011 00:57
Release Notes: --timeout will now kill grandchildren.
- 22 Nov 2011 22:25
Release Notes: This is a bugfix release with no new features. It is probably a good release for stable long-term use.
- 23 Oct 2011 20:50
Release Notes: --tag prepends lines with the argument(s). If used with --(n)onall, the line will be prepended with the sshlogin instead. --shellquote does not run the command, but quotes it using \'s, which is useful for making quoted composed commands. --profile can now be repeated, merging multiple profiles. --bibtex now gives the BibTeX entry. A makefile for a simple .deb package: cd packager/debian; make. sql: --list-databases lists the databases.
- 22 Aug 2011 00:45
Release Notes: A --timeout option has been implemented so that slow commands can be killed. CPU detection has been improved for Mac OS X. There is an example of a parallel Web crawler in the man page.
Heartbeat
- Popularity Score
- 480.80
- Vitality Score
- 42.77
- Subscriptions
- 42
