pigz
pigz (parallel implementation of gzip) is a fully functional replacement for gzip that exploits multiple processors and multiple cores when compressing data. pigz is extremely helpful when compacting a large directory. unpigz (equivalent to 'gunzip') is used to uncompress gzip'ed file and unpigz utilizes only one thread.
pigz can be accessed by loading a pigz module (run "module spider pigz" to find the module name). On FTN nodes, pigz is available without loading the module.
Simple examples
-
Compress a file
pigz my_file
-
Compress a file with best compression rate
pigz -9 my_file pigz --best my_file
-
Uncompress
unpigz my_file.gz
-
Specify number of threads ('-p no_thread'). The number of compress threads is set by default to the number of online processors.
pigz -9 -p 10 my_big_file
Using tar with pigz
-
specify compress program for tar
tar -I pgz cf my_dir.tar.gz my_dir/ tar --use=compress-program pigz my_dir.tar.gz my_dir/
-
specify more options with pigz
tar cf - my_dir/ | pigz -9 > my_dir.tgz
tar cf - my_dir1 my_dir2 | pigz -9 -p 15 > my_backup.tgz
References
- pigz home page: https://zlib.net/pigz/
- pigz manual page: https://zlib.net/pigz/pigz.pdf
- some speed test results: https://rachaellappan.github.io/pigz/