CSE Courses

Engineering at Illinois Engineering at Illinois

Bibliography on Parallel Computing

Internet Resources
Parallel Computing (general surveys)
  1. A. Abbas, Grid Computing: A Practical Guide to Technology and Applications, Charles River Media, 2004.
  2. G. S. Almasi and A. Gottlieb, Highly Parallel Computing, 2nd ed., Benjamin/Cummings, 1994.
  3. F. Berman, A. Hey, and G. Fox, eds., Grid Computing: Making the Global Infrastructure a Reality, John Wiley & Sons, 2003.
  4. A. Chalmers and J. Tidmus, Practical Parallel Processing, International Thomson, 1996.
  5. B. Codenotti and M. Leoncini, Introduction to Parallel Processing, Addison-Wesley, 1993.
  6. J. M. Crichlow, Introduction to Distributed and Parallel Computing, Prentice Hall, 1988.
  7. Y. Deng, Applied Parallel Computing, World Scientific, 2011.
  8. G. R. Desrochers, Principles of Parallel and Multiprocessing, McGraw-Hill, 1987.
  9. J. Dongarra, et al., eds., Sourcebook of Parallel Computing, Morgan Kaufmann, 2003.
  10. K. Dowd and C. R. Severance, High Performance Computing, 2nd. ed., O'Reilly, 1998.
  11. H. El-Rewini and T. G. Lewis, Distributed and Parallel Computing, Manning, 1998.
  12. I. Foster and C. Kesselman, eds., The Grid: Blueprint for a New Computing Infrastructure, Morgan Kaufmann, 1999.
  13. I. Foster and C. Kesselman, eds., The Grid 2: Blueprint for a New Computing Infrastructure, Morgan Kaufmann, 2003.
  14. T. J. Fountain, Parallel Computing: Principles and Practice, Cambridge University Press, 1994.
  15. F. Gebali, Algorithms and Parallel Computing, John Wiley & Sons, 2011.
  16. R. W. Hockney and C. R. Jesshope, Parallel Computers 2: Architecture, Programming and Algorithms, Institute of Physics Publishing, 1988.
  17. R. M. Hord, Understanding Parallel Supercomputing, IEEE Press, 1999.
  18. E. V. Krishnamurthy, Parallel Processing: Principles and Practice, Addison-Wesley, 1989.
  19. H. F. Jordan and G. Alaghband, Fundamentals of Parallel Processing, Prentice Hall, 2003.
  20. E. L. Leiss, Parallel and Vector Computing: A Practical Introduction, McGraw-Hill, 1995.
  21. T. G. Lewis and H. El-Rewini, Introduction to Parallel Computing, Prentice Hall, 1992.
  22. G. J. Lipovski and M. Malek, Parallel Computing: Theory and Comparisons, John Wiley & Sons, 1987.
  23. D. I. Moldovan, Parallel Processing: From Applications to Systems, Morgan Kaufmann, 1993.
  24. H. S. Morse, Practical Parallel Computing, Academic Press, 1994.
  25. W. P. Petersen and P. Arbenz, Introduction to Parallel Computing: A Practical Guide with Examples in C, Oxford University Press, 2004.
  26. M. J. Quinn, Parallel Computing: Theory and Practice, McGraw-Hill, 1994.
  27. S. Rajasekaran and J. Reif, eds., Handbook of Parallel Computing: Models, Algorithms, and Applications, Chapman & Hall/CRC, 2008.
  28. S. H. Roosta, Parallel Processing and Parallel Algorithms: Theory and Computation, Springer, 2000.
  29. J. A. Sharp, An Introduction to Distributed and Parallel Processing, Blackwell, 1987.
  30. A. Y. Zomaya, ed., Parallel and Distributed Computing Handbook, McGraw-Hill, 1996.
Parallel Programming
  1. G. R. Andrews, Concurrent Programming: Principles and Practice, Benjamin/Cummings, 1991.
  2. G. R. Andrews, Foundations of Multithreaded, Parallel, and Distributed Programming, Addison-Wesley, 2000.
  3. R. G. Babb, ed., Programming Parallel Processors, Addison-Wesley, 1988.
  4. L. Baker and B. J. Smith, Parallel Programming, McGraw-Hill, 1996.
  5. B. E. Bauer, Practical Parallel Programming, Academic Press, 1992.
  6. M. Ben-Ari, Principles of Concurrent and Distributed Programming, 2nd ed., Addison-Wesley, 2006.
  7. G. E. Blelloch, Vector Models for Data-Parallel Computing, MIT Press, 1990.
  8. T. Braunl, Parallel Programming: An Introduction, Prentice Hall, 1993.
  9. S. Brawer, Introduction to Parallel Programming, Academic Press, 1989.
  10. C. Breshears, The Art of Concurrency: A Thread Monkey's Guide to Writing Parallel Applications, O'Reilly Media, 2009.
  11. P. Brinch Hansen, Studies in Computational Science: Parallel Programming Paradigms, Prentice Hall, 1995.
  12. A. Burns and G. Davies, Concurrent Programming, Addison-Wesley, 1993.
  13. N. Carriero and D. Gelernter, How to Write Parallel Programs: A First Course, MIT Press, 1990.
  14. K. M. Chandy and J. Misra, Parallel Program Design: A Foundation, Addison-Wesley, 1988.
  15. K. M. Chandy and S. Taylor, An Introduction to Parallel Programming, Jones & Bartlett, 1992.
  16. I. T. Foster, Designing and Building Parallel Programs, Addison-Wesley, 1995. Available online
  17. S. Goedecker and A. Hoisie, Performance Optimization of Numerically Intensive Codes, SIAM, 2001.
  18. G. Hager and G. Wellein, Introduction to High Performance Computing for Scientists and Engineers, CRC Press, 2011.
  19. M. Herlihy and N. Shavit, The Art of Multiprocessor Programming, Morgan Kaufmann, 2008.
  20. C. A. R. Hoare, Communicating Sequential Processes, Prentice Hall, 1985.
  21. A. H. Karp, “Programming for parallelism,” IEEE Computer Vol. 20, No. 9, pp. 43-57, 1987.
  22. J. Kepner, Parallel MATLAB for Multicore and Multinode Computers, SIAM, 2009.
  23. D. B. Kirk and W. W. Hwu, Programming Massively Parallel Processors: A Hands-on Approach, Morgan Kaufmann, 2010.
  24. A. L. Lastovetsky, Parallel Computing on Heterogeneous Networks, John Wiley & Sons, 2003.
  25. B. P. Lester, The Art of Parallel Programming, 2nd ed., 1st World Publishing, 2006.
  26. C. Lin and L. Snyder, Principles of Parallel Programming, Addison-Wesley, 2008.
  27. P. Luszczek, Parallel programming in MATLAB, Internat. J. High Perf. Comput. Appl., 23:277-283, 2009.
  28. T. G. Mattson, B. A. Sanders, and B. L. Massingill, Patterns for Parallel Programming, Addison-Wesley, 2005.
  29. J. L. Ortega-Arjona, Patterns for Parallel Software Design, Wiley, 2010.
  30. R. H. Perrott, Parallel Programming, Addison-Wesley, 1987.
  31. M. J. Quinn, Parallel Programming in C with MPI and OpenMP, McGraw-Hill, 2003
  32. S. Ragsdale, ed., Parallel Programming, McGraw-Hill, 1991.
  33. T. Rauber and G. Runger, Parallel Programming for Multicore and Cluster Systems, Springer, 2010.
  34. B. Wilkinson and M. Allen, Parallel Programming, 2nd ed., Prentice Hall, 2004.
  35. G. V. Wilson, Practical Parallel Programming, MIT Press, 1995.
Parallel Programming Languages and Systems
  1. C. Amza, A. Cox, S. Dwarkadas, P. Keleher, H. Lu, R. Rajamony, W. Yu, and W. Zwaenepoel, “TreadMarks: Shared Memory Computing on Networks of Workstations,” IEEE Computer, Vol. 29, No. 2, February 1996, pp. 18-28.
  2. D. F. Bacon, S. L. Graham, and O. J. Sharp, “Compiler transformations for high-performance computing,” ACM Computing Surveys Vol. 26, pp. 345-420, 1994
  3. H. E. Bal, J. G. Steiner, and A. S. Tanenbaum, Programming languages for distributed computing systems, ACM Computing Surveys Vol. 21, pp. 261-322, 1989
  4. D. R. Butenhof, Programming with POSIX Threads, Addison-Wesley, 1997.
  5. R. Chandra, R. Menon, L. Dagum, D. Kohr, D. Maydan, and J. McDonald, Parallel Programming in OpenMP, Morgan Kaufmann, 2000.
  6. B. Chapman, G. Jost, and R. van der Pas, Using OpenMP: Portable Shared Memory Parallel Programming, MIT Press, 2008.
  7. A. Geist, A. Beguelin, J. Dongarra, W. Jiang, R. Manchek, and V. Sunderam, PVM: Parallel Virtual Machine, A User's Guide and Tutorial for Networked Parallel Computing, MIT Press, 1994. Available online
  8. W. Gropp, S. Huss-Lederman, A. Lumsdaine, E. Lusk, B. Nitzberg, W. Saphir, and M. Snir, MPI: The Complete Reference, Vol. 2, The MPI Extensions, MIT Press, 1998.
  9. W. Gropp, E. Lusk, and A. Skjellum, Using MPI: Portable Parallel Programming with the Message-Passing Interface, 2nd ed., MIT Press, 1999.
  10. W. Gropp, E. Lusk, and R. Thakur, Using MPI-2: Advanced Features of the Message Passing Interface, MIT Press, 1999.
  11. P. J. Hatcher and M. J. Quinn, Data-Parallel Programming on MIMD Computers, MIT Press, 1991.
  12. S. Kleiman, D. Shah, and B. Smaalders, Programming with Threads, Prentice Hall, 1996.
  13. C. H. Koelbel, D. B. Loveman, R. S. Schreiber, G. L. Steele, and M. E. Zosel, The High Performance Fortran Handbook, MIT Press, 1994.
  14. B. Nichols, D. Buttlar, and J. P. Farrell, Pthreads Programming, O'Reilly & Associates, 1996.
  15. P. S. Pacheco, Parallel Programming with MPI, Morgan Kaufmann, 1997.
  16. P. S. Pacheco, An Introduction to Parallel Programming, Morgan Kaufmann, 2011.
  17. S. Prasad, Multithreading Programming Techniques, McGraw-Hill, 1997.
  18. M. Snir, S. Otto, S. Huss-Lederman, D. Walker, and J. Dongarra, MPI: The Complete Reference, Vol. 1, The MPI Core, 2nd ed., MIT Press, 1998. 1st ed. available online
  19. M. J. Sottile, T. G. Mattson, and C. E. Rasmussen, Introduction to Concurrency in Programming Languages, CRC Press, 2009.
  20. G. V. Wilson and P. Lu, eds., Parallel Programming Using C++, MIT Press, 1996.
  21. M. Wolfe, High Performance Compilers for Parallel Computing, Addison-Wesley, 1996.
Parallel Architectures
  1. R. Buyya, editor, High Performance Cluster Computing, two volumes, Prentice Hall, 1999.
  2. D. E. Culler, J. P. Singh, and A. Gupta, Parallel Computer Architecture, Morgan Kaufmann, 1998.
  3. A. L. DeCegama, Parallel Processing Architectures and VLSI Hardware, Prentice Hall, 1989.
  4. W. J. Dally and B. P. Towles, Principles and Practices of Interconnection Networks, Morgan Kaufmann, 2004.
  5. J. Duato, S. Yalamanchili, and L. Ni, Interconnection Networks: An Engineering Approach, IEEE Computer Society Press, 1997.
  6. M. Dubois, M. Annavaram, and P. Stenstrom, Parallel Computer Organization and Design, Cambridge University Press, 2012.
  7. M. J. Flynn, Computer Architecture: Pipelined and Parallel Processor Design, Jones & Bartlett, 1995.
  8. K. Hwang and F. A. Briggs, Computer Architecture and Parallel Processing, McGraw-Hill, 1984.
  9. K. Hwang, Advanced Computer Architecture: Parallelism, Scalability, Programmability, McGraw-Hill, 1993.
  10. K. Hwang and Z. Xu, Scalable Parallel Computing, McGraw-Hill, 1998.
  11. D. E. Lenoski and W.-D. Weber, Scalable Shared-Memory Multiprocessing, Morgan Kaufmann, 1995.
  12. G. F. Pfister, In Search of Clusters: The Ongoing Battle in Lowly Parallel Computing, 2nd ed., Prentice Hall, 1998.
  13. D. A. Reed and R. M. Fujimoto, Multicomputer Networks: Message-Based Parallel Processing, MIT Press, 1987.
  14. I. D. Scherson and A. S. Youssef, eds., Interconnection Networks for High-Performance Parallel Computers, IEEE Computer Society Press, 1994
  15. H. J. Siegel, Interconnection Networks for Large-Scale Parallel Processing, D. C. Heath, 1985.
  16. H. S. Stone, High-Performance Computer Architecture, 3rd ed., Addison-Wesley, 1993.
  17. A. Varma and C. S. Raghavendra, Interconnection Networks for Multiprocessors and Multicomputers: Theory and Practice, IEEE Computer Society, 1994
Parallel Algorithms (theoretical)
  1. S. G. Akl, The Design and Analysis of Parallel Algorithms, Prentice Hall, 1989.
  2. S. G. Akl, Parallel Computation: Models and Methods, Prentice Hall, 1997.
  3. K. A. Berman and J. L. Paul, Fundamentals of Sequential and Parallel Algorithms, PWS Publishing, 1997.
  4. P. Chaudhuri, Parallel Algorithms: Design and Analysis, Prentice Hall, 1992.
  5. A. Gibbons and W. Rytter, Efficient Parallel Algorithms, Cambridge University Press, 1988.
  6. A. Gibbons and P. Spirakis, eds., Lectures on Parallel Computation, Cambridge University Press, 1993.
  7. R. Greenlaw, H. J. Hoover, and W. L. Ruzzo, Limits to Parallel Computation: P-Completeness Theory, Oxford University Press, 1995.
  8. J. JaJa, An Introduction to Parallel Algorithms, Addison-Wesley, 1992.
  9. L. H. Jamieson, D. B. Gannon, and R. J. Douglass, eds., The Characteristics of Parallel Algorithms, MIT Press, 1987.
  10. J. Reif, ed., Synthesis of Parallel Algorithms, Morgan Kaufmann, 1993.
  11. J. R. Smith, The Design and Analysis of Parallel Algorithms, Oxford University Press, 1993.
  12. C. Xavier and S. S. Iyengar, Introduction to Parallel Algorithms, John Wiley & Sons, 1998.
Parallel Algorithms (practical)
  1. H. Casanova, A. Legrand, and Y. Robert, Parallel Algorithms, CRC Press, 2009.
  2. M. Cosnard and D. Trystram, Parallel Algorithms and Architectures, International Thomson, 1995.
  3. A. Grama, A. Gupta, G. Karypis, and V. Kumar, Introduction to Parallel Computing, 2nd ed., Addison-Wesley, 2003.
  4. S. Lakshmivarahan and S. K. Dhall, Analysis and Design of Parallel Algorithms: Arithmetic and Matrix Problems, McGraw-Hill, 1990.
  5. F. T. Leighton, Introduction to Parallel Algorithms and Architectures: Arrays, Trees, Hypercubes, Morgan Kaufmann, 1992.
  6. R. Miller and Q. F. Stout, Parallel Algorithms for Regular Architectures: Meshes and Pyramids, MIT Press, 1996.
  7. B. Parhami, Introduction to Parallel Processing: Algorithms and Architectures, Plenum, 1999.
  8. M. J. Quinn, Designing Efficient Algorithms for Parallel Computers, McGraw-Hill, 1987.
  9. S. Ranka and S. Sahni, Hypercube Algorithms, Springer, 1990.
Parallel Numerical Algorithms and Applications
  1. D. Bader, ed., Petascale Computing: Algorithms and Applications, Chapman & Hall/CRC, 2007.
  2. J. M. Bahi, S. Contassot-Vivier, and R. Couturier, Parallel Iterative Algorithms, Chapman & Hall/CRC, 2007.
  3. D. P. Bertsekas and J. N. Tsitsiklis, Parallel and Distributed Computation: Numerical Methods, Prentice Hall, 1989.
  4. R. H. Bisseling, Parallel Scientific Computation: A Structured Approach using BSP and MPI, Oxford University Press, 2004.
  5. A. M. Bruaset and A. Tveito, eds., Numerical Solution of Partial Differential Equations on Parallel Computers, Springer, 2006.
  6. K. Burrage, Parallel and Sequential Methods for Ordinary Differential Equations, Oxford University Press, 1995.
  7. G. F. Carey, ed., Parallel Supercomputing: Methods, Algorithms and Applications, John Wiley & Sons, 1989.
  8. Y. Censor and S. A. Zenios, Parallel Optimization: Theory, Algorithms, and Applications, Oxford University Press, 1997.
  9. E. Chu and A. George, Inside the FFT Black Box: Serial and Parallel Fast Fourier Transform Algorithms, CRC Press, 1999.
  10. C. C. Douglas, G. Haase, and U. Langer, A Tutorial on Elliptic PDE Solvers and Their Parallelization, SIAM, 2003.
  11. L. D. Fosdick, E. R. Jessup, C. J. C. Schauble, and G. Domik, An Introduction to High-Performance Scientific Computing, MIT Press, 1996.
  12. G. C. Fox, M. A. Johnson, G. A. Lyzenga, S. W. Otto, J. K. Salmon, and D. W. Walker, Solving Problems on Concurrent Processors, Prentice Hall, 1988.
  13. G. C. Fox, R. D. Williams, and P. C. Messina, Parallel Computing Works!, Morgan Kaufmann, 1994. Available online
  14. T. L. Freeman and C. Phillips, Parallel Numerical Algorithms, Prentice Hall, 1992.
  15. G. Golub and J. M. Ortega, Scientific Computing: An Introduction with Parallel Computing, Academic Press, 1993.
  16. G. E. Karniadakis and R. M. Kirby, Parallel Scientific Computing in C++ and MPI, Cambridge University Press, 2003.
  17. D. E. Keyes, A. Sameh, and V. Venkatakrishnan, eds., Parallel Numerical Algorithms, Kluwer, 1997.
  18. A. E. Koniges, ed. Industrial Strength Parallel Computing, Morgan Kaufmann, 2000.
  19. E. J. Kontoghiorghes, Parallel Algorithms for Linear Models: Numerical Methods and Estimation Problems, Kluwer, 2000.
  20. E. J. Kontoghiorghes, editor, Handbook of Parallel Computing and Statistics Chapman & Hall/CRC, 2006.
  21. J. M. Ortega and R. G. Voigt, Solution of Partial Differential Equations on Vector and Parallel Computers, SIAM, 1985.
  22. W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery, Numerical Recipes in Fortran 90, Cambridge University Press, 1996.
  23. A. Quarteroni and A. Valli, Domain Decomposition Methods for Partial Differential Equations, Oxford University Press, 1999.
  24. U. Schendel, Introduction to Numerical Methods for Parallel Computers, John Wiley & Sons, 1984.
  25. L. R. Scott, T. W. Clark, and B. Bagheri, Scientific Parallel Computing, Princeton University Press, 2005.
  26. R. W. Shonkwiler and L. Lefton, An Introduction to Parallel and Vector Scientific Computing, Cambridge University Press, 2006.
  27. B. F. Smith, P. E. Bjorstad, and W. D. Gropp, Domain Decomposition: Parallel Multilevel Methods for Elliptic Partial Differential Equations, Cambridge University Press, 1996.
  28. E. F. Van de Velde, Concurrent Scientific Computing, Springer, 1994.
  29. R. E. White, Computational Mathematics: Models, Methods, and Analysis with MATLAB and MPI, Chapman & Hall, 2004.
Parallel Matrix Computations
  1. P. Arbenz, M. Paprzycki, A. H. Sameh, and V. Sarin, eds., High Performance Algorithms for Structured Matrix Problems, Nova Science Publishers, 1999.
  2. J. W. Demmel, M. T. Heath, and H. A. van der Vorst, “Parallel Numerical Linear Algebra,” Acta Numerica, 2 (1993), pp. 111-197. Available online
  3. J. J. Dongarra, I. S. Duff, D. C. Sorenson, and H. A. van der Vorst, Numerical Linear Algebra for High-Performance Computers, SIAM, 1998.
  4. K. A. Gallivan, M. T. Heath, E. Ng, J. M. Ortega, B. W. Peyton, R. J. Plemmons, C. H. Romine, A. H. Sameh, and R. G. Voigt, Parallel Algorithms for Matrix Computations, SIAM, 1990.
  5. G. H. Golub and C. F. Van Loan, Matrix Computations, 4th ed., Johns Hopkins University Press, 2013.
  6. J. J. Modi, Parallel Algorithms and Matrix Computation, Oxford University Press, 1988.
  7. C. Murthy, K. Murthy, and S. Aluru, New Parallel Algorithms for Direct Solution of Linear Equations, John Wiley & Sons, 2001.
  8. J. M. Ortega, Introduction to Parallel and Vector Solution of Linear Systems, Plenum Press, 1988.
  9. Y. Robert, The Impact of Vector and Parallel Architectures on the Gaussian Elimination Algorithm, John Wiley & Sons, 1990.