Overview

In this series, we’ll begin by covering the basics of MPI functions in Python. We’ll learn the syntax and use cases for the most popular functions and cover the differences between blocking and non-blocking functions. From there, we’ll tackle a couple of tricky challenge problems as we learn how buffering works under the hood in different MPI implementations. We’ll also learn about the synchronized versions of the functions we’ve already seen and explore their nuances. Finally, we’ll round out the series with a common problem facing all disciplines of parallel computing whether you’re using MPI, CUDA, or something else entirely.

At the end of the day, I hope that you not only take away an understanding of the basics of MPI but also that you realize how easy it is to encounter bugs in weird edge-cases. I hope that the deep dives that we take through these lessons will prepare you to ask questions and challenge your own assumptions about the code you write. There’s almost always a simple test case you can write to either prove or disprove your hypothesis, and making sure you understand the fundamentals is critical to keeping bugs out of your code. If debugging your code in a serialized environment is challenging, then doing so in a parallelized environment is exponentially harder.

Most importantly, I hope you enjoy learning about a really interesting coding paradigm and that it gets you interested in learning more about parallel computing and seeing how you can leverage all of the hardware at your fingertips to address the tasks at hand. If you like this series (or if you’re beyond an introduction), I encourage you to take a look at the next MPI series I’ve written, which focuses on calculating key statistics in a distributed environment (even if you think this will be easy, I’d encourage you to at least ask how efficiently could you calculate the median under these circumstances). And with that, I hope you enjoy the series!