MPI send/recv latency jump at 32 MiB message size
For a university project I'm currently working on a slurm supercomputer cluster and have written a number of C programs using MPI.
While profiling one of them I have observed that the time elapsed between an MPI_Send and an accompanying MPI_Recv operation is a mostly linear function of the message length. However, at around 32 MiB there is a sudden jump in latency from around 10ms to around 20ms. This happens both for two processes on the same node and two processes on separate nodes.
Now I would like to find out why this happens. I'm aware that this is not an MPI intrinsic phenomenon but must be related to the underlying hardware setup, but I'm not sure where to begin looking for an explanation. What are some possible explanations for this and how could I check whether they apply in my case?
mpi hpc
asked Nov 20 at 21:09
Peter
32219
add a comment |
For a university project I'm currently working on a slurm supercomputer cluster and have written a number of C programs using MPI.
While profiling one of them I have observed that the time elapsed between an MPI_Send and an accompanying MPI_Recv operation is a mostly linear function of the message length. However, at around 32 MiB there is a sudden jump in latency from around 10ms to around 20ms. This happens both for two processes on the same node and two processes on separate nodes.
Now I would like to find out why this happens. I'm aware that this is not an MPI intrinsic phenomenon but must be related to the underlying hardware setup, but I'm not sure where to begin looking for an explanation. What are some possible explanations for this and how could I check whether they apply in my case?
mpi hpc
asked Nov 20 at 21:09
Peter
32219
Is there a spike or a jump ? I mean, does the latency for messages larger than 32MiB go down again, or stay high ? For anything better than guesses and waffle you will have to tell us what hardware your cluster comprises, especially what kind of interconnect it has.
– High Performance Mark
Nov 21 at 6:32
It's a jump, you're right, spike is the wrong word. I'm not completely sure which hardware specs are relevant here but I know that the nodes I'm working on are comprised of around two dozen somewhat current Intel Xeon processors and Infiniband (FDR) is used for interconnect.
– Peter
Nov 21 at 7:46
You should first measure the latency/bandwidth with a trusted benchmark such as IMB (from Intel) or the OSU benchmark.
– Gilles Gouaillardet
Nov 21 at 7:57
add a comment |
For a university project I'm currently working on a slurm supercomputer cluster and have written a number of C programs using MPI.
While profiling one of them I have observed that the time elapsed between an MPI_Send and an accompanying MPI_Recv operation is a mostly linear function of the message length. However, at around 32 MiB there is a sudden jump in latency from around 10ms to around 20ms. This happens both for two processes on the same node and two processes on separate nodes.
Now I would like to find out why this happens. I'm aware that this is not an MPI intrinsic phenomenon but must be related to the underlying hardware setup, but I'm not sure where to begin looking for an explanation. What are some possible explanations for this and how could I check whether they apply in my case?
mpi hpc
asked Nov 20 at 21:09
Peter
32219
For a university project I'm currently working on a slurm supercomputer cluster and have written a number of C programs using MPI.
While profiling one of them I have observed that the time elapsed between an MPI_Send and an accompanying MPI_Recv operation is a mostly linear function of the message length. However, at around 32 MiB there is a sudden jump in latency from around 10ms to around 20ms. This happens both for two processes on the same node and two processes on separate nodes.
Now I would like to find out why this happens. I'm aware that this is not an MPI intrinsic phenomenon but must be related to the underlying hardware setup, but I'm not sure where to begin looking for an explanation. What are some possible explanations for this and how could I check whether they apply in my case?
mpi hpc
mpi hpc
asked Nov 20 at 21:09
Peter
32219
asked Nov 20 at 21:09
Peter
32219
edited Nov 21 at 7:46
asked Nov 20 at 21:09
Peter
32219
asked Nov 20 at 21:09
Peter
32219
asked Nov 20 at 21:09
Peter
32219
32219
Is there a spike or a jump ? I mean, does the latency for messages larger than 32MiB go down again, or stay high ? For anything better than guesses and waffle you will have to tell us what hardware your cluster comprises, especially what kind of interconnect it has.
– High Performance Mark
Nov 21 at 6:32
It's a jump, you're right, spike is the wrong word. I'm not completely sure which hardware specs are relevant here but I know that the nodes I'm working on are comprised of around two dozen somewhat current Intel Xeon processors and Infiniband (FDR) is used for interconnect.
– Peter
Nov 21 at 7:46
You should first measure the latency/bandwidth with a trusted benchmark such as IMB (from Intel) or the OSU benchmark.
– Gilles Gouaillardet
Nov 21 at 7:57
add a comment |
Is there a spike or a jump ? I mean, does the latency for messages larger than 32MiB go down again, or stay high ? For anything better than guesses and waffle you will have to tell us what hardware your cluster comprises, especially what kind of interconnect it has.
– High Performance Mark
Nov 21 at 6:32
It's a jump, you're right, spike is the wrong word. I'm not completely sure which hardware specs are relevant here but I know that the nodes I'm working on are comprised of around two dozen somewhat current Intel Xeon processors and Infiniband (FDR) is used for interconnect.
– Peter
Nov 21 at 7:46
You should first measure the latency/bandwidth with a trusted benchmark such as IMB (from Intel) or the OSU benchmark.
– Gilles Gouaillardet
Nov 21 at 7:57
Is there a spike or a jump ? I mean, does the latency for messages larger than 32MiB go down again, or stay high ? For anything better than guesses and waffle you will have to tell us what hardware your cluster comprises, especially what kind of interconnect it has.
– High Performance Mark
Nov 21 at 6:32
Is there a spike or a jump ? I mean, does the latency for messages larger than 32MiB go down again, or stay high ? For anything better than guesses and waffle you will have to tell us what hardware your cluster comprises, especially what kind of interconnect it has.
– High Performance Mark
Nov 21 at 6:32
It's a jump, you're right, spike is the wrong word. I'm not completely sure which hardware specs are relevant here but I know that the nodes I'm working on are comprised of around two dozen somewhat current Intel Xeon processors and Infiniband (FDR) is used for interconnect.
– Peter
Nov 21 at 7:46
It's a jump, you're right, spike is the wrong word. I'm not completely sure which hardware specs are relevant here but I know that the nodes I'm working on are comprised of around two dozen somewhat current Intel Xeon processors and Infiniband (FDR) is used for interconnect.
– Peter
Nov 21 at 7:46
You should first measure the latency/bandwidth with a trusted benchmark such as IMB (from Intel) or the OSU benchmark.
– Gilles Gouaillardet
Nov 21 at 7:57
You should first measure the latency/bandwidth with a trusted benchmark such as IMB (from Intel) or the OSU benchmark.
– Gilles Gouaillardet
Nov 21 at 7:57
add a comment |
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53401577%2fmpi-send-recv-latency-jump-at-32-mib-message-size%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53401577%2fmpi-send-recv-latency-jump-at-32-mib-message-size%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Is there a spike or a jump ? I mean, does the latency for messages larger than 32MiB go down again, or stay high ? For anything better than guesses and waffle you will have to tell us what hardware your cluster comprises, especially what kind of interconnect it has.
– High Performance Mark
Nov 21 at 6:32
It's a jump, you're right, spike is the wrong word. I'm not completely sure which hardware specs are relevant here but I know that the nodes I'm working on are comprised of around two dozen somewhat current Intel Xeon processors and Infiniband (FDR) is used for interconnect.
– Peter
Nov 21 at 7:46
You should first measure the latency/bandwidth with a trusted benchmark such as IMB (from Intel) or the OSU benchmark.
– Gilles Gouaillardet
Nov 21 at 7:57