When to turn off Transparent Huge Pages for redis
According to redis docs, it's advisable to disable Transparent Huge Pages.
Would the guidance be the same if the machine was shared between the redis server and the application.
Moreover, for other technologies, I've also read guidance that THP should be disabled for all production environments when setting up the server. Is this kind of pre-emptiveness applicable to redis as well, or one must first strictly monitor latency issues before deciding to turn off THP?
redis
asked Mar 4 '17 at 2:51
Hassan BaigHassan Baig
4,31983181
add a comment |
According to redis docs, it's advisable to disable Transparent Huge Pages.
Would the guidance be the same if the machine was shared between the redis server and the application.
Moreover, for other technologies, I've also read guidance that THP should be disabled for all production environments when setting up the server. Is this kind of pre-emptiveness applicable to redis as well, or one must first strictly monitor latency issues before deciding to turn off THP?
redis
asked Mar 4 '17 at 2:51
Hassan BaigHassan Baig
4,31983181
add a comment |
According to redis docs, it's advisable to disable Transparent Huge Pages.
Would the guidance be the same if the machine was shared between the redis server and the application.
Moreover, for other technologies, I've also read guidance that THP should be disabled for all production environments when setting up the server. Is this kind of pre-emptiveness applicable to redis as well, or one must first strictly monitor latency issues before deciding to turn off THP?
redis
asked Mar 4 '17 at 2:51
Hassan BaigHassan Baig
4,31983181
According to redis docs, it's advisable to disable Transparent Huge Pages.
Would the guidance be the same if the machine was shared between the redis server and the application.
Moreover, for other technologies, I've also read guidance that THP should be disabled for all production environments when setting up the server. Is this kind of pre-emptiveness applicable to redis as well, or one must first strictly monitor latency issues before deciding to turn off THP?
redis
redis
asked Mar 4 '17 at 2:51
Hassan BaigHassan Baig
4,31983181
asked Mar 4 '17 at 2:51
Hassan BaigHassan Baig
4,31983181
asked Mar 4 '17 at 2:51
Hassan BaigHassan Baig
4,31983181
asked Mar 4 '17 at 2:51
Hassan BaigHassan Baig
4,31983181
asked Mar 4 '17 at 2:51
Hassan BaigHassan Baig
4,31983181
4,31983181
add a comment |
add a comment |
3 Answers
3
active
oldest
votes
Turn it off. The problem lies in how THP shifts memory around to try and keep or create contiguous pages. Some applications can tolerate this, most databases cannot and it causes intermittent performance problems, some pretty bad. This is not unique to Redis by any means.
For your application, especially if it is JAVA, set up real HugePages and leave the transparent variety out of it. If you do that just make sure you alocate memory correctly for the app and redis. Though I have to say, I probably would not recommend running both the app and redis on the same instance/server/vm.
answered Mar 4 '17 at 5:13
KirkKirk
2,0261120
Well to be exact, I've got a Django (Python) application, although I fathom that won't change anything vis-a-vis your answer, would it?
– Hassan Baig
Mar 4 '17 at 11:22
No, it would not.
– Kirk
Mar 5 '17 at 17:31
Thanks Kirk! Btw to further clarify, I need to do bothecho never > /sys/kernel/mm/transparent_hugepage/enabledandecho never > /sys/kernel/mm/transparent_hugepage/defrag, correct?
– Hassan Baig
Mar 6 '17 at 0:09
Moreover, in your experience, does req/s throughput suffer as a result of turning off THP?
– Hassan Baig
Mar 6 '17 at 1:10
More than likely, you need both places. You will need to have some kind of init script that starts on boot that will do it every time.
– Kirk
Mar 6 '17 at 21:27
add a comment |
The overhead THP imposes occurs only during memory allocation, because of defragmentation costs.
If your redis instance has a (near-)constant memory footprint, you can only benefit from THP. Same applies to java or any other long-lived service that does its own memory management. Pre-allocate memory once and benefit.
answered Nov 25 '18 at 17:42
milanmilan
1,36111630
add a comment |
why playing such echo-games when there is a kernel-param you can boot with?
transparent_hugepage=never
answered Dec 12 '17 at 22:29
Harald ReindlHarald Reindl
1
not sure why this answer got downvoted. Here's another reference unix.stackexchange.com/a/99172/29023
– chachan
Jan 8 '18 at 17:22
perhaps you didn't need to call the other answer "echo-games" and show more details about what you wanted to say as the link above
– chachan
Jan 8 '18 at 17:24
sorry, but when one is not capable to copy&paste "transparent_hugepage=never" into Google or don't know what a kernel parameter is he's not the audience anyways
– Harald Reindl
Jan 9 '18 at 18:00
Simple. The OP raised a number of concerns and none of them were addressed by this answer.
– Amir
Aug 19 '18 at 11:39
i refered to "echo never > /sys/kernel" which is not persistent while "transparent_hugepage=never" as kenerl param is a) persistent and b) as early as possible set
– Harald Reindl
Aug 20 '18 at 12:05
|
show 1 more comment
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f42591511%2fwhen-to-turn-off-transparent-huge-pages-for-redis%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
3 Answers
3
active
oldest
votes
3 Answers
3
active
oldest
votes
active
oldest
votes
active
oldest
votes
Turn it off. The problem lies in how THP shifts memory around to try and keep or create contiguous pages. Some applications can tolerate this, most databases cannot and it causes intermittent performance problems, some pretty bad. This is not unique to Redis by any means.
For your application, especially if it is JAVA, set up real HugePages and leave the transparent variety out of it. If you do that just make sure you alocate memory correctly for the app and redis. Though I have to say, I probably would not recommend running both the app and redis on the same instance/server/vm.
answered Mar 4 '17 at 5:13
KirkKirk
2,0261120
Well to be exact, I've got a Django (Python) application, although I fathom that won't change anything vis-a-vis your answer, would it?
– Hassan Baig
Mar 4 '17 at 11:22
No, it would not.
– Kirk
Mar 5 '17 at 17:31
Thanks Kirk! Btw to further clarify, I need to do bothecho never > /sys/kernel/mm/transparent_hugepage/enabledandecho never > /sys/kernel/mm/transparent_hugepage/defrag, correct?
– Hassan Baig
Mar 6 '17 at 0:09
Moreover, in your experience, does req/s throughput suffer as a result of turning off THP?
– Hassan Baig
Mar 6 '17 at 1:10
More than likely, you need both places. You will need to have some kind of init script that starts on boot that will do it every time.
– Kirk
Mar 6 '17 at 21:27
add a comment |
Turn it off. The problem lies in how THP shifts memory around to try and keep or create contiguous pages. Some applications can tolerate this, most databases cannot and it causes intermittent performance problems, some pretty bad. This is not unique to Redis by any means.
For your application, especially if it is JAVA, set up real HugePages and leave the transparent variety out of it. If you do that just make sure you alocate memory correctly for the app and redis. Though I have to say, I probably would not recommend running both the app and redis on the same instance/server/vm.
answered Mar 4 '17 at 5:13
KirkKirk
2,0261120
Well to be exact, I've got a Django (Python) application, although I fathom that won't change anything vis-a-vis your answer, would it?
– Hassan Baig
Mar 4 '17 at 11:22
No, it would not.
– Kirk
Mar 5 '17 at 17:31
Thanks Kirk! Btw to further clarify, I need to do bothecho never > /sys/kernel/mm/transparent_hugepage/enabledandecho never > /sys/kernel/mm/transparent_hugepage/defrag, correct?
– Hassan Baig
Mar 6 '17 at 0:09
Moreover, in your experience, does req/s throughput suffer as a result of turning off THP?
– Hassan Baig
Mar 6 '17 at 1:10
More than likely, you need both places. You will need to have some kind of init script that starts on boot that will do it every time.
– Kirk
Mar 6 '17 at 21:27
add a comment |
Turn it off. The problem lies in how THP shifts memory around to try and keep or create contiguous pages. Some applications can tolerate this, most databases cannot and it causes intermittent performance problems, some pretty bad. This is not unique to Redis by any means.
For your application, especially if it is JAVA, set up real HugePages and leave the transparent variety out of it. If you do that just make sure you alocate memory correctly for the app and redis. Though I have to say, I probably would not recommend running both the app and redis on the same instance/server/vm.
answered Mar 4 '17 at 5:13
KirkKirk
2,0261120
Turn it off. The problem lies in how THP shifts memory around to try and keep or create contiguous pages. Some applications can tolerate this, most databases cannot and it causes intermittent performance problems, some pretty bad. This is not unique to Redis by any means.
For your application, especially if it is JAVA, set up real HugePages and leave the transparent variety out of it. If you do that just make sure you alocate memory correctly for the app and redis. Though I have to say, I probably would not recommend running both the app and redis on the same instance/server/vm.
answered Mar 4 '17 at 5:13
KirkKirk
2,0261120
answered Mar 4 '17 at 5:13
KirkKirk
2,0261120
answered Mar 4 '17 at 5:13
KirkKirk
2,0261120
answered Mar 4 '17 at 5:13
KirkKirk
2,0261120
2,0261120
Well to be exact, I've got a Django (Python) application, although I fathom that won't change anything vis-a-vis your answer, would it?
– Hassan Baig
Mar 4 '17 at 11:22
No, it would not.
– Kirk
Mar 5 '17 at 17:31
Thanks Kirk! Btw to further clarify, I need to do bothecho never > /sys/kernel/mm/transparent_hugepage/enabledandecho never > /sys/kernel/mm/transparent_hugepage/defrag, correct?
– Hassan Baig
Mar 6 '17 at 0:09
Moreover, in your experience, does req/s throughput suffer as a result of turning off THP?
– Hassan Baig
Mar 6 '17 at 1:10
More than likely, you need both places. You will need to have some kind of init script that starts on boot that will do it every time.
– Kirk
Mar 6 '17 at 21:27
add a comment |
Well to be exact, I've got a Django (Python) application, although I fathom that won't change anything vis-a-vis your answer, would it?
– Hassan Baig
Mar 4 '17 at 11:22
No, it would not.
– Kirk
Mar 5 '17 at 17:31
Thanks Kirk! Btw to further clarify, I need to do bothecho never > /sys/kernel/mm/transparent_hugepage/enabledandecho never > /sys/kernel/mm/transparent_hugepage/defrag, correct?
– Hassan Baig
Mar 6 '17 at 0:09
Moreover, in your experience, does req/s throughput suffer as a result of turning off THP?
– Hassan Baig
Mar 6 '17 at 1:10
More than likely, you need both places. You will need to have some kind of init script that starts on boot that will do it every time.
– Kirk
Mar 6 '17 at 21:27
Well to be exact, I've got a Django (Python) application, although I fathom that won't change anything vis-a-vis your answer, would it?
– Hassan Baig
Mar 4 '17 at 11:22
Well to be exact, I've got a Django (Python) application, although I fathom that won't change anything vis-a-vis your answer, would it?
– Hassan Baig
Mar 4 '17 at 11:22
No, it would not.
– Kirk
Mar 5 '17 at 17:31
No, it would not.
– Kirk
Mar 5 '17 at 17:31
Thanks Kirk! Btw to further clarify, I need to do both
echo never > /sys/kernel/mm/transparent_hugepage/enabled and echo never > /sys/kernel/mm/transparent_hugepage/defrag, correct?– Hassan Baig
Mar 6 '17 at 0:09
Thanks Kirk! Btw to further clarify, I need to do both
echo never > /sys/kernel/mm/transparent_hugepage/enabled and echo never > /sys/kernel/mm/transparent_hugepage/defrag, correct?– Hassan Baig
Mar 6 '17 at 0:09
Moreover, in your experience, does req/s throughput suffer as a result of turning off THP?
– Hassan Baig
Mar 6 '17 at 1:10
Moreover, in your experience, does req/s throughput suffer as a result of turning off THP?
– Hassan Baig
Mar 6 '17 at 1:10
More than likely, you need both places. You will need to have some kind of init script that starts on boot that will do it every time.
– Kirk
Mar 6 '17 at 21:27
More than likely, you need both places. You will need to have some kind of init script that starts on boot that will do it every time.
– Kirk
Mar 6 '17 at 21:27
add a comment |
The overhead THP imposes occurs only during memory allocation, because of defragmentation costs.
If your redis instance has a (near-)constant memory footprint, you can only benefit from THP. Same applies to java or any other long-lived service that does its own memory management. Pre-allocate memory once and benefit.
answered Nov 25 '18 at 17:42
milanmilan
1,36111630
add a comment |
The overhead THP imposes occurs only during memory allocation, because of defragmentation costs.
If your redis instance has a (near-)constant memory footprint, you can only benefit from THP. Same applies to java or any other long-lived service that does its own memory management. Pre-allocate memory once and benefit.
answered Nov 25 '18 at 17:42
milanmilan
1,36111630
add a comment |
The overhead THP imposes occurs only during memory allocation, because of defragmentation costs.
If your redis instance has a (near-)constant memory footprint, you can only benefit from THP. Same applies to java or any other long-lived service that does its own memory management. Pre-allocate memory once and benefit.
answered Nov 25 '18 at 17:42
milanmilan
1,36111630
The overhead THP imposes occurs only during memory allocation, because of defragmentation costs.
If your redis instance has a (near-)constant memory footprint, you can only benefit from THP. Same applies to java or any other long-lived service that does its own memory management. Pre-allocate memory once and benefit.
answered Nov 25 '18 at 17:42
milanmilan
1,36111630
edited Nov 26 '18 at 16:56
answered Nov 25 '18 at 17:42
milanmilan
1,36111630
answered Nov 25 '18 at 17:42
milanmilan
1,36111630
answered Nov 25 '18 at 17:42
milanmilan
1,36111630
1,36111630
add a comment |
add a comment |
why playing such echo-games when there is a kernel-param you can boot with?
transparent_hugepage=never
answered Dec 12 '17 at 22:29
Harald ReindlHarald Reindl
1
not sure why this answer got downvoted. Here's another reference unix.stackexchange.com/a/99172/29023
– chachan
Jan 8 '18 at 17:22
perhaps you didn't need to call the other answer "echo-games" and show more details about what you wanted to say as the link above
– chachan
Jan 8 '18 at 17:24
sorry, but when one is not capable to copy&paste "transparent_hugepage=never" into Google or don't know what a kernel parameter is he's not the audience anyways
– Harald Reindl
Jan 9 '18 at 18:00
Simple. The OP raised a number of concerns and none of them were addressed by this answer.
– Amir
Aug 19 '18 at 11:39
i refered to "echo never > /sys/kernel" which is not persistent while "transparent_hugepage=never" as kenerl param is a) persistent and b) as early as possible set
– Harald Reindl
Aug 20 '18 at 12:05
|
show 1 more comment
why playing such echo-games when there is a kernel-param you can boot with?
transparent_hugepage=never
answered Dec 12 '17 at 22:29
Harald ReindlHarald Reindl
1
not sure why this answer got downvoted. Here's another reference unix.stackexchange.com/a/99172/29023
– chachan
Jan 8 '18 at 17:22
perhaps you didn't need to call the other answer "echo-games" and show more details about what you wanted to say as the link above
– chachan
Jan 8 '18 at 17:24
sorry, but when one is not capable to copy&paste "transparent_hugepage=never" into Google or don't know what a kernel parameter is he's not the audience anyways
– Harald Reindl
Jan 9 '18 at 18:00
Simple. The OP raised a number of concerns and none of them were addressed by this answer.
– Amir
Aug 19 '18 at 11:39
i refered to "echo never > /sys/kernel" which is not persistent while "transparent_hugepage=never" as kenerl param is a) persistent and b) as early as possible set
– Harald Reindl
Aug 20 '18 at 12:05
|
show 1 more comment
why playing such echo-games when there is a kernel-param you can boot with?
transparent_hugepage=never
answered Dec 12 '17 at 22:29
Harald ReindlHarald Reindl
1
why playing such echo-games when there is a kernel-param you can boot with?
transparent_hugepage=never
answered Dec 12 '17 at 22:29
Harald ReindlHarald Reindl
1
answered Dec 12 '17 at 22:29
Harald ReindlHarald Reindl
1
answered Dec 12 '17 at 22:29
Harald ReindlHarald Reindl
1
answered Dec 12 '17 at 22:29
Harald ReindlHarald Reindl
1
1
not sure why this answer got downvoted. Here's another reference unix.stackexchange.com/a/99172/29023
– chachan
Jan 8 '18 at 17:22
perhaps you didn't need to call the other answer "echo-games" and show more details about what you wanted to say as the link above
– chachan
Jan 8 '18 at 17:24
sorry, but when one is not capable to copy&paste "transparent_hugepage=never" into Google or don't know what a kernel parameter is he's not the audience anyways
– Harald Reindl
Jan 9 '18 at 18:00
Simple. The OP raised a number of concerns and none of them were addressed by this answer.
– Amir
Aug 19 '18 at 11:39
i refered to "echo never > /sys/kernel" which is not persistent while "transparent_hugepage=never" as kenerl param is a) persistent and b) as early as possible set
– Harald Reindl
Aug 20 '18 at 12:05
|
show 1 more comment
not sure why this answer got downvoted. Here's another reference unix.stackexchange.com/a/99172/29023
– chachan
Jan 8 '18 at 17:22
perhaps you didn't need to call the other answer "echo-games" and show more details about what you wanted to say as the link above
– chachan
Jan 8 '18 at 17:24
sorry, but when one is not capable to copy&paste "transparent_hugepage=never" into Google or don't know what a kernel parameter is he's not the audience anyways
– Harald Reindl
Jan 9 '18 at 18:00
Simple. The OP raised a number of concerns and none of them were addressed by this answer.
– Amir
Aug 19 '18 at 11:39
i refered to "echo never > /sys/kernel" which is not persistent while "transparent_hugepage=never" as kenerl param is a) persistent and b) as early as possible set
– Harald Reindl
Aug 20 '18 at 12:05
not sure why this answer got downvoted. Here's another reference unix.stackexchange.com/a/99172/29023
– chachan
Jan 8 '18 at 17:22
not sure why this answer got downvoted. Here's another reference unix.stackexchange.com/a/99172/29023
– chachan
Jan 8 '18 at 17:22
perhaps you didn't need to call the other answer "echo-games" and show more details about what you wanted to say as the link above
– chachan
Jan 8 '18 at 17:24
perhaps you didn't need to call the other answer "echo-games" and show more details about what you wanted to say as the link above
– chachan
Jan 8 '18 at 17:24
sorry, but when one is not capable to copy&paste "transparent_hugepage=never" into Google or don't know what a kernel parameter is he's not the audience anyways
– Harald Reindl
Jan 9 '18 at 18:00
sorry, but when one is not capable to copy&paste "transparent_hugepage=never" into Google or don't know what a kernel parameter is he's not the audience anyways
– Harald Reindl
Jan 9 '18 at 18:00
Simple. The OP raised a number of concerns and none of them were addressed by this answer.
– Amir
Aug 19 '18 at 11:39
Simple. The OP raised a number of concerns and none of them were addressed by this answer.
– Amir
Aug 19 '18 at 11:39
i refered to "echo never > /sys/kernel" which is not persistent while "transparent_hugepage=never" as kenerl param is a) persistent and b) as early as possible set
– Harald Reindl
Aug 20 '18 at 12:05
i refered to "echo never > /sys/kernel" which is not persistent while "transparent_hugepage=never" as kenerl param is a) persistent and b) as early as possible set
– Harald Reindl
Aug 20 '18 at 12:05
|
show 1 more comment
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f42591511%2fwhen-to-turn-off-transparent-huge-pages-for-redis%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
