Free up workers for processing while waiting for long DB queries (uWSGI)?
I am maintaining API server for my company which runs a python flask app in uwsgi on top of nginx.
...
@app.route('/getquick', methods=["GET"])
def GET_GET_IP_DATA():
sp_final = "CALL sp_quick()"
cursor.execute(sp_final)
@app.route('/get_massive_log', methods=["POST"])
def get_massive_log():
sp_final = "CALL sp_slow()"
cursor.execute(sp_final)
...
While the first request /getquick gets processed very quickly, /get_massive_log can take up to five seconds due to a rather long and complex mySQL query. The server can handle few of these queries but starts creating broken pipe errors when called to much.
The problem is, the other /getquick requests get blocked by these long I/O requests.
My manager suggested that I use gevent to somehow free up the server to process the other requests while waiting for the mySQL queries, but I am not sure if I am looking in the correct direction.
I am using pymysql to run queries, which google seems to suggest to work with gevent on top of uwsgi, but I have not been able to produce better results with it.
I have googled for days now, and while I am trying to understand threads, concurrency, asynchronous requests, I don't know where to start digging to find a solution. Is it even possible? Any suggestions or even pointers to where to research would be greatly appreciated.
EDIT : Perhaps my questions wasn't too clear, so I'll try to restate it:
What's the best way to free up workers for processing other requests while waiting for long database queries with uwsgi?
nginx flask uwsgi gevent pymysql
asked Nov 23 '18 at 10:25
DanielDaniel
105
add a comment |
I am maintaining API server for my company which runs a python flask app in uwsgi on top of nginx.
...
@app.route('/getquick', methods=["GET"])
def GET_GET_IP_DATA():
sp_final = "CALL sp_quick()"
cursor.execute(sp_final)
@app.route('/get_massive_log', methods=["POST"])
def get_massive_log():
sp_final = "CALL sp_slow()"
cursor.execute(sp_final)
...
While the first request /getquick gets processed very quickly, /get_massive_log can take up to five seconds due to a rather long and complex mySQL query. The server can handle few of these queries but starts creating broken pipe errors when called to much.
The problem is, the other /getquick requests get blocked by these long I/O requests.
My manager suggested that I use gevent to somehow free up the server to process the other requests while waiting for the mySQL queries, but I am not sure if I am looking in the correct direction.
I am using pymysql to run queries, which google seems to suggest to work with gevent on top of uwsgi, but I have not been able to produce better results with it.
I have googled for days now, and while I am trying to understand threads, concurrency, asynchronous requests, I don't know where to start digging to find a solution. Is it even possible? Any suggestions or even pointers to where to research would be greatly appreciated.
EDIT : Perhaps my questions wasn't too clear, so I'll try to restate it:
What's the best way to free up workers for processing other requests while waiting for long database queries with uwsgi?
nginx flask uwsgi gevent pymysql
asked Nov 23 '18 at 10:25
DanielDaniel
105
add a comment |
I am maintaining API server for my company which runs a python flask app in uwsgi on top of nginx.
...
@app.route('/getquick', methods=["GET"])
def GET_GET_IP_DATA():
sp_final = "CALL sp_quick()"
cursor.execute(sp_final)
@app.route('/get_massive_log', methods=["POST"])
def get_massive_log():
sp_final = "CALL sp_slow()"
cursor.execute(sp_final)
...
While the first request /getquick gets processed very quickly, /get_massive_log can take up to five seconds due to a rather long and complex mySQL query. The server can handle few of these queries but starts creating broken pipe errors when called to much.
The problem is, the other /getquick requests get blocked by these long I/O requests.
My manager suggested that I use gevent to somehow free up the server to process the other requests while waiting for the mySQL queries, but I am not sure if I am looking in the correct direction.
I am using pymysql to run queries, which google seems to suggest to work with gevent on top of uwsgi, but I have not been able to produce better results with it.
I have googled for days now, and while I am trying to understand threads, concurrency, asynchronous requests, I don't know where to start digging to find a solution. Is it even possible? Any suggestions or even pointers to where to research would be greatly appreciated.
EDIT : Perhaps my questions wasn't too clear, so I'll try to restate it:
What's the best way to free up workers for processing other requests while waiting for long database queries with uwsgi?
nginx flask uwsgi gevent pymysql
asked Nov 23 '18 at 10:25
DanielDaniel
105
I am maintaining API server for my company which runs a python flask app in uwsgi on top of nginx.
...
@app.route('/getquick', methods=["GET"])
def GET_GET_IP_DATA():
sp_final = "CALL sp_quick()"
cursor.execute(sp_final)
@app.route('/get_massive_log', methods=["POST"])
def get_massive_log():
sp_final = "CALL sp_slow()"
cursor.execute(sp_final)
...
While the first request /getquick gets processed very quickly, /get_massive_log can take up to five seconds due to a rather long and complex mySQL query. The server can handle few of these queries but starts creating broken pipe errors when called to much.
The problem is, the other /getquick requests get blocked by these long I/O requests.
My manager suggested that I use gevent to somehow free up the server to process the other requests while waiting for the mySQL queries, but I am not sure if I am looking in the correct direction.
I am using pymysql to run queries, which google seems to suggest to work with gevent on top of uwsgi, but I have not been able to produce better results with it.
I have googled for days now, and while I am trying to understand threads, concurrency, asynchronous requests, I don't know where to start digging to find a solution. Is it even possible? Any suggestions or even pointers to where to research would be greatly appreciated.
EDIT : Perhaps my questions wasn't too clear, so I'll try to restate it:
What's the best way to free up workers for processing other requests while waiting for long database queries with uwsgi?
nginx flask uwsgi gevent pymysql
nginx flask uwsgi gevent pymysql
asked Nov 23 '18 at 10:25
DanielDaniel
105
asked Nov 23 '18 at 10:25
DanielDaniel
105
edited Nov 26 '18 at 3:02
Daniel
asked Nov 23 '18 at 10:25
DanielDaniel
105
asked Nov 23 '18 at 10:25
DanielDaniel
105
asked Nov 23 '18 at 10:25
DanielDaniel
105
105
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
You need to learn about Uwsgi offloading
Offloading is a way to optimize tiny tasks, delegating them to one or
more threads.
These threads run such tasks in a non-blocking/evented way allowing
for a huge amount of concurrency.
You can read about offloading subsystem in the docs
answered Nov 24 '18 at 15:46
Kamil NiskiKamil Niski
2,6091314
Thank you, didn't know about offloading! Although what would be the point of using gevent with uwsgi?
– Daniel
Nov 26 '18 at 2:09
Did a quick search on uwsgi offloading, there seems to be little extra information beyond the documentation (which is really brief due to the feature being apparently new and experimental).
– Daniel
Nov 26 '18 at 2:15
Okay, so offloading turns out to be for static file transfers and I don't think this would help me with waiting for pymysql queries (I/O on the machine or another server).
– Daniel
Nov 26 '18 at 2:53
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53444882%2ffree-up-workers-for-processing-while-waiting-for-long-db-queries-uwsgi%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
You need to learn about Uwsgi offloading
Offloading is a way to optimize tiny tasks, delegating them to one or
more threads.
These threads run such tasks in a non-blocking/evented way allowing
for a huge amount of concurrency.
You can read about offloading subsystem in the docs
answered Nov 24 '18 at 15:46
Kamil NiskiKamil Niski
2,6091314
Thank you, didn't know about offloading! Although what would be the point of using gevent with uwsgi?
– Daniel
Nov 26 '18 at 2:09
Did a quick search on uwsgi offloading, there seems to be little extra information beyond the documentation (which is really brief due to the feature being apparently new and experimental).
– Daniel
Nov 26 '18 at 2:15
Okay, so offloading turns out to be for static file transfers and I don't think this would help me with waiting for pymysql queries (I/O on the machine or another server).
– Daniel
Nov 26 '18 at 2:53
add a comment |
You need to learn about Uwsgi offloading
Offloading is a way to optimize tiny tasks, delegating them to one or
more threads.
These threads run such tasks in a non-blocking/evented way allowing
for a huge amount of concurrency.
You can read about offloading subsystem in the docs
answered Nov 24 '18 at 15:46
Kamil NiskiKamil Niski
2,6091314
Thank you, didn't know about offloading! Although what would be the point of using gevent with uwsgi?
– Daniel
Nov 26 '18 at 2:09
Did a quick search on uwsgi offloading, there seems to be little extra information beyond the documentation (which is really brief due to the feature being apparently new and experimental).
– Daniel
Nov 26 '18 at 2:15
Okay, so offloading turns out to be for static file transfers and I don't think this would help me with waiting for pymysql queries (I/O on the machine or another server).
– Daniel
Nov 26 '18 at 2:53
add a comment |
You need to learn about Uwsgi offloading
Offloading is a way to optimize tiny tasks, delegating them to one or
more threads.
These threads run such tasks in a non-blocking/evented way allowing
for a huge amount of concurrency.
You can read about offloading subsystem in the docs
answered Nov 24 '18 at 15:46
Kamil NiskiKamil Niski
2,6091314
You need to learn about Uwsgi offloading
Offloading is a way to optimize tiny tasks, delegating them to one or
more threads.
These threads run such tasks in a non-blocking/evented way allowing
for a huge amount of concurrency.
You can read about offloading subsystem in the docs
answered Nov 24 '18 at 15:46
Kamil NiskiKamil Niski
2,6091314
answered Nov 24 '18 at 15:46
Kamil NiskiKamil Niski
2,6091314
answered Nov 24 '18 at 15:46
Kamil NiskiKamil Niski
2,6091314
answered Nov 24 '18 at 15:46
Kamil NiskiKamil Niski
2,6091314
2,6091314
Thank you, didn't know about offloading! Although what would be the point of using gevent with uwsgi?
– Daniel
Nov 26 '18 at 2:09
Did a quick search on uwsgi offloading, there seems to be little extra information beyond the documentation (which is really brief due to the feature being apparently new and experimental).
– Daniel
Nov 26 '18 at 2:15
Okay, so offloading turns out to be for static file transfers and I don't think this would help me with waiting for pymysql queries (I/O on the machine or another server).
– Daniel
Nov 26 '18 at 2:53
add a comment |
Thank you, didn't know about offloading! Although what would be the point of using gevent with uwsgi?
– Daniel
Nov 26 '18 at 2:09
Did a quick search on uwsgi offloading, there seems to be little extra information beyond the documentation (which is really brief due to the feature being apparently new and experimental).
– Daniel
Nov 26 '18 at 2:15
Okay, so offloading turns out to be for static file transfers and I don't think this would help me with waiting for pymysql queries (I/O on the machine or another server).
– Daniel
Nov 26 '18 at 2:53
Thank you, didn't know about offloading! Although what would be the point of using gevent with uwsgi?
– Daniel
Nov 26 '18 at 2:09
Thank you, didn't know about offloading! Although what would be the point of using gevent with uwsgi?
– Daniel
Nov 26 '18 at 2:09
Did a quick search on uwsgi offloading, there seems to be little extra information beyond the documentation (which is really brief due to the feature being apparently new and experimental).
– Daniel
Nov 26 '18 at 2:15
Did a quick search on uwsgi offloading, there seems to be little extra information beyond the documentation (which is really brief due to the feature being apparently new and experimental).
– Daniel
Nov 26 '18 at 2:15
Okay, so offloading turns out to be for static file transfers and I don't think this would help me with waiting for pymysql queries (I/O on the machine or another server).
– Daniel
Nov 26 '18 at 2:53
Okay, so offloading turns out to be for static file transfers and I don't think this would help me with waiting for pymysql queries (I/O on the machine or another server).
– Daniel
Nov 26 '18 at 2:53
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53444882%2ffree-up-workers-for-processing-while-waiting-for-long-db-queries-uwsgi%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
