Curl -u in scrapy
How to do this curl on scrapy?
curl –i -u account_id:api_key
"https://xecdapi.xe.com/v1/convert_from.json/?from=USD&to=CAD,EUR
&amount=110.23"
python curl web-scraping scrapy scrapy-spider
asked Mar 16 '17 at 10:58
Aminah NurainiAminah Nuraini
6,85744863
add a comment |
How to do this curl on scrapy?
curl –i -u account_id:api_key
"https://xecdapi.xe.com/v1/convert_from.json/?from=USD&to=CAD,EUR
&amount=110.23"
python curl web-scraping scrapy scrapy-spider
asked Mar 16 '17 at 10:58
Aminah NurainiAminah Nuraini
6,85744863
add a comment |
How to do this curl on scrapy?
curl –i -u account_id:api_key
"https://xecdapi.xe.com/v1/convert_from.json/?from=USD&to=CAD,EUR
&amount=110.23"
python curl web-scraping scrapy scrapy-spider
asked Mar 16 '17 at 10:58
Aminah NurainiAminah Nuraini
6,85744863
How to do this curl on scrapy?
curl –i -u account_id:api_key
"https://xecdapi.xe.com/v1/convert_from.json/?from=USD&to=CAD,EUR
&amount=110.23"
python curl web-scraping scrapy scrapy-spider
python curl web-scraping scrapy scrapy-spider
asked Mar 16 '17 at 10:58
Aminah NurainiAminah Nuraini
6,85744863
asked Mar 16 '17 at 10:58
Aminah NurainiAminah Nuraini
6,85744863
edited Nov 24 '18 at 14:20
Aminah Nuraini
asked Mar 16 '17 at 10:58
Aminah NurainiAminah Nuraini
6,85744863
asked Mar 16 '17 at 10:58
Aminah NurainiAminah Nuraini
6,85744863
asked Mar 16 '17 at 10:58
Aminah NurainiAminah Nuraini
6,85744863
6,85744863
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
You can use scrapy fetch command:
scrapy fetch http://stackoverflow.com --nolog > output.html
To use authentication you can try passing credentials via url itself:
scrapy fetch "http://username:password@stackoverflow.com" --nolog > output.html
answered Mar 16 '17 at 12:31
GranitosaurusGranitosaurus
11.3k22345
I want to do it inside the spider
– Aminah Nuraini
Mar 16 '17 at 12:32
And I get 500 Internal Server Error again using Scrapy fetch
– Aminah Nuraini
Mar 16 '17 at 12:33
@AminahNuraini but that not what it says in your question; I see you have another question open that does indicate that you want to do it in spider, so maybe you should make it more clear and keep one question open that describes your issue fully.
– Granitosaurus
Mar 16 '17 at 12:35
Doesn't matter anyway. Scrapy fetch and inside spider gives the same result anyway
– Aminah Nuraini
Mar 16 '17 at 12:36
add a comment |
Found the answer!!!
Scrapy does not allow HTTP Basic Auth via URL or Authorization header because of a bug in Twisted Twisted HTTPS Client.
We have to use a not so famous middleware, HTTPAuthMiddleware https://github.com/scrapy/scrapy/blob/master/docs/topics/downloader-middleware.rst#httpauthmiddleware
To use it, activate the middleware in the settings.py like this:
DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware': 811,
}
And add these attributes to the spider.
from scrapy.spiders import CrawlSpider
class SomeIntranetSiteSpider(CrawlSpider):
http_user = 'someuser'
http_pass = 'somepass'
name = 'intranet.example.com'
# .. rest of the spider code omitted ...
edited May 23 '17 at 10:30
Community♦
11
answered Mar 16 '17 at 13:10
Aminah NurainiAminah Nuraini
6,85744863
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f42832143%2fcurl-u-in-scrapy%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
You can use scrapy fetch command:
scrapy fetch http://stackoverflow.com --nolog > output.html
To use authentication you can try passing credentials via url itself:
scrapy fetch "http://username:password@stackoverflow.com" --nolog > output.html
answered Mar 16 '17 at 12:31
GranitosaurusGranitosaurus
11.3k22345
I want to do it inside the spider
– Aminah Nuraini
Mar 16 '17 at 12:32
And I get 500 Internal Server Error again using Scrapy fetch
– Aminah Nuraini
Mar 16 '17 at 12:33
@AminahNuraini but that not what it says in your question; I see you have another question open that does indicate that you want to do it in spider, so maybe you should make it more clear and keep one question open that describes your issue fully.
– Granitosaurus
Mar 16 '17 at 12:35
Doesn't matter anyway. Scrapy fetch and inside spider gives the same result anyway
– Aminah Nuraini
Mar 16 '17 at 12:36
add a comment |
You can use scrapy fetch command:
scrapy fetch http://stackoverflow.com --nolog > output.html
To use authentication you can try passing credentials via url itself:
scrapy fetch "http://username:password@stackoverflow.com" --nolog > output.html
answered Mar 16 '17 at 12:31
GranitosaurusGranitosaurus
11.3k22345
I want to do it inside the spider
– Aminah Nuraini
Mar 16 '17 at 12:32
And I get 500 Internal Server Error again using Scrapy fetch
– Aminah Nuraini
Mar 16 '17 at 12:33
@AminahNuraini but that not what it says in your question; I see you have another question open that does indicate that you want to do it in spider, so maybe you should make it more clear and keep one question open that describes your issue fully.
– Granitosaurus
Mar 16 '17 at 12:35
Doesn't matter anyway. Scrapy fetch and inside spider gives the same result anyway
– Aminah Nuraini
Mar 16 '17 at 12:36
add a comment |
You can use scrapy fetch command:
scrapy fetch http://stackoverflow.com --nolog > output.html
To use authentication you can try passing credentials via url itself:
scrapy fetch "http://username:password@stackoverflow.com" --nolog > output.html
answered Mar 16 '17 at 12:31
GranitosaurusGranitosaurus
11.3k22345
You can use scrapy fetch command:
scrapy fetch http://stackoverflow.com --nolog > output.html
To use authentication you can try passing credentials via url itself:
scrapy fetch "http://username:password@stackoverflow.com" --nolog > output.html
answered Mar 16 '17 at 12:31
GranitosaurusGranitosaurus
11.3k22345
answered Mar 16 '17 at 12:31
GranitosaurusGranitosaurus
11.3k22345
answered Mar 16 '17 at 12:31
GranitosaurusGranitosaurus
11.3k22345
answered Mar 16 '17 at 12:31
GranitosaurusGranitosaurus
11.3k22345
11.3k22345
I want to do it inside the spider
– Aminah Nuraini
Mar 16 '17 at 12:32
And I get 500 Internal Server Error again using Scrapy fetch
– Aminah Nuraini
Mar 16 '17 at 12:33
@AminahNuraini but that not what it says in your question; I see you have another question open that does indicate that you want to do it in spider, so maybe you should make it more clear and keep one question open that describes your issue fully.
– Granitosaurus
Mar 16 '17 at 12:35
Doesn't matter anyway. Scrapy fetch and inside spider gives the same result anyway
– Aminah Nuraini
Mar 16 '17 at 12:36
add a comment |
I want to do it inside the spider
– Aminah Nuraini
Mar 16 '17 at 12:32
And I get 500 Internal Server Error again using Scrapy fetch
– Aminah Nuraini
Mar 16 '17 at 12:33
@AminahNuraini but that not what it says in your question; I see you have another question open that does indicate that you want to do it in spider, so maybe you should make it more clear and keep one question open that describes your issue fully.
– Granitosaurus
Mar 16 '17 at 12:35
Doesn't matter anyway. Scrapy fetch and inside spider gives the same result anyway
– Aminah Nuraini
Mar 16 '17 at 12:36
I want to do it inside the spider
– Aminah Nuraini
Mar 16 '17 at 12:32
I want to do it inside the spider
– Aminah Nuraini
Mar 16 '17 at 12:32
And I get 500 Internal Server Error again using Scrapy fetch
– Aminah Nuraini
Mar 16 '17 at 12:33
And I get 500 Internal Server Error again using Scrapy fetch
– Aminah Nuraini
Mar 16 '17 at 12:33
@AminahNuraini but that not what it says in your question; I see you have another question open that does indicate that you want to do it in spider, so maybe you should make it more clear and keep one question open that describes your issue fully.
– Granitosaurus
Mar 16 '17 at 12:35
@AminahNuraini but that not what it says in your question; I see you have another question open that does indicate that you want to do it in spider, so maybe you should make it more clear and keep one question open that describes your issue fully.
– Granitosaurus
Mar 16 '17 at 12:35
Doesn't matter anyway. Scrapy fetch and inside spider gives the same result anyway
– Aminah Nuraini
Mar 16 '17 at 12:36
Doesn't matter anyway. Scrapy fetch and inside spider gives the same result anyway
– Aminah Nuraini
Mar 16 '17 at 12:36
add a comment |
Found the answer!!!
Scrapy does not allow HTTP Basic Auth via URL or Authorization header because of a bug in Twisted Twisted HTTPS Client.
We have to use a not so famous middleware, HTTPAuthMiddleware https://github.com/scrapy/scrapy/blob/master/docs/topics/downloader-middleware.rst#httpauthmiddleware
To use it, activate the middleware in the settings.py like this:
DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware': 811,
}
And add these attributes to the spider.
from scrapy.spiders import CrawlSpider
class SomeIntranetSiteSpider(CrawlSpider):
http_user = 'someuser'
http_pass = 'somepass'
name = 'intranet.example.com'
# .. rest of the spider code omitted ...
edited May 23 '17 at 10:30
Community♦
11
answered Mar 16 '17 at 13:10
Aminah NurainiAminah Nuraini
6,85744863
add a comment |
Found the answer!!!
Scrapy does not allow HTTP Basic Auth via URL or Authorization header because of a bug in Twisted Twisted HTTPS Client.
We have to use a not so famous middleware, HTTPAuthMiddleware https://github.com/scrapy/scrapy/blob/master/docs/topics/downloader-middleware.rst#httpauthmiddleware
To use it, activate the middleware in the settings.py like this:
DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware': 811,
}
And add these attributes to the spider.
from scrapy.spiders import CrawlSpider
class SomeIntranetSiteSpider(CrawlSpider):
http_user = 'someuser'
http_pass = 'somepass'
name = 'intranet.example.com'
# .. rest of the spider code omitted ...
edited May 23 '17 at 10:30
Community♦
11
answered Mar 16 '17 at 13:10
Aminah NurainiAminah Nuraini
6,85744863
add a comment |
Found the answer!!!
Scrapy does not allow HTTP Basic Auth via URL or Authorization header because of a bug in Twisted Twisted HTTPS Client.
We have to use a not so famous middleware, HTTPAuthMiddleware https://github.com/scrapy/scrapy/blob/master/docs/topics/downloader-middleware.rst#httpauthmiddleware
To use it, activate the middleware in the settings.py like this:
DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware': 811,
}
And add these attributes to the spider.
from scrapy.spiders import CrawlSpider
class SomeIntranetSiteSpider(CrawlSpider):
http_user = 'someuser'
http_pass = 'somepass'
name = 'intranet.example.com'
# .. rest of the spider code omitted ...
edited May 23 '17 at 10:30
Community♦
11
answered Mar 16 '17 at 13:10
Aminah NurainiAminah Nuraini
6,85744863
Found the answer!!!
Scrapy does not allow HTTP Basic Auth via URL or Authorization header because of a bug in Twisted Twisted HTTPS Client.
We have to use a not so famous middleware, HTTPAuthMiddleware https://github.com/scrapy/scrapy/blob/master/docs/topics/downloader-middleware.rst#httpauthmiddleware
To use it, activate the middleware in the settings.py like this:
DOWNLOADER_MIDDLEWARES = {
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware': 811,
}
And add these attributes to the spider.
from scrapy.spiders import CrawlSpider
class SomeIntranetSiteSpider(CrawlSpider):
http_user = 'someuser'
http_pass = 'somepass'
name = 'intranet.example.com'
# .. rest of the spider code omitted ...
edited May 23 '17 at 10:30
Community♦
11
answered Mar 16 '17 at 13:10
Aminah NurainiAminah Nuraini
6,85744863
edited May 23 '17 at 10:30
Community♦
11
edited May 23 '17 at 10:30
Community♦
11
edited May 23 '17 at 10:30
Community♦
11
11
answered Mar 16 '17 at 13:10
Aminah NurainiAminah Nuraini
6,85744863
answered Mar 16 '17 at 13:10
Aminah NurainiAminah Nuraini
6,85744863
answered Mar 16 '17 at 13:10
Aminah NurainiAminah Nuraini
6,85744863
6,85744863
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f42832143%2fcurl-u-in-scrapy%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
