Connect to elasticsearch in AWS using key credentials
up vote
0
down vote
favorite
I'm trying to post a request using curl to my es cluster in AWS using my accessKey and secretKey. I have successfully done this through postman (details here) where you can specify AWS credentials but I would like to make this work with curl. Postman can auto-generate your curl request for you but all I get are errors.
This is the generated curl request along with the response
curl -X GET
https://search-00000000000001.eu-west-1.es.amazonaws.com/_cat/indices
-H 'Authorization: AWS4-HMAC-SHA256 Credential=11111111111111111111/20181119/eu-west-1/es/aws4_request, SignedHeaders=cache-control;content-type;host;postman-token;x-amz-date, Signature=11111111116401882398f46011f14fdb9d55e012a4fb912706d67c1111111111'
-H 'Content-Type: application/x-www-form-urlencoded'
-H 'Host: search-00000000000001.eu-west-1.es.amazonaws.com'
-H 'Postman-Token: 00000000-0000-4001-8006-9291e208a000'
-H 'X-Amz-Date: 20181119T220000Z'
-H 'cache-control: no-cache'
{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."}%
IDs have been changed to protect the innocent.
I have checked all my keys and region, and like i said this works through postman. Is it possible to access this AWS service using my keys through curl?
amazon-web-services
elasticsearch curl aws-elasticsearch asked Nov 19 at 23:01
Damo
647523
add a comment |
up vote
0
down vote
favorite
I'm trying to post a request using curl to my es cluster in AWS using my accessKey and secretKey. I have successfully done this through postman (details here) where you can specify AWS credentials but I would like to make this work with curl. Postman can auto-generate your curl request for you but all I get are errors.
This is the generated curl request along with the response
curl -X GET
https://search-00000000000001.eu-west-1.es.amazonaws.com/_cat/indices
-H 'Authorization: AWS4-HMAC-SHA256 Credential=11111111111111111111/20181119/eu-west-1/es/aws4_request, SignedHeaders=cache-control;content-type;host;postman-token;x-amz-date, Signature=11111111116401882398f46011f14fdb9d55e012a4fb912706d67c1111111111'
-H 'Content-Type: application/x-www-form-urlencoded'
-H 'Host: search-00000000000001.eu-west-1.es.amazonaws.com'
-H 'Postman-Token: 00000000-0000-4001-8006-9291e208a000'
-H 'X-Amz-Date: 20181119T220000Z'
-H 'cache-control: no-cache'
{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."}%
IDs have been changed to protect the innocent.
I have checked all my keys and region, and like i said this works through postman. Is it possible to access this AWS service using my keys through curl?
amazon-web-services
elasticsearch curl aws-elasticsearch asked Nov 19 at 23:01
Damo
647523
It's telling you the issue directly in the error message: it's not that you aren't connecting, it's that your Signature does not match what the service is computing for this message. If this worked from Postman, than it could be that a) you've changed some value being sent; b) Postman is sending some hidden value or header that it didn't export (unlikely);
– sofend
Nov 19 at 23:34
Could it be that there is a one time token or something else generated that can only be used once? i know for a fact i didn't change anything.
– Damo
Nov 20 at 8:59
1
I'm afraid I have bad news for you - signing an AWS request to Elastic Search is an extremely demanding, laborious process. Once you get the code right you're fine but it depends on calculating a hash for a very convoluted string - get one character wrong and ... well ... you've seen what happens. Search for "AWS4 signing." This ref gives you a start - docs.aws.amazon.com/apigateway/api-reference/signing-requests (If I had the time I'd post more details.) Good luck! Adam.
– Adam Benson
Nov 20 at 12:45
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I'm trying to post a request using curl to my es cluster in AWS using my accessKey and secretKey. I have successfully done this through postman (details here) where you can specify AWS credentials but I would like to make this work with curl. Postman can auto-generate your curl request for you but all I get are errors.
This is the generated curl request along with the response
curl -X GET
https://search-00000000000001.eu-west-1.es.amazonaws.com/_cat/indices
-H 'Authorization: AWS4-HMAC-SHA256 Credential=11111111111111111111/20181119/eu-west-1/es/aws4_request, SignedHeaders=cache-control;content-type;host;postman-token;x-amz-date, Signature=11111111116401882398f46011f14fdb9d55e012a4fb912706d67c1111111111'
-H 'Content-Type: application/x-www-form-urlencoded'
-H 'Host: search-00000000000001.eu-west-1.es.amazonaws.com'
-H 'Postman-Token: 00000000-0000-4001-8006-9291e208a000'
-H 'X-Amz-Date: 20181119T220000Z'
-H 'cache-control: no-cache'
{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."}%
IDs have been changed to protect the innocent.
I have checked all my keys and region, and like i said this works through postman. Is it possible to access this AWS service using my keys through curl?
amazon-web-services
elasticsearch curl aws-elasticsearch asked Nov 19 at 23:01
Damo
647523
I'm trying to post a request using curl to my es cluster in AWS using my accessKey and secretKey. I have successfully done this through postman (details here) where you can specify AWS credentials but I would like to make this work with curl. Postman can auto-generate your curl request for you but all I get are errors.
This is the generated curl request along with the response
curl -X GET
https://search-00000000000001.eu-west-1.es.amazonaws.com/_cat/indices
-H 'Authorization: AWS4-HMAC-SHA256 Credential=11111111111111111111/20181119/eu-west-1/es/aws4_request, SignedHeaders=cache-control;content-type;host;postman-token;x-amz-date, Signature=11111111116401882398f46011f14fdb9d55e012a4fb912706d67c1111111111'
-H 'Content-Type: application/x-www-form-urlencoded'
-H 'Host: search-00000000000001.eu-west-1.es.amazonaws.com'
-H 'Postman-Token: 00000000-0000-4001-8006-9291e208a000'
-H 'X-Amz-Date: 20181119T220000Z'
-H 'cache-control: no-cache'
{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."}%
IDs have been changed to protect the innocent.
I have checked all my keys and region, and like i said this works through postman. Is it possible to access this AWS service using my keys through curl?
amazon-web-services
elasticsearch curl aws-elasticsearch amazon-web-services
elasticsearch curl aws-elasticsearch asked Nov 19 at 23:01
Damo
647523
asked Nov 19 at 23:01
Damo
647523
edited Nov 21 at 9:26
asked Nov 19 at 23:01
Damo
647523
asked Nov 19 at 23:01
Damo
647523
asked Nov 19 at 23:01
Damo
647523
647523
It's telling you the issue directly in the error message: it's not that you aren't connecting, it's that your Signature does not match what the service is computing for this message. If this worked from Postman, than it could be that a) you've changed some value being sent; b) Postman is sending some hidden value or header that it didn't export (unlikely);
– sofend
Nov 19 at 23:34
Could it be that there is a one time token or something else generated that can only be used once? i know for a fact i didn't change anything.
– Damo
Nov 20 at 8:59
1
I'm afraid I have bad news for you - signing an AWS request to Elastic Search is an extremely demanding, laborious process. Once you get the code right you're fine but it depends on calculating a hash for a very convoluted string - get one character wrong and ... well ... you've seen what happens. Search for "AWS4 signing." This ref gives you a start - docs.aws.amazon.com/apigateway/api-reference/signing-requests (If I had the time I'd post more details.) Good luck! Adam.
– Adam Benson
Nov 20 at 12:45
add a comment |
It's telling you the issue directly in the error message: it's not that you aren't connecting, it's that your Signature does not match what the service is computing for this message. If this worked from Postman, than it could be that a) you've changed some value being sent; b) Postman is sending some hidden value or header that it didn't export (unlikely);
– sofend
Nov 19 at 23:34
Could it be that there is a one time token or something else generated that can only be used once? i know for a fact i didn't change anything.
– Damo
Nov 20 at 8:59
1
I'm afraid I have bad news for you - signing an AWS request to Elastic Search is an extremely demanding, laborious process. Once you get the code right you're fine but it depends on calculating a hash for a very convoluted string - get one character wrong and ... well ... you've seen what happens. Search for "AWS4 signing." This ref gives you a start - docs.aws.amazon.com/apigateway/api-reference/signing-requests (If I had the time I'd post more details.) Good luck! Adam.
– Adam Benson
Nov 20 at 12:45
It's telling you the issue directly in the error message: it's not that you aren't connecting, it's that your Signature does not match what the service is computing for this message. If this worked from Postman, than it could be that a) you've changed some value being sent; b) Postman is sending some hidden value or header that it didn't export (unlikely);
– sofend
Nov 19 at 23:34
It's telling you the issue directly in the error message: it's not that you aren't connecting, it's that your Signature does not match what the service is computing for this message. If this worked from Postman, than it could be that a) you've changed some value being sent; b) Postman is sending some hidden value or header that it didn't export (unlikely);
– sofend
Nov 19 at 23:34
Could it be that there is a one time token or something else generated that can only be used once? i know for a fact i didn't change anything.
– Damo
Nov 20 at 8:59
Could it be that there is a one time token or something else generated that can only be used once? i know for a fact i didn't change anything.
– Damo
Nov 20 at 8:59
1
1
I'm afraid I have bad news for you - signing an AWS request to Elastic Search is an extremely demanding, laborious process. Once you get the code right you're fine but it depends on calculating a hash for a very convoluted string - get one character wrong and ... well ... you've seen what happens. Search for "AWS4 signing." This ref gives you a start - docs.aws.amazon.com/apigateway/api-reference/signing-requests (If I had the time I'd post more details.) Good luck! Adam.
– Adam Benson
Nov 20 at 12:45
I'm afraid I have bad news for you - signing an AWS request to Elastic Search is an extremely demanding, laborious process. Once you get the code right you're fine but it depends on calculating a hash for a very convoluted string - get one character wrong and ... well ... you've seen what happens. Search for "AWS4 signing." This ref gives you a start - docs.aws.amazon.com/apigateway/api-reference/signing-requests (If I had the time I'd post more details.) Good luck! Adam.
– Adam Benson
Nov 20 at 12:45
add a comment |
1 Answer
1
active
oldest
votes
up vote
0
down vote
accepted
This is quite a long rabbit hole. Thanks to Adam for the comment that sent me in the correct direction. The link https://docs.aws.amazon.com/apigateway/api-reference/signing-requests/ really helps you understand what you need to do.
I've since found a script that follows the signing requests method outlined above. It runs in bash and whilst it is not written for use with elasticsearch requests it can be used for them.
https://github.com/riboseinc/aws-authenticating-secgroup-scripts many thanks to https://www.ribose.com for putting this on github.
answered Nov 21 at 9:24
Damo
647523
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
accepted
This is quite a long rabbit hole. Thanks to Adam for the comment that sent me in the correct direction. The link https://docs.aws.amazon.com/apigateway/api-reference/signing-requests/ really helps you understand what you need to do.
I've since found a script that follows the signing requests method outlined above. It runs in bash and whilst it is not written for use with elasticsearch requests it can be used for them.
https://github.com/riboseinc/aws-authenticating-secgroup-scripts many thanks to https://www.ribose.com for putting this on github.
answered Nov 21 at 9:24
Damo
647523
add a comment |
up vote
0
down vote
accepted
This is quite a long rabbit hole. Thanks to Adam for the comment that sent me in the correct direction. The link https://docs.aws.amazon.com/apigateway/api-reference/signing-requests/ really helps you understand what you need to do.
I've since found a script that follows the signing requests method outlined above. It runs in bash and whilst it is not written for use with elasticsearch requests it can be used for them.
https://github.com/riboseinc/aws-authenticating-secgroup-scripts many thanks to https://www.ribose.com for putting this on github.
answered Nov 21 at 9:24
Damo
647523
add a comment |
up vote
0
down vote
accepted
up vote
0
down vote
accepted
This is quite a long rabbit hole. Thanks to Adam for the comment that sent me in the correct direction. The link https://docs.aws.amazon.com/apigateway/api-reference/signing-requests/ really helps you understand what you need to do.
I've since found a script that follows the signing requests method outlined above. It runs in bash and whilst it is not written for use with elasticsearch requests it can be used for them.
https://github.com/riboseinc/aws-authenticating-secgroup-scripts many thanks to https://www.ribose.com for putting this on github.
answered Nov 21 at 9:24
Damo
647523
This is quite a long rabbit hole. Thanks to Adam for the comment that sent me in the correct direction. The link https://docs.aws.amazon.com/apigateway/api-reference/signing-requests/ really helps you understand what you need to do.
I've since found a script that follows the signing requests method outlined above. It runs in bash and whilst it is not written for use with elasticsearch requests it can be used for them.
https://github.com/riboseinc/aws-authenticating-secgroup-scripts many thanks to https://www.ribose.com for putting this on github.
answered Nov 21 at 9:24
Damo
647523
answered Nov 21 at 9:24
Damo
647523
answered Nov 21 at 9:24
Damo
647523
answered Nov 21 at 9:24
Damo
647523
647523
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53383877%2fconnect-to-elasticsearch-in-aws-using-key-credentials%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
It's telling you the issue directly in the error message: it's not that you aren't connecting, it's that your Signature does not match what the service is computing for this message. If this worked from Postman, than it could be that a) you've changed some value being sent; b) Postman is sending some hidden value or header that it didn't export (unlikely);
– sofend
Nov 19 at 23:34
Could it be that there is a one time token or something else generated that can only be used once? i know for a fact i didn't change anything.
– Damo
Nov 20 at 8:59
1
I'm afraid I have bad news for you - signing an AWS request to Elastic Search is an extremely demanding, laborious process. Once you get the code right you're fine but it depends on calculating a hash for a very convoluted string - get one character wrong and ... well ... you've seen what happens. Search for "AWS4 signing." This ref gives you a start - docs.aws.amazon.com/apigateway/api-reference/signing-requests (If I had the time I'd post more details.) Good luck! Adam.
– Adam Benson
Nov 20 at 12:45