Mixing
PutObject signature issues on boto3 1.7.74 (with botocore 1.10.84) .. but not on a more recent boto3
I am getting the following error on one version of boto3, and not on another, without changing code:
An error occurred (SignatureDoesNotMatch) when calling the PutObject operation: The request signature we calculated does not match the signature you provided. Check your key and signing method.
I encounter this error on boto3 1.7.74 (with botocore 1.10.84), in the aws lambda environment. I can reproduce the error if I install the same boto3 version with pip in my dev environment.
The problematic code is quite simple:
# python3
s3 = boto3.client('s3')
base64_md5 = hex_to_b64(md5_hexdigest)
extra_args = {
'ContentType': 'application/octet-stream',
'ContentMD5': base64_md5, # "dsE2vA1oCr2XjZk0BwwB5Q=="
'ContentLength': content_length, # 62914560
'Metadata': {}
}
try:
completed = s3.put_object(Bucket=bucketname, Key=keyname, Body=inputfp, **extra_args)
Sometimes these signature issues are associated with passing bad credentials to the code. My credentials are stored under ~/.aws/, and I have AWS_PROFILE exported in my dev environment. Otherwise, I'm not passing any keys or tokens to my code.
In the code above, inputfp is a non-seekable() file object (I'm feeding boto3 the raw response body of a GET request to a local webserver). Content-Length and Content-MD5 are known by the time I feed it to boto3. My particular test file happens to be 60MB (62914560bytes) of zeroes (dd if=/dev/null bs=1M count=0 seek=60 of=zeroes). The md5 is 76c136bc0d680abd978d9934070c01e5, which corresponds to the b64-encoded value dsE2vA1oCr2XjZk0BwwB5Q==, passed via the ContentMD5 parameter.
I do not encounter this error on boto3 v1.9.35 (with botocore v1.12.35) via pip on my dev box. Similarly, I do not get the error if I include that version in my lambda code zip.
I suspect boto 1.7.74 / botocore 1.10.84 doesn't handle stream fds as well as a more recent version, but it's not obvious to verify -- the botocore library appears to be almost entirely procedurally generated from stubs. Anyone knows where in botocore the signature headers are produced by any chance? -- I would compare the two versions.
I will add a few more wrappers on my file object, and see where possibly it might go wrong.
amazon-web-services aws-lambda boto3 botocore
asked Nov 19 '18 at 18:05
init_jsinit_js
946724
add a comment |
I am getting the following error on one version of boto3, and not on another, without changing code:
An error occurred (SignatureDoesNotMatch) when calling the PutObject operation: The request signature we calculated does not match the signature you provided. Check your key and signing method.
I encounter this error on boto3 1.7.74 (with botocore 1.10.84), in the aws lambda environment. I can reproduce the error if I install the same boto3 version with pip in my dev environment.
The problematic code is quite simple:
# python3
s3 = boto3.client('s3')
base64_md5 = hex_to_b64(md5_hexdigest)
extra_args = {
'ContentType': 'application/octet-stream',
'ContentMD5': base64_md5, # "dsE2vA1oCr2XjZk0BwwB5Q=="
'ContentLength': content_length, # 62914560
'Metadata': {}
}
try:
completed = s3.put_object(Bucket=bucketname, Key=keyname, Body=inputfp, **extra_args)
Sometimes these signature issues are associated with passing bad credentials to the code. My credentials are stored under ~/.aws/, and I have AWS_PROFILE exported in my dev environment. Otherwise, I'm not passing any keys or tokens to my code.
In the code above, inputfp is a non-seekable() file object (I'm feeding boto3 the raw response body of a GET request to a local webserver). Content-Length and Content-MD5 are known by the time I feed it to boto3. My particular test file happens to be 60MB (62914560bytes) of zeroes (dd if=/dev/null bs=1M count=0 seek=60 of=zeroes). The md5 is 76c136bc0d680abd978d9934070c01e5, which corresponds to the b64-encoded value dsE2vA1oCr2XjZk0BwwB5Q==, passed via the ContentMD5 parameter.
I do not encounter this error on boto3 v1.9.35 (with botocore v1.12.35) via pip on my dev box. Similarly, I do not get the error if I include that version in my lambda code zip.
I suspect boto 1.7.74 / botocore 1.10.84 doesn't handle stream fds as well as a more recent version, but it's not obvious to verify -- the botocore library appears to be almost entirely procedurally generated from stubs. Anyone knows where in botocore the signature headers are produced by any chance? -- I would compare the two versions.
I will add a few more wrappers on my file object, and see where possibly it might go wrong.
amazon-web-services aws-lambda boto3 botocore
asked Nov 19 '18 at 18:05
init_jsinit_js
946724
add a comment |
I am getting the following error on one version of boto3, and not on another, without changing code:
An error occurred (SignatureDoesNotMatch) when calling the PutObject operation: The request signature we calculated does not match the signature you provided. Check your key and signing method.
I encounter this error on boto3 1.7.74 (with botocore 1.10.84), in the aws lambda environment. I can reproduce the error if I install the same boto3 version with pip in my dev environment.
The problematic code is quite simple:
# python3
s3 = boto3.client('s3')
base64_md5 = hex_to_b64(md5_hexdigest)
extra_args = {
'ContentType': 'application/octet-stream',
'ContentMD5': base64_md5, # "dsE2vA1oCr2XjZk0BwwB5Q=="
'ContentLength': content_length, # 62914560
'Metadata': {}
}
try:
completed = s3.put_object(Bucket=bucketname, Key=keyname, Body=inputfp, **extra_args)
Sometimes these signature issues are associated with passing bad credentials to the code. My credentials are stored under ~/.aws/, and I have AWS_PROFILE exported in my dev environment. Otherwise, I'm not passing any keys or tokens to my code.
In the code above, inputfp is a non-seekable() file object (I'm feeding boto3 the raw response body of a GET request to a local webserver). Content-Length and Content-MD5 are known by the time I feed it to boto3. My particular test file happens to be 60MB (62914560bytes) of zeroes (dd if=/dev/null bs=1M count=0 seek=60 of=zeroes). The md5 is 76c136bc0d680abd978d9934070c01e5, which corresponds to the b64-encoded value dsE2vA1oCr2XjZk0BwwB5Q==, passed via the ContentMD5 parameter.
I do not encounter this error on boto3 v1.9.35 (with botocore v1.12.35) via pip on my dev box. Similarly, I do not get the error if I include that version in my lambda code zip.
I suspect boto 1.7.74 / botocore 1.10.84 doesn't handle stream fds as well as a more recent version, but it's not obvious to verify -- the botocore library appears to be almost entirely procedurally generated from stubs. Anyone knows where in botocore the signature headers are produced by any chance? -- I would compare the two versions.
I will add a few more wrappers on my file object, and see where possibly it might go wrong.
amazon-web-services aws-lambda boto3 botocore
asked Nov 19 '18 at 18:05
init_jsinit_js
946724
I am getting the following error on one version of boto3, and not on another, without changing code:
An error occurred (SignatureDoesNotMatch) when calling the PutObject operation: The request signature we calculated does not match the signature you provided. Check your key and signing method.
I encounter this error on boto3 1.7.74 (with botocore 1.10.84), in the aws lambda environment. I can reproduce the error if I install the same boto3 version with pip in my dev environment.
The problematic code is quite simple:
# python3
s3 = boto3.client('s3')
base64_md5 = hex_to_b64(md5_hexdigest)
extra_args = {
'ContentType': 'application/octet-stream',
'ContentMD5': base64_md5, # "dsE2vA1oCr2XjZk0BwwB5Q=="
'ContentLength': content_length, # 62914560
'Metadata': {}
}
try:
completed = s3.put_object(Bucket=bucketname, Key=keyname, Body=inputfp, **extra_args)
Sometimes these signature issues are associated with passing bad credentials to the code. My credentials are stored under ~/.aws/, and I have AWS_PROFILE exported in my dev environment. Otherwise, I'm not passing any keys or tokens to my code.
In the code above, inputfp is a non-seekable() file object (I'm feeding boto3 the raw response body of a GET request to a local webserver). Content-Length and Content-MD5 are known by the time I feed it to boto3. My particular test file happens to be 60MB (62914560bytes) of zeroes (dd if=/dev/null bs=1M count=0 seek=60 of=zeroes). The md5 is 76c136bc0d680abd978d9934070c01e5, which corresponds to the b64-encoded value dsE2vA1oCr2XjZk0BwwB5Q==, passed via the ContentMD5 parameter.
I do not encounter this error on boto3 v1.9.35 (with botocore v1.12.35) via pip on my dev box. Similarly, I do not get the error if I include that version in my lambda code zip.
I suspect boto 1.7.74 / botocore 1.10.84 doesn't handle stream fds as well as a more recent version, but it's not obvious to verify -- the botocore library appears to be almost entirely procedurally generated from stubs. Anyone knows where in botocore the signature headers are produced by any chance? -- I would compare the two versions.
I will add a few more wrappers on my file object, and see where possibly it might go wrong.
amazon-web-services aws-lambda boto3 botocore
amazon-web-services aws-lambda boto3 botocore
asked Nov 19 '18 at 18:05
init_jsinit_js
946724
asked Nov 19 '18 at 18:05
init_jsinit_js
946724
asked Nov 19 '18 at 18:05
init_jsinit_js
946724
asked Nov 19 '18 at 18:05
init_jsinit_js
946724
asked Nov 19 '18 at 18:05
init_jsinit_js
946724
946724
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53380330%2fputobject-signature-issues-on-boto3-1-7-74-with-botocore-1-10-84-but-not-on%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53380330%2fputobject-signature-issues-on-boto3-1-7-74-with-botocore-1-10-84-but-not-on%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
