Mixing
Listing contents of a bucket with boto3
93
How can I see what's inside a bucket in S3 with boto3? (i.e. do an "ls")?
Doing the following:
import boto3
s3 = boto3.resource('s3')
my_bucket = s3.Bucket('some/path/')
returns:
s3.Bucket(name='some/path/')
How do I see its contents?
python amazon-s3 boto boto3
asked May 14 '15 at 23:22
Amelio Vazquez-ReinaAmelio Vazquez-Reina
27.1k73250439
add a comment |
93
How can I see what's inside a bucket in S3 with boto3? (i.e. do an "ls")?
Doing the following:
import boto3
s3 = boto3.resource('s3')
my_bucket = s3.Bucket('some/path/')
returns:
s3.Bucket(name='some/path/')
How do I see its contents?
python amazon-s3 boto boto3
asked May 14 '15 at 23:22
Amelio Vazquez-ReinaAmelio Vazquez-Reina
27.1k73250439
add a comment |
93
93
93
23
How can I see what's inside a bucket in S3 with boto3? (i.e. do an "ls")?
Doing the following:
import boto3
s3 = boto3.resource('s3')
my_bucket = s3.Bucket('some/path/')
returns:
s3.Bucket(name='some/path/')
How do I see its contents?
python amazon-s3 boto boto3
asked May 14 '15 at 23:22
Amelio Vazquez-ReinaAmelio Vazquez-Reina
27.1k73250439
How can I see what's inside a bucket in S3 with boto3? (i.e. do an "ls")?
Doing the following:
import boto3
s3 = boto3.resource('s3')
my_bucket = s3.Bucket('some/path/')
returns:
s3.Bucket(name='some/path/')
How do I see its contents?
python amazon-s3 boto boto3
python amazon-s3 boto boto3
asked May 14 '15 at 23:22
Amelio Vazquez-ReinaAmelio Vazquez-Reina
27.1k73250439
asked May 14 '15 at 23:22
Amelio Vazquez-ReinaAmelio Vazquez-Reina
27.1k73250439
asked May 14 '15 at 23:22
Amelio Vazquez-ReinaAmelio Vazquez-Reina
27.1k73250439
asked May 14 '15 at 23:22
Amelio Vazquez-ReinaAmelio Vazquez-Reina
27.1k73250439
asked May 14 '15 at 23:22
Amelio Vazquez-ReinaAmelio Vazquez-Reina
27.1k73250439
27.1k73250439
add a comment |
add a comment |
9 Answers
9
active
oldest
votes
130
One way to see the contents would be:
for object in my_bucket.objects.all():
print(object)
edited Aug 3 '18 at 21:18
zvikico
9,13043344
answered May 15 '15 at 0:17
garnaatgarnaat
27.9k78482
1
can i fetch the keys under particular path in bucket or with particular delimiter using boto3??
– Rahul KP
Dec 14 '15 at 11:44
57
You should be able to saymybucket.objects.filter(Prefix='foo/bar')and it will only list objects with that prefix. You can also pass aDelimiterparameter.
– garnaat
Dec 14 '15 at 12:53
3
not working with boto3 AttributeError: 'S3' object has no attribute 'objects'
– Shek
Jun 30 '17 at 17:45
2
@garnaat Your comment mentioning that filter method really helped me (my code ended up much simpler and faster) - thank you!
– Edward Dixon
Aug 2 '17 at 16:25
11
I would advise against usingobjectas a variable name as it will shadow the global typeobject.
– oliland
May 8 '18 at 10:28
add a comment |
59
This is similar to an 'ls' but it does not take into account the prefix folder convention and will list the objects in the bucket. It's left up to the reader to filter out prefixes which are part of the Key name.
In Python 2:
from boto.s3.connection import S3Connection
conn = S3Connection() # assumes boto.cfg setup
bucket = conn.get_bucket('bucket_name')
for obj in bucket.get_all_keys():
print(obj.key)
In Python 3:
from boto3 import client
conn = client('s3') # again assumes boto.cfg setup, assume AWS S3
for key in conn.list_objects(Bucket='bucket_name')['Contents']:
print(key['Key'])
edited Apr 5 '17 at 10:35
Amelio Vazquez-Reina
27.1k73250439
answered May 15 '15 at 14:45
cgsellercgseller
1,9031218
31
If you want to use the prefix as well, you can do it like this:conn.list_objects(Bucket='bucket_name', Prefix='prefix_string')['Contents']
– markonovak
Mar 21 '16 at 13:14
8
This only lists the first 1000 keys. From the docstring: "Returns some or all (up to 1000) of the objects in a bucket." Also, it is recommended that you use list_objects_v2 instead of list_objects (although, this also only returns the first 1000 keys).
– Brett Widmeier
Mar 21 '18 at 14:18
add a comment |
20
If you want to pass the ACCESS and SECRET keys:
from boto3.session import Session
ACCESS_KEY='your_access_key'
SECRET_KEY='your_secret_key'
session = Session(aws_access_key_id=ACCESS_KEY,
aws_secret_access_key=SECRET_KEY)
s3 = session.resource('s3')
your_bucket = s3.Bucket('your_bucket')
for s3_file in your_bucket.objects.all():
print(s3_file.key)
answered Apr 7 '17 at 13:16
Erwin AlbertoErwin Alberto
40447
8
This is less secure than having a credentials file at ~/.aws/credentials. Though it is a valid solution.
– nu everest
Dec 27 '17 at 0:32
2
This would require committing secrets to source control. Not good.
– jan groth
Dec 7 '18 at 1:41
add a comment |
16
I'm assuming you have configured authentication separately.
import boto3
s3 = boto3.resource('s3')
my_bucket = s3.Bucket('bucket_name')
for file in my_bucket.objects.all():
print file.key
edited Oct 17 '18 at 5:44
BMW
22.8k45371
answered Apr 5 '17 at 4:04
Tushar NirasTushar Niras
856914
add a comment |
8
In order to handle large key listings (i.e. when the directory list is greater than 1000 items), I used the following code to accumulate key values (i.e. filenames) with multiple listings (thanks to Amelio above for the first lines). Code is for python3:
from boto3 import client
bucket_name = "my_bucket"
prefix = "my_key/sub_key/lots_o_files"
s3_conn = client('s3') # type: BaseClient ## again assumes boto.cfg setup, assume AWS S3
s3_result = s3_conn.list_objects_v2(Bucket=bucket_name, Prefix=prefix, Delimiter = "/")
if 'Contents' not in s3_result:
#print(s3_result)
return
file_list =
for key in s3_result['Contents']:
file_list.append(key['Key'])
print(f"List count = {len(file_list)}")
while s3_result['IsTruncated']:
continuation_key = s3_result['NextContinuationToken']
s3_result = s3_conn.list_objects_v2(Bucket=bucket_name, Prefix=prefix, Delimiter="/", ContinuationToken=continuation_key)
for key in s3_result['Contents']:
file_list.append(key['Key'])
print(f"List count = {len(file_list)}")
return file_list
answered Nov 22 '18 at 1:17
HephaestusHephaestus
1,15621422
add a comment |
5
My s3 keys utility function is essentially an optimized version of @Hephaestus's answer:
import boto3
s3_paginator = boto3.client('s3').get_paginator('list_objects_v2')
def keys(bucket_name, prefix='/', delimiter='/', start_after=''):
prefix = prefix[1:] if prefix.startswith(delimiter) else prefix
start_after = (start_after or prefix) if prefix.endswith(delimiter) else start_after
for page in s3_paginator.paginate(Bucket=bucket_name, Prefix=prefix, StartAfter=start_after):
for content in page.get('Contents', ()):
yield content['Key']
In my tests (boto3 1.9.84), it's significantly faster than the equivalent (but simpler) code:
import boto3
def keys(bucket_name, prefix='/', delimiter='/'):
prefix = prefix[1:] if prefix.startswith(delimiter) else prefix
bucket = boto3.resource('s3').Bucket(bucket_name)
return (_.key for _ in bucket.objects.filter(Prefix=prefix))
As S3 guarantees UTF-8 binary sorted results, a start_after optimization has been added to the first function.
answered Jan 3 at 0:19
Sean SummersSean Summers
1,1501018
add a comment |
4
A more parsimonious way, rather than iterating through via a for loop you could also just print the original object containing all files inside your S3 bucket:
session = Session(aws_access_key_id=aws_access_key_id,aws_secret_access_key=aws_secret_access_key)
s3 = session.resource('s3')
bucket = s3.Bucket('bucket_name')
files_in_s3 = bucket.objects.all()
#you can print this iterable with print(list(files_in_s3))
edited Aug 23 '18 at 14:21
petezurich
3,65081834
answered Jun 24 '17 at 7:14
Daniel VieiraDaniel Vieira
1236
@petezurich , can you please explain why such a petty edit of my answer - replacing an “a” with a capital “A” at the beginning of my answer brought down my reputation by -2 , however I reckon both you and I can agree that not only is your correction NOT Relevant at all, but actually rather petty, wouldn’t you say so? Please focus on the content rather than childish revisions , most obliged ol’boy
– Daniel Vieira
Aug 23 '18 at 13:41
These were two different interactions. 1. I edited your answer which is recommended even for minor misspellings. I agree, that the boundaries between minor and trivial are ambiguous. I do not downvote any post because I see errors and I didn't in this case. I simply fix all the errors that I see.
– petezurich
Aug 23 '18 at 14:26
2. I downvoted your answer because you wrote thatfiles_in_s3is a "list object". There is no such thing in Python. It rather is an iterable and I couldn't make your code work and therefore downvoted. Than I found the error and saw your point but couldn't undo my downvote.
– petezurich
Aug 23 '18 at 14:29
I now have edited your answer again and could undo the downvote. Feel free to reedit.
– petezurich
Aug 23 '18 at 14:30
1
@petezurich no problem , understood your , point , just one thing, in Python a list IS an object because pretty much everything in python is an object , then it also follows that a list is also an iterable, but first and foremost , it’s an object! that is why I did not understand your downvote- you were down voting something that was correct and code that works. Anyway , thanks for your apology and all the best
– Daniel Vieira
Aug 23 '18 at 19:02
|
show 1 more comment
1
ObjectSummary:
There are two identifiers that are attached to the ObjectSummary:
- bucket_name
- key
boto3 S3: ObjectSummary
More on Object Keys from AWS S3 Documentation:
Object Keys:
When you create an object, you specify the key name, which uniquely identifies the object in the bucket. For example, in the Amazon S3 console (see AWS Management Console), when you highlight a bucket, a list of objects in your bucket appears. These names are the object keys. The name for a key is a sequence of Unicode characters whose UTF-8 encoding is at most 1024 bytes long.
The Amazon S3 data model is a flat structure: you create a bucket, and the bucket stores objects. There is no hierarchy of subbuckets or subfolders; however, you can infer logical hierarchy using key name prefixes and delimiters as the Amazon S3 console does. The Amazon S3 console supports a concept of folders. Suppose that your bucket (admin-created) has four objects with the following object keys:
Development/Projects1.xls
Finance/statement1.pdf
Private/taxdocument.pdf
s3-dg.pdf
Reference:
AWS S3: Object Keys
Here is some example code that demonstrates how to get the bucket name and the object key.
Example:
import boto3
from pprint import pprint
def main():
def enumerate_s3():
s3 = boto3.resource('s3')
for bucket in s3.buckets.all():
print("Name: {}".format(bucket.name))
print("Creation Date: {}".format(bucket.creation_date))
for object in bucket.objects.all():
print("Object: {}".format(object))
print("Object bucket_name: {}".format(object.bucket_name))
print("Object key: {}".format(object.key))
enumerate_s3()
if __name__ == '__main__':
main()
answered Oct 16 '18 at 19:26
GothburzGothburz
2,25611119
add a comment |
0
I just did it like this, including the authentication method:
s3_client = boto3.client(
's3',
aws_access_key_id='access_key',
aws_secret_access_key='access_key_secret',
config=boto3.session.Config(signature_version='s3v4'),
region_name='region'
)
response = s3_client.list_objects(Bucket='bucket_name', Prefix=key)
if ('Contents' in response):
# Object / key exists!
return True
else:
# Object / key DOES NOT exist!
return False
answered Sep 30 '18 at 19:21
MileanMilean
678615
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f30249069%2flisting-contents-of-a-bucket-with-boto3%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
9 Answers
9
active
oldest
votes
9 Answers
9
active
oldest
votes
active
oldest
votes
active
oldest
votes
130
One way to see the contents would be:
for object in my_bucket.objects.all():
print(object)
edited Aug 3 '18 at 21:18
zvikico
9,13043344
answered May 15 '15 at 0:17
garnaatgarnaat
27.9k78482
1
can i fetch the keys under particular path in bucket or with particular delimiter using boto3??
– Rahul KP
Dec 14 '15 at 11:44
57
You should be able to saymybucket.objects.filter(Prefix='foo/bar')and it will only list objects with that prefix. You can also pass aDelimiterparameter.
– garnaat
Dec 14 '15 at 12:53
3
not working with boto3 AttributeError: 'S3' object has no attribute 'objects'
– Shek
Jun 30 '17 at 17:45
2
@garnaat Your comment mentioning that filter method really helped me (my code ended up much simpler and faster) - thank you!
– Edward Dixon
Aug 2 '17 at 16:25
11
I would advise against usingobjectas a variable name as it will shadow the global typeobject.
– oliland
May 8 '18 at 10:28
add a comment |
130
One way to see the contents would be:
for object in my_bucket.objects.all():
print(object)
edited Aug 3 '18 at 21:18
zvikico
9,13043344
answered May 15 '15 at 0:17
garnaatgarnaat
27.9k78482
1
can i fetch the keys under particular path in bucket or with particular delimiter using boto3??
– Rahul KP
Dec 14 '15 at 11:44
57
You should be able to saymybucket.objects.filter(Prefix='foo/bar')and it will only list objects with that prefix. You can also pass aDelimiterparameter.
– garnaat
Dec 14 '15 at 12:53
3
not working with boto3 AttributeError: 'S3' object has no attribute 'objects'
– Shek
Jun 30 '17 at 17:45
2
@garnaat Your comment mentioning that filter method really helped me (my code ended up much simpler and faster) - thank you!
– Edward Dixon
Aug 2 '17 at 16:25
11
I would advise against usingobjectas a variable name as it will shadow the global typeobject.
– oliland
May 8 '18 at 10:28
add a comment |
130
130
130
One way to see the contents would be:
for object in my_bucket.objects.all():
print(object)
edited Aug 3 '18 at 21:18
zvikico
9,13043344
answered May 15 '15 at 0:17
garnaatgarnaat
27.9k78482
One way to see the contents would be:
for object in my_bucket.objects.all():
print(object)
edited Aug 3 '18 at 21:18
zvikico
9,13043344
answered May 15 '15 at 0:17
garnaatgarnaat
27.9k78482
edited Aug 3 '18 at 21:18
zvikico
9,13043344
edited Aug 3 '18 at 21:18
zvikico
9,13043344
edited Aug 3 '18 at 21:18
zvikico
9,13043344
9,13043344
answered May 15 '15 at 0:17
garnaatgarnaat
27.9k78482
answered May 15 '15 at 0:17
garnaatgarnaat
27.9k78482
answered May 15 '15 at 0:17
garnaatgarnaat
27.9k78482
27.9k78482
1
can i fetch the keys under particular path in bucket or with particular delimiter using boto3??
– Rahul KP
Dec 14 '15 at 11:44
57
You should be able to saymybucket.objects.filter(Prefix='foo/bar')and it will only list objects with that prefix. You can also pass aDelimiterparameter.
– garnaat
Dec 14 '15 at 12:53
3
not working with boto3 AttributeError: 'S3' object has no attribute 'objects'
– Shek
Jun 30 '17 at 17:45
2
@garnaat Your comment mentioning that filter method really helped me (my code ended up much simpler and faster) - thank you!
– Edward Dixon
Aug 2 '17 at 16:25
11
I would advise against usingobjectas a variable name as it will shadow the global typeobject.
– oliland
May 8 '18 at 10:28
add a comment |
1
can i fetch the keys under particular path in bucket or with particular delimiter using boto3??
– Rahul KP
Dec 14 '15 at 11:44
57
You should be able to saymybucket.objects.filter(Prefix='foo/bar')and it will only list objects with that prefix. You can also pass aDelimiterparameter.
– garnaat
Dec 14 '15 at 12:53
3
not working with boto3 AttributeError: 'S3' object has no attribute 'objects'
– Shek
Jun 30 '17 at 17:45
2
@garnaat Your comment mentioning that filter method really helped me (my code ended up much simpler and faster) - thank you!
– Edward Dixon
Aug 2 '17 at 16:25
11
I would advise against usingobjectas a variable name as it will shadow the global typeobject.
– oliland
May 8 '18 at 10:28
1
1
can i fetch the keys under particular path in bucket or with particular delimiter using boto3??
– Rahul KP
Dec 14 '15 at 11:44
can i fetch the keys under particular path in bucket or with particular delimiter using boto3??
– Rahul KP
Dec 14 '15 at 11:44
57
57
You should be able to say
mybucket.objects.filter(Prefix='foo/bar') and it will only list objects with that prefix. You can also pass a Delimiter parameter.– garnaat
Dec 14 '15 at 12:53
You should be able to say
mybucket.objects.filter(Prefix='foo/bar') and it will only list objects with that prefix. You can also pass a Delimiter parameter.– garnaat
Dec 14 '15 at 12:53
3
3
not working with boto3 AttributeError: 'S3' object has no attribute 'objects'
– Shek
Jun 30 '17 at 17:45
not working with boto3 AttributeError: 'S3' object has no attribute 'objects'
– Shek
Jun 30 '17 at 17:45
2
2
@garnaat Your comment mentioning that filter method really helped me (my code ended up much simpler and faster) - thank you!
– Edward Dixon
Aug 2 '17 at 16:25
@garnaat Your comment mentioning that filter method really helped me (my code ended up much simpler and faster) - thank you!
– Edward Dixon
Aug 2 '17 at 16:25
11
11
I would advise against using
object as a variable name as it will shadow the global type object.– oliland
May 8 '18 at 10:28
I would advise against using
object as a variable name as it will shadow the global type object.– oliland
May 8 '18 at 10:28
add a comment |
59
This is similar to an 'ls' but it does not take into account the prefix folder convention and will list the objects in the bucket. It's left up to the reader to filter out prefixes which are part of the Key name.
In Python 2:
from boto.s3.connection import S3Connection
conn = S3Connection() # assumes boto.cfg setup
bucket = conn.get_bucket('bucket_name')
for obj in bucket.get_all_keys():
print(obj.key)
In Python 3:
from boto3 import client
conn = client('s3') # again assumes boto.cfg setup, assume AWS S3
for key in conn.list_objects(Bucket='bucket_name')['Contents']:
print(key['Key'])
edited Apr 5 '17 at 10:35
Amelio Vazquez-Reina
27.1k73250439
answered May 15 '15 at 14:45
cgsellercgseller
1,9031218
31
If you want to use the prefix as well, you can do it like this:conn.list_objects(Bucket='bucket_name', Prefix='prefix_string')['Contents']
– markonovak
Mar 21 '16 at 13:14
8
This only lists the first 1000 keys. From the docstring: "Returns some or all (up to 1000) of the objects in a bucket." Also, it is recommended that you use list_objects_v2 instead of list_objects (although, this also only returns the first 1000 keys).
– Brett Widmeier
Mar 21 '18 at 14:18
add a comment |
59
This is similar to an 'ls' but it does not take into account the prefix folder convention and will list the objects in the bucket. It's left up to the reader to filter out prefixes which are part of the Key name.
In Python 2:
from boto.s3.connection import S3Connection
conn = S3Connection() # assumes boto.cfg setup
bucket = conn.get_bucket('bucket_name')
for obj in bucket.get_all_keys():
print(obj.key)
In Python 3:
from boto3 import client
conn = client('s3') # again assumes boto.cfg setup, assume AWS S3
for key in conn.list_objects(Bucket='bucket_name')['Contents']:
print(key['Key'])
edited Apr 5 '17 at 10:35
Amelio Vazquez-Reina
27.1k73250439
answered May 15 '15 at 14:45
cgsellercgseller
1,9031218
31
If you want to use the prefix as well, you can do it like this:conn.list_objects(Bucket='bucket_name', Prefix='prefix_string')['Contents']
– markonovak
Mar 21 '16 at 13:14
8
This only lists the first 1000 keys. From the docstring: "Returns some or all (up to 1000) of the objects in a bucket." Also, it is recommended that you use list_objects_v2 instead of list_objects (although, this also only returns the first 1000 keys).
– Brett Widmeier
Mar 21 '18 at 14:18
add a comment |
59
59
59
This is similar to an 'ls' but it does not take into account the prefix folder convention and will list the objects in the bucket. It's left up to the reader to filter out prefixes which are part of the Key name.
In Python 2:
from boto.s3.connection import S3Connection
conn = S3Connection() # assumes boto.cfg setup
bucket = conn.get_bucket('bucket_name')
for obj in bucket.get_all_keys():
print(obj.key)
In Python 3:
from boto3 import client
conn = client('s3') # again assumes boto.cfg setup, assume AWS S3
for key in conn.list_objects(Bucket='bucket_name')['Contents']:
print(key['Key'])
edited Apr 5 '17 at 10:35
Amelio Vazquez-Reina
27.1k73250439
answered May 15 '15 at 14:45
cgsellercgseller
1,9031218
This is similar to an 'ls' but it does not take into account the prefix folder convention and will list the objects in the bucket. It's left up to the reader to filter out prefixes which are part of the Key name.
In Python 2:
from boto.s3.connection import S3Connection
conn = S3Connection() # assumes boto.cfg setup
bucket = conn.get_bucket('bucket_name')
for obj in bucket.get_all_keys():
print(obj.key)
In Python 3:
from boto3 import client
conn = client('s3') # again assumes boto.cfg setup, assume AWS S3
for key in conn.list_objects(Bucket='bucket_name')['Contents']:
print(key['Key'])
edited Apr 5 '17 at 10:35
Amelio Vazquez-Reina
27.1k73250439
answered May 15 '15 at 14:45
cgsellercgseller
1,9031218
edited Apr 5 '17 at 10:35
Amelio Vazquez-Reina
27.1k73250439
edited Apr 5 '17 at 10:35
Amelio Vazquez-Reina
27.1k73250439
edited Apr 5 '17 at 10:35
Amelio Vazquez-Reina
27.1k73250439
27.1k73250439
answered May 15 '15 at 14:45
cgsellercgseller
1,9031218
answered May 15 '15 at 14:45
cgsellercgseller
1,9031218
answered May 15 '15 at 14:45
cgsellercgseller
1,9031218
1,9031218
31
If you want to use the prefix as well, you can do it like this:conn.list_objects(Bucket='bucket_name', Prefix='prefix_string')['Contents']
– markonovak
Mar 21 '16 at 13:14
8
This only lists the first 1000 keys. From the docstring: "Returns some or all (up to 1000) of the objects in a bucket." Also, it is recommended that you use list_objects_v2 instead of list_objects (although, this also only returns the first 1000 keys).
– Brett Widmeier
Mar 21 '18 at 14:18
add a comment |
31
If you want to use the prefix as well, you can do it like this:conn.list_objects(Bucket='bucket_name', Prefix='prefix_string')['Contents']
– markonovak
Mar 21 '16 at 13:14
8
This only lists the first 1000 keys. From the docstring: "Returns some or all (up to 1000) of the objects in a bucket." Also, it is recommended that you use list_objects_v2 instead of list_objects (although, this also only returns the first 1000 keys).
– Brett Widmeier
Mar 21 '18 at 14:18
31
31
If you want to use the prefix as well, you can do it like this:
conn.list_objects(Bucket='bucket_name', Prefix='prefix_string')['Contents']– markonovak
Mar 21 '16 at 13:14
If you want to use the prefix as well, you can do it like this:
conn.list_objects(Bucket='bucket_name', Prefix='prefix_string')['Contents']– markonovak
Mar 21 '16 at 13:14
8
8
This only lists the first 1000 keys. From the docstring: "Returns some or all (up to 1000) of the objects in a bucket." Also, it is recommended that you use list_objects_v2 instead of list_objects (although, this also only returns the first 1000 keys).
– Brett Widmeier
Mar 21 '18 at 14:18
This only lists the first 1000 keys. From the docstring: "Returns some or all (up to 1000) of the objects in a bucket." Also, it is recommended that you use list_objects_v2 instead of list_objects (although, this also only returns the first 1000 keys).
– Brett Widmeier
Mar 21 '18 at 14:18
add a comment |
20
If you want to pass the ACCESS and SECRET keys:
from boto3.session import Session
ACCESS_KEY='your_access_key'
SECRET_KEY='your_secret_key'
session = Session(aws_access_key_id=ACCESS_KEY,
aws_secret_access_key=SECRET_KEY)
s3 = session.resource('s3')
your_bucket = s3.Bucket('your_bucket')
for s3_file in your_bucket.objects.all():
print(s3_file.key)
answered Apr 7 '17 at 13:16
Erwin AlbertoErwin Alberto
40447
8
This is less secure than having a credentials file at ~/.aws/credentials. Though it is a valid solution.
– nu everest
Dec 27 '17 at 0:32
2
This would require committing secrets to source control. Not good.
– jan groth
Dec 7 '18 at 1:41
add a comment |
20
If you want to pass the ACCESS and SECRET keys:
from boto3.session import Session
ACCESS_KEY='your_access_key'
SECRET_KEY='your_secret_key'
session = Session(aws_access_key_id=ACCESS_KEY,
aws_secret_access_key=SECRET_KEY)
s3 = session.resource('s3')
your_bucket = s3.Bucket('your_bucket')
for s3_file in your_bucket.objects.all():
print(s3_file.key)
answered Apr 7 '17 at 13:16
Erwin AlbertoErwin Alberto
40447
8
This is less secure than having a credentials file at ~/.aws/credentials. Though it is a valid solution.
– nu everest
Dec 27 '17 at 0:32
2
This would require committing secrets to source control. Not good.
– jan groth
Dec 7 '18 at 1:41
add a comment |
20
20
20
If you want to pass the ACCESS and SECRET keys:
from boto3.session import Session
ACCESS_KEY='your_access_key'
SECRET_KEY='your_secret_key'
session = Session(aws_access_key_id=ACCESS_KEY,
aws_secret_access_key=SECRET_KEY)
s3 = session.resource('s3')
your_bucket = s3.Bucket('your_bucket')
for s3_file in your_bucket.objects.all():
print(s3_file.key)
answered Apr 7 '17 at 13:16
Erwin AlbertoErwin Alberto
40447
If you want to pass the ACCESS and SECRET keys:
from boto3.session import Session
ACCESS_KEY='your_access_key'
SECRET_KEY='your_secret_key'
session = Session(aws_access_key_id=ACCESS_KEY,
aws_secret_access_key=SECRET_KEY)
s3 = session.resource('s3')
your_bucket = s3.Bucket('your_bucket')
for s3_file in your_bucket.objects.all():
print(s3_file.key)
answered Apr 7 '17 at 13:16
Erwin AlbertoErwin Alberto
40447
answered Apr 7 '17 at 13:16
Erwin AlbertoErwin Alberto
40447
answered Apr 7 '17 at 13:16
Erwin AlbertoErwin Alberto
40447
answered Apr 7 '17 at 13:16
Erwin AlbertoErwin Alberto
40447
40447
8
This is less secure than having a credentials file at ~/.aws/credentials. Though it is a valid solution.
– nu everest
Dec 27 '17 at 0:32
2
This would require committing secrets to source control. Not good.
– jan groth
Dec 7 '18 at 1:41
add a comment |
8
This is less secure than having a credentials file at ~/.aws/credentials. Though it is a valid solution.
– nu everest
Dec 27 '17 at 0:32
2
This would require committing secrets to source control. Not good.
– jan groth
Dec 7 '18 at 1:41
8
8
This is less secure than having a credentials file at ~/.aws/credentials. Though it is a valid solution.
– nu everest
Dec 27 '17 at 0:32
This is less secure than having a credentials file at ~/.aws/credentials. Though it is a valid solution.
– nu everest
Dec 27 '17 at 0:32
2
2
This would require committing secrets to source control. Not good.
– jan groth
Dec 7 '18 at 1:41
This would require committing secrets to source control. Not good.
– jan groth
Dec 7 '18 at 1:41
add a comment |
16
I'm assuming you have configured authentication separately.
import boto3
s3 = boto3.resource('s3')
my_bucket = s3.Bucket('bucket_name')
for file in my_bucket.objects.all():
print file.key
edited Oct 17 '18 at 5:44
BMW
22.8k45371
answered Apr 5 '17 at 4:04
Tushar NirasTushar Niras
856914
add a comment |
16
I'm assuming you have configured authentication separately.
import boto3
s3 = boto3.resource('s3')
my_bucket = s3.Bucket('bucket_name')
for file in my_bucket.objects.all():
print file.key
edited Oct 17 '18 at 5:44
BMW
22.8k45371
answered Apr 5 '17 at 4:04
Tushar NirasTushar Niras
856914
add a comment |
16
16
16
I'm assuming you have configured authentication separately.
import boto3
s3 = boto3.resource('s3')
my_bucket = s3.Bucket('bucket_name')
for file in my_bucket.objects.all():
print file.key
edited Oct 17 '18 at 5:44
BMW
22.8k45371
answered Apr 5 '17 at 4:04
Tushar NirasTushar Niras
856914
I'm assuming you have configured authentication separately.
import boto3
s3 = boto3.resource('s3')
my_bucket = s3.Bucket('bucket_name')
for file in my_bucket.objects.all():
print file.key
edited Oct 17 '18 at 5:44
BMW
22.8k45371
answered Apr 5 '17 at 4:04
Tushar NirasTushar Niras
856914
edited Oct 17 '18 at 5:44
BMW
22.8k45371
edited Oct 17 '18 at 5:44
BMW
22.8k45371
edited Oct 17 '18 at 5:44
BMW
22.8k45371
22.8k45371
answered Apr 5 '17 at 4:04
Tushar NirasTushar Niras
856914
answered Apr 5 '17 at 4:04
Tushar NirasTushar Niras
856914
answered Apr 5 '17 at 4:04
Tushar NirasTushar Niras
856914
856914
add a comment |
add a comment |
8
In order to handle large key listings (i.e. when the directory list is greater than 1000 items), I used the following code to accumulate key values (i.e. filenames) with multiple listings (thanks to Amelio above for the first lines). Code is for python3:
from boto3 import client
bucket_name = "my_bucket"
prefix = "my_key/sub_key/lots_o_files"
s3_conn = client('s3') # type: BaseClient ## again assumes boto.cfg setup, assume AWS S3
s3_result = s3_conn.list_objects_v2(Bucket=bucket_name, Prefix=prefix, Delimiter = "/")
if 'Contents' not in s3_result:
#print(s3_result)
return
file_list =
for key in s3_result['Contents']:
file_list.append(key['Key'])
print(f"List count = {len(file_list)}")
while s3_result['IsTruncated']:
continuation_key = s3_result['NextContinuationToken']
s3_result = s3_conn.list_objects_v2(Bucket=bucket_name, Prefix=prefix, Delimiter="/", ContinuationToken=continuation_key)
for key in s3_result['Contents']:
file_list.append(key['Key'])
print(f"List count = {len(file_list)}")
return file_list
answered Nov 22 '18 at 1:17
HephaestusHephaestus
1,15621422
add a comment |
8
In order to handle large key listings (i.e. when the directory list is greater than 1000 items), I used the following code to accumulate key values (i.e. filenames) with multiple listings (thanks to Amelio above for the first lines). Code is for python3:
from boto3 import client
bucket_name = "my_bucket"
prefix = "my_key/sub_key/lots_o_files"
s3_conn = client('s3') # type: BaseClient ## again assumes boto.cfg setup, assume AWS S3
s3_result = s3_conn.list_objects_v2(Bucket=bucket_name, Prefix=prefix, Delimiter = "/")
if 'Contents' not in s3_result:
#print(s3_result)
return
file_list =
for key in s3_result['Contents']:
file_list.append(key['Key'])
print(f"List count = {len(file_list)}")
while s3_result['IsTruncated']:
continuation_key = s3_result['NextContinuationToken']
s3_result = s3_conn.list_objects_v2(Bucket=bucket_name, Prefix=prefix, Delimiter="/", ContinuationToken=continuation_key)
for key in s3_result['Contents']:
file_list.append(key['Key'])
print(f"List count = {len(file_list)}")
return file_list
answered Nov 22 '18 at 1:17
HephaestusHephaestus
1,15621422
add a comment |
8
8
8
In order to handle large key listings (i.e. when the directory list is greater than 1000 items), I used the following code to accumulate key values (i.e. filenames) with multiple listings (thanks to Amelio above for the first lines). Code is for python3:
from boto3 import client
bucket_name = "my_bucket"
prefix = "my_key/sub_key/lots_o_files"
s3_conn = client('s3') # type: BaseClient ## again assumes boto.cfg setup, assume AWS S3
s3_result = s3_conn.list_objects_v2(Bucket=bucket_name, Prefix=prefix, Delimiter = "/")
if 'Contents' not in s3_result:
#print(s3_result)
return
file_list =
for key in s3_result['Contents']:
file_list.append(key['Key'])
print(f"List count = {len(file_list)}")
while s3_result['IsTruncated']:
continuation_key = s3_result['NextContinuationToken']
s3_result = s3_conn.list_objects_v2(Bucket=bucket_name, Prefix=prefix, Delimiter="/", ContinuationToken=continuation_key)
for key in s3_result['Contents']:
file_list.append(key['Key'])
print(f"List count = {len(file_list)}")
return file_list
answered Nov 22 '18 at 1:17
HephaestusHephaestus
1,15621422
In order to handle large key listings (i.e. when the directory list is greater than 1000 items), I used the following code to accumulate key values (i.e. filenames) with multiple listings (thanks to Amelio above for the first lines). Code is for python3:
from boto3 import client
bucket_name = "my_bucket"
prefix = "my_key/sub_key/lots_o_files"
s3_conn = client('s3') # type: BaseClient ## again assumes boto.cfg setup, assume AWS S3
s3_result = s3_conn.list_objects_v2(Bucket=bucket_name, Prefix=prefix, Delimiter = "/")
if 'Contents' not in s3_result:
#print(s3_result)
return
file_list =
for key in s3_result['Contents']:
file_list.append(key['Key'])
print(f"List count = {len(file_list)}")
while s3_result['IsTruncated']:
continuation_key = s3_result['NextContinuationToken']
s3_result = s3_conn.list_objects_v2(Bucket=bucket_name, Prefix=prefix, Delimiter="/", ContinuationToken=continuation_key)
for key in s3_result['Contents']:
file_list.append(key['Key'])
print(f"List count = {len(file_list)}")
return file_list
answered Nov 22 '18 at 1:17
HephaestusHephaestus
1,15621422
edited Nov 22 '18 at 1:49
answered Nov 22 '18 at 1:17
HephaestusHephaestus
1,15621422
answered Nov 22 '18 at 1:17
HephaestusHephaestus
1,15621422
answered Nov 22 '18 at 1:17
HephaestusHephaestus
1,15621422
1,15621422
add a comment |
add a comment |
5
My s3 keys utility function is essentially an optimized version of @Hephaestus's answer:
import boto3
s3_paginator = boto3.client('s3').get_paginator('list_objects_v2')
def keys(bucket_name, prefix='/', delimiter='/', start_after=''):
prefix = prefix[1:] if prefix.startswith(delimiter) else prefix
start_after = (start_after or prefix) if prefix.endswith(delimiter) else start_after
for page in s3_paginator.paginate(Bucket=bucket_name, Prefix=prefix, StartAfter=start_after):
for content in page.get('Contents', ()):
yield content['Key']
In my tests (boto3 1.9.84), it's significantly faster than the equivalent (but simpler) code:
import boto3
def keys(bucket_name, prefix='/', delimiter='/'):
prefix = prefix[1:] if prefix.startswith(delimiter) else prefix
bucket = boto3.resource('s3').Bucket(bucket_name)
return (_.key for _ in bucket.objects.filter(Prefix=prefix))
As S3 guarantees UTF-8 binary sorted results, a start_after optimization has been added to the first function.
answered Jan 3 at 0:19
Sean SummersSean Summers
1,1501018
add a comment |
5
My s3 keys utility function is essentially an optimized version of @Hephaestus's answer:
import boto3
s3_paginator = boto3.client('s3').get_paginator('list_objects_v2')
def keys(bucket_name, prefix='/', delimiter='/', start_after=''):
prefix = prefix[1:] if prefix.startswith(delimiter) else prefix
start_after = (start_after or prefix) if prefix.endswith(delimiter) else start_after
for page in s3_paginator.paginate(Bucket=bucket_name, Prefix=prefix, StartAfter=start_after):
for content in page.get('Contents', ()):
yield content['Key']
In my tests (boto3 1.9.84), it's significantly faster than the equivalent (but simpler) code:
import boto3
def keys(bucket_name, prefix='/', delimiter='/'):
prefix = prefix[1:] if prefix.startswith(delimiter) else prefix
bucket = boto3.resource('s3').Bucket(bucket_name)
return (_.key for _ in bucket.objects.filter(Prefix=prefix))
As S3 guarantees UTF-8 binary sorted results, a start_after optimization has been added to the first function.
answered Jan 3 at 0:19
Sean SummersSean Summers
1,1501018
add a comment |
5
5
5
My s3 keys utility function is essentially an optimized version of @Hephaestus's answer:
import boto3
s3_paginator = boto3.client('s3').get_paginator('list_objects_v2')
def keys(bucket_name, prefix='/', delimiter='/', start_after=''):
prefix = prefix[1:] if prefix.startswith(delimiter) else prefix
start_after = (start_after or prefix) if prefix.endswith(delimiter) else start_after
for page in s3_paginator.paginate(Bucket=bucket_name, Prefix=prefix, StartAfter=start_after):
for content in page.get('Contents', ()):
yield content['Key']
In my tests (boto3 1.9.84), it's significantly faster than the equivalent (but simpler) code:
import boto3
def keys(bucket_name, prefix='/', delimiter='/'):
prefix = prefix[1:] if prefix.startswith(delimiter) else prefix
bucket = boto3.resource('s3').Bucket(bucket_name)
return (_.key for _ in bucket.objects.filter(Prefix=prefix))
As S3 guarantees UTF-8 binary sorted results, a start_after optimization has been added to the first function.
answered Jan 3 at 0:19
Sean SummersSean Summers
1,1501018
My s3 keys utility function is essentially an optimized version of @Hephaestus's answer:
import boto3
s3_paginator = boto3.client('s3').get_paginator('list_objects_v2')
def keys(bucket_name, prefix='/', delimiter='/', start_after=''):
prefix = prefix[1:] if prefix.startswith(delimiter) else prefix
start_after = (start_after or prefix) if prefix.endswith(delimiter) else start_after
for page in s3_paginator.paginate(Bucket=bucket_name, Prefix=prefix, StartAfter=start_after):
for content in page.get('Contents', ()):
yield content['Key']
In my tests (boto3 1.9.84), it's significantly faster than the equivalent (but simpler) code:
import boto3
def keys(bucket_name, prefix='/', delimiter='/'):
prefix = prefix[1:] if prefix.startswith(delimiter) else prefix
bucket = boto3.resource('s3').Bucket(bucket_name)
return (_.key for _ in bucket.objects.filter(Prefix=prefix))
As S3 guarantees UTF-8 binary sorted results, a start_after optimization has been added to the first function.
answered Jan 3 at 0:19
Sean SummersSean Summers
1,1501018
edited Feb 19 at 21:34
answered Jan 3 at 0:19
Sean SummersSean Summers
1,1501018
answered Jan 3 at 0:19
Sean SummersSean Summers
1,1501018
answered Jan 3 at 0:19
Sean SummersSean Summers
1,1501018
1,1501018
add a comment |
add a comment |
4
A more parsimonious way, rather than iterating through via a for loop you could also just print the original object containing all files inside your S3 bucket:
session = Session(aws_access_key_id=aws_access_key_id,aws_secret_access_key=aws_secret_access_key)
s3 = session.resource('s3')
bucket = s3.Bucket('bucket_name')
files_in_s3 = bucket.objects.all()
#you can print this iterable with print(list(files_in_s3))
edited Aug 23 '18 at 14:21
petezurich
3,65081834
answered Jun 24 '17 at 7:14
Daniel VieiraDaniel Vieira
1236
@petezurich , can you please explain why such a petty edit of my answer - replacing an “a” with a capital “A” at the beginning of my answer brought down my reputation by -2 , however I reckon both you and I can agree that not only is your correction NOT Relevant at all, but actually rather petty, wouldn’t you say so? Please focus on the content rather than childish revisions , most obliged ol’boy
– Daniel Vieira
Aug 23 '18 at 13:41
These were two different interactions. 1. I edited your answer which is recommended even for minor misspellings. I agree, that the boundaries between minor and trivial are ambiguous. I do not downvote any post because I see errors and I didn't in this case. I simply fix all the errors that I see.
– petezurich
Aug 23 '18 at 14:26
2. I downvoted your answer because you wrote thatfiles_in_s3is a "list object". There is no such thing in Python. It rather is an iterable and I couldn't make your code work and therefore downvoted. Than I found the error and saw your point but couldn't undo my downvote.
– petezurich
Aug 23 '18 at 14:29
I now have edited your answer again and could undo the downvote. Feel free to reedit.
– petezurich
Aug 23 '18 at 14:30
1
@petezurich no problem , understood your , point , just one thing, in Python a list IS an object because pretty much everything in python is an object , then it also follows that a list is also an iterable, but first and foremost , it’s an object! that is why I did not understand your downvote- you were down voting something that was correct and code that works. Anyway , thanks for your apology and all the best
– Daniel Vieira
Aug 23 '18 at 19:02
|
show 1 more comment
4
A more parsimonious way, rather than iterating through via a for loop you could also just print the original object containing all files inside your S3 bucket:
session = Session(aws_access_key_id=aws_access_key_id,aws_secret_access_key=aws_secret_access_key)
s3 = session.resource('s3')
bucket = s3.Bucket('bucket_name')
files_in_s3 = bucket.objects.all()
#you can print this iterable with print(list(files_in_s3))
edited Aug 23 '18 at 14:21
petezurich
3,65081834
answered Jun 24 '17 at 7:14
Daniel VieiraDaniel Vieira
1236
@petezurich , can you please explain why such a petty edit of my answer - replacing an “a” with a capital “A” at the beginning of my answer brought down my reputation by -2 , however I reckon both you and I can agree that not only is your correction NOT Relevant at all, but actually rather petty, wouldn’t you say so? Please focus on the content rather than childish revisions , most obliged ol’boy
– Daniel Vieira
Aug 23 '18 at 13:41
These were two different interactions. 1. I edited your answer which is recommended even for minor misspellings. I agree, that the boundaries between minor and trivial are ambiguous. I do not downvote any post because I see errors and I didn't in this case. I simply fix all the errors that I see.
– petezurich
Aug 23 '18 at 14:26
2. I downvoted your answer because you wrote thatfiles_in_s3is a "list object". There is no such thing in Python. It rather is an iterable and I couldn't make your code work and therefore downvoted. Than I found the error and saw your point but couldn't undo my downvote.
– petezurich
Aug 23 '18 at 14:29
I now have edited your answer again and could undo the downvote. Feel free to reedit.
– petezurich
Aug 23 '18 at 14:30
1
@petezurich no problem , understood your , point , just one thing, in Python a list IS an object because pretty much everything in python is an object , then it also follows that a list is also an iterable, but first and foremost , it’s an object! that is why I did not understand your downvote- you were down voting something that was correct and code that works. Anyway , thanks for your apology and all the best
– Daniel Vieira
Aug 23 '18 at 19:02
|
show 1 more comment
4
4
4
A more parsimonious way, rather than iterating through via a for loop you could also just print the original object containing all files inside your S3 bucket:
session = Session(aws_access_key_id=aws_access_key_id,aws_secret_access_key=aws_secret_access_key)
s3 = session.resource('s3')
bucket = s3.Bucket('bucket_name')
files_in_s3 = bucket.objects.all()
#you can print this iterable with print(list(files_in_s3))
edited Aug 23 '18 at 14:21
petezurich
3,65081834
answered Jun 24 '17 at 7:14
Daniel VieiraDaniel Vieira
1236
A more parsimonious way, rather than iterating through via a for loop you could also just print the original object containing all files inside your S3 bucket:
session = Session(aws_access_key_id=aws_access_key_id,aws_secret_access_key=aws_secret_access_key)
s3 = session.resource('s3')
bucket = s3.Bucket('bucket_name')
files_in_s3 = bucket.objects.all()
#you can print this iterable with print(list(files_in_s3))
edited Aug 23 '18 at 14:21
petezurich
3,65081834
answered Jun 24 '17 at 7:14
Daniel VieiraDaniel Vieira
1236
edited Aug 23 '18 at 14:21
petezurich
3,65081834
edited Aug 23 '18 at 14:21
petezurich
3,65081834
edited Aug 23 '18 at 14:21
petezurich
3,65081834
3,65081834
answered Jun 24 '17 at 7:14
Daniel VieiraDaniel Vieira
1236
answered Jun 24 '17 at 7:14
Daniel VieiraDaniel Vieira
1236
answered Jun 24 '17 at 7:14
Daniel VieiraDaniel Vieira
1236
1236
@petezurich , can you please explain why such a petty edit of my answer - replacing an “a” with a capital “A” at the beginning of my answer brought down my reputation by -2 , however I reckon both you and I can agree that not only is your correction NOT Relevant at all, but actually rather petty, wouldn’t you say so? Please focus on the content rather than childish revisions , most obliged ol’boy
– Daniel Vieira
Aug 23 '18 at 13:41
These were two different interactions. 1. I edited your answer which is recommended even for minor misspellings. I agree, that the boundaries between minor and trivial are ambiguous. I do not downvote any post because I see errors and I didn't in this case. I simply fix all the errors that I see.
– petezurich
Aug 23 '18 at 14:26
2. I downvoted your answer because you wrote thatfiles_in_s3is a "list object". There is no such thing in Python. It rather is an iterable and I couldn't make your code work and therefore downvoted. Than I found the error and saw your point but couldn't undo my downvote.
– petezurich
Aug 23 '18 at 14:29
I now have edited your answer again and could undo the downvote. Feel free to reedit.
– petezurich
Aug 23 '18 at 14:30
1
@petezurich no problem , understood your , point , just one thing, in Python a list IS an object because pretty much everything in python is an object , then it also follows that a list is also an iterable, but first and foremost , it’s an object! that is why I did not understand your downvote- you were down voting something that was correct and code that works. Anyway , thanks for your apology and all the best
– Daniel Vieira
Aug 23 '18 at 19:02
|
show 1 more comment
@petezurich , can you please explain why such a petty edit of my answer - replacing an “a” with a capital “A” at the beginning of my answer brought down my reputation by -2 , however I reckon both you and I can agree that not only is your correction NOT Relevant at all, but actually rather petty, wouldn’t you say so? Please focus on the content rather than childish revisions , most obliged ol’boy
– Daniel Vieira
Aug 23 '18 at 13:41
These were two different interactions. 1. I edited your answer which is recommended even for minor misspellings. I agree, that the boundaries between minor and trivial are ambiguous. I do not downvote any post because I see errors and I didn't in this case. I simply fix all the errors that I see.
– petezurich
Aug 23 '18 at 14:26
2. I downvoted your answer because you wrote thatfiles_in_s3is a "list object". There is no such thing in Python. It rather is an iterable and I couldn't make your code work and therefore downvoted. Than I found the error and saw your point but couldn't undo my downvote.
– petezurich
Aug 23 '18 at 14:29
I now have edited your answer again and could undo the downvote. Feel free to reedit.
– petezurich
Aug 23 '18 at 14:30
1
@petezurich no problem , understood your , point , just one thing, in Python a list IS an object because pretty much everything in python is an object , then it also follows that a list is also an iterable, but first and foremost , it’s an object! that is why I did not understand your downvote- you were down voting something that was correct and code that works. Anyway , thanks for your apology and all the best
– Daniel Vieira
Aug 23 '18 at 19:02
@petezurich , can you please explain why such a petty edit of my answer - replacing an “a” with a capital “A” at the beginning of my answer brought down my reputation by -2 , however I reckon both you and I can agree that not only is your correction NOT Relevant at all, but actually rather petty, wouldn’t you say so? Please focus on the content rather than childish revisions , most obliged ol’boy
– Daniel Vieira
Aug 23 '18 at 13:41
@petezurich , can you please explain why such a petty edit of my answer - replacing an “a” with a capital “A” at the beginning of my answer brought down my reputation by -2 , however I reckon both you and I can agree that not only is your correction NOT Relevant at all, but actually rather petty, wouldn’t you say so? Please focus on the content rather than childish revisions , most obliged ol’boy
– Daniel Vieira
Aug 23 '18 at 13:41
These were two different interactions. 1. I edited your answer which is recommended even for minor misspellings. I agree, that the boundaries between minor and trivial are ambiguous. I do not downvote any post because I see errors and I didn't in this case. I simply fix all the errors that I see.
– petezurich
Aug 23 '18 at 14:26
These were two different interactions. 1. I edited your answer which is recommended even for minor misspellings. I agree, that the boundaries between minor and trivial are ambiguous. I do not downvote any post because I see errors and I didn't in this case. I simply fix all the errors that I see.
– petezurich
Aug 23 '18 at 14:26
2. I downvoted your answer because you wrote that
files_in_s3is a "list object". There is no such thing in Python. It rather is an iterable and I couldn't make your code work and therefore downvoted. Than I found the error and saw your point but couldn't undo my downvote.– petezurich
Aug 23 '18 at 14:29
2. I downvoted your answer because you wrote that
files_in_s3is a "list object". There is no such thing in Python. It rather is an iterable and I couldn't make your code work and therefore downvoted. Than I found the error and saw your point but couldn't undo my downvote.– petezurich
Aug 23 '18 at 14:29
I now have edited your answer again and could undo the downvote. Feel free to reedit.
– petezurich
Aug 23 '18 at 14:30
I now have edited your answer again and could undo the downvote. Feel free to reedit.
– petezurich
Aug 23 '18 at 14:30
1
1
@petezurich no problem , understood your , point , just one thing, in Python a list IS an object because pretty much everything in python is an object , then it also follows that a list is also an iterable, but first and foremost , it’s an object! that is why I did not understand your downvote- you were down voting something that was correct and code that works. Anyway , thanks for your apology and all the best
– Daniel Vieira
Aug 23 '18 at 19:02
@petezurich no problem , understood your , point , just one thing, in Python a list IS an object because pretty much everything in python is an object , then it also follows that a list is also an iterable, but first and foremost , it’s an object! that is why I did not understand your downvote- you were down voting something that was correct and code that works. Anyway , thanks for your apology and all the best
– Daniel Vieira
Aug 23 '18 at 19:02
|
show 1 more comment
1
ObjectSummary:
There are two identifiers that are attached to the ObjectSummary:
- bucket_name
- key
boto3 S3: ObjectSummary
More on Object Keys from AWS S3 Documentation:
Object Keys:
When you create an object, you specify the key name, which uniquely identifies the object in the bucket. For example, in the Amazon S3 console (see AWS Management Console), when you highlight a bucket, a list of objects in your bucket appears. These names are the object keys. The name for a key is a sequence of Unicode characters whose UTF-8 encoding is at most 1024 bytes long.
The Amazon S3 data model is a flat structure: you create a bucket, and the bucket stores objects. There is no hierarchy of subbuckets or subfolders; however, you can infer logical hierarchy using key name prefixes and delimiters as the Amazon S3 console does. The Amazon S3 console supports a concept of folders. Suppose that your bucket (admin-created) has four objects with the following object keys:
Development/Projects1.xls
Finance/statement1.pdf
Private/taxdocument.pdf
s3-dg.pdf
Reference:
AWS S3: Object Keys
Here is some example code that demonstrates how to get the bucket name and the object key.
Example:
import boto3
from pprint import pprint
def main():
def enumerate_s3():
s3 = boto3.resource('s3')
for bucket in s3.buckets.all():
print("Name: {}".format(bucket.name))
print("Creation Date: {}".format(bucket.creation_date))
for object in bucket.objects.all():
print("Object: {}".format(object))
print("Object bucket_name: {}".format(object.bucket_name))
print("Object key: {}".format(object.key))
enumerate_s3()
if __name__ == '__main__':
main()
answered Oct 16 '18 at 19:26
GothburzGothburz
2,25611119
add a comment |
1
ObjectSummary:
There are two identifiers that are attached to the ObjectSummary:
- bucket_name
- key
boto3 S3: ObjectSummary
More on Object Keys from AWS S3 Documentation:
Object Keys:
When you create an object, you specify the key name, which uniquely identifies the object in the bucket. For example, in the Amazon S3 console (see AWS Management Console), when you highlight a bucket, a list of objects in your bucket appears. These names are the object keys. The name for a key is a sequence of Unicode characters whose UTF-8 encoding is at most 1024 bytes long.
The Amazon S3 data model is a flat structure: you create a bucket, and the bucket stores objects. There is no hierarchy of subbuckets or subfolders; however, you can infer logical hierarchy using key name prefixes and delimiters as the Amazon S3 console does. The Amazon S3 console supports a concept of folders. Suppose that your bucket (admin-created) has four objects with the following object keys:
Development/Projects1.xls
Finance/statement1.pdf
Private/taxdocument.pdf
s3-dg.pdf
Reference:
AWS S3: Object Keys
Here is some example code that demonstrates how to get the bucket name and the object key.
Example:
import boto3
from pprint import pprint
def main():
def enumerate_s3():
s3 = boto3.resource('s3')
for bucket in s3.buckets.all():
print("Name: {}".format(bucket.name))
print("Creation Date: {}".format(bucket.creation_date))
for object in bucket.objects.all():
print("Object: {}".format(object))
print("Object bucket_name: {}".format(object.bucket_name))
print("Object key: {}".format(object.key))
enumerate_s3()
if __name__ == '__main__':
main()
answered Oct 16 '18 at 19:26
GothburzGothburz
2,25611119
add a comment |
1
1
1
ObjectSummary:
There are two identifiers that are attached to the ObjectSummary:
- bucket_name
- key
boto3 S3: ObjectSummary
More on Object Keys from AWS S3 Documentation:
Object Keys:
When you create an object, you specify the key name, which uniquely identifies the object in the bucket. For example, in the Amazon S3 console (see AWS Management Console), when you highlight a bucket, a list of objects in your bucket appears. These names are the object keys. The name for a key is a sequence of Unicode characters whose UTF-8 encoding is at most 1024 bytes long.
The Amazon S3 data model is a flat structure: you create a bucket, and the bucket stores objects. There is no hierarchy of subbuckets or subfolders; however, you can infer logical hierarchy using key name prefixes and delimiters as the Amazon S3 console does. The Amazon S3 console supports a concept of folders. Suppose that your bucket (admin-created) has four objects with the following object keys:
Development/Projects1.xls
Finance/statement1.pdf
Private/taxdocument.pdf
s3-dg.pdf
Reference:
AWS S3: Object Keys
Here is some example code that demonstrates how to get the bucket name and the object key.
Example:
import boto3
from pprint import pprint
def main():
def enumerate_s3():
s3 = boto3.resource('s3')
for bucket in s3.buckets.all():
print("Name: {}".format(bucket.name))
print("Creation Date: {}".format(bucket.creation_date))
for object in bucket.objects.all():
print("Object: {}".format(object))
print("Object bucket_name: {}".format(object.bucket_name))
print("Object key: {}".format(object.key))
enumerate_s3()
if __name__ == '__main__':
main()
answered Oct 16 '18 at 19:26
GothburzGothburz
2,25611119
ObjectSummary:
There are two identifiers that are attached to the ObjectSummary:
- bucket_name
- key
boto3 S3: ObjectSummary
More on Object Keys from AWS S3 Documentation:
Object Keys:
When you create an object, you specify the key name, which uniquely identifies the object in the bucket. For example, in the Amazon S3 console (see AWS Management Console), when you highlight a bucket, a list of objects in your bucket appears. These names are the object keys. The name for a key is a sequence of Unicode characters whose UTF-8 encoding is at most 1024 bytes long.
The Amazon S3 data model is a flat structure: you create a bucket, and the bucket stores objects. There is no hierarchy of subbuckets or subfolders; however, you can infer logical hierarchy using key name prefixes and delimiters as the Amazon S3 console does. The Amazon S3 console supports a concept of folders. Suppose that your bucket (admin-created) has four objects with the following object keys:
Development/Projects1.xls
Finance/statement1.pdf
Private/taxdocument.pdf
s3-dg.pdf
Reference:
AWS S3: Object Keys
Here is some example code that demonstrates how to get the bucket name and the object key.
Example:
import boto3
from pprint import pprint
def main():
def enumerate_s3():
s3 = boto3.resource('s3')
for bucket in s3.buckets.all():
print("Name: {}".format(bucket.name))
print("Creation Date: {}".format(bucket.creation_date))
for object in bucket.objects.all():
print("Object: {}".format(object))
print("Object bucket_name: {}".format(object.bucket_name))
print("Object key: {}".format(object.key))
enumerate_s3()
if __name__ == '__main__':
main()
answered Oct 16 '18 at 19:26
GothburzGothburz
2,25611119
answered Oct 16 '18 at 19:26
GothburzGothburz
2,25611119
answered Oct 16 '18 at 19:26
GothburzGothburz
2,25611119
answered Oct 16 '18 at 19:26
GothburzGothburz
2,25611119
2,25611119
add a comment |
add a comment |
0
I just did it like this, including the authentication method:
s3_client = boto3.client(
's3',
aws_access_key_id='access_key',
aws_secret_access_key='access_key_secret',
config=boto3.session.Config(signature_version='s3v4'),
region_name='region'
)
response = s3_client.list_objects(Bucket='bucket_name', Prefix=key)
if ('Contents' in response):
# Object / key exists!
return True
else:
# Object / key DOES NOT exist!
return False
answered Sep 30 '18 at 19:21
MileanMilean
678615
add a comment |
0
I just did it like this, including the authentication method:
s3_client = boto3.client(
's3',
aws_access_key_id='access_key',
aws_secret_access_key='access_key_secret',
config=boto3.session.Config(signature_version='s3v4'),
region_name='region'
)
response = s3_client.list_objects(Bucket='bucket_name', Prefix=key)
if ('Contents' in response):
# Object / key exists!
return True
else:
# Object / key DOES NOT exist!
return False
answered Sep 30 '18 at 19:21
MileanMilean
678615
add a comment |
0
0
0
I just did it like this, including the authentication method:
s3_client = boto3.client(
's3',
aws_access_key_id='access_key',
aws_secret_access_key='access_key_secret',
config=boto3.session.Config(signature_version='s3v4'),
region_name='region'
)
response = s3_client.list_objects(Bucket='bucket_name', Prefix=key)
if ('Contents' in response):
# Object / key exists!
return True
else:
# Object / key DOES NOT exist!
return False
answered Sep 30 '18 at 19:21
MileanMilean
678615
I just did it like this, including the authentication method:
s3_client = boto3.client(
's3',
aws_access_key_id='access_key',
aws_secret_access_key='access_key_secret',
config=boto3.session.Config(signature_version='s3v4'),
region_name='region'
)
response = s3_client.list_objects(Bucket='bucket_name', Prefix=key)
if ('Contents' in response):
# Object / key exists!
return True
else:
# Object / key DOES NOT exist!
return False
answered Sep 30 '18 at 19:21
MileanMilean
678615
answered Sep 30 '18 at 19:21
MileanMilean
678615
answered Sep 30 '18 at 19:21
MileanMilean
678615
answered Sep 30 '18 at 19:21
MileanMilean
678615
678615
add a comment |
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f30249069%2flisting-contents-of-a-bucket-with-boto3%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
