Mixing
AWS CLI: aws sync between 2 different s3 providers
Does anybody have a solution to sync a bucket between 2 different s3 providers?
For example one is Amazon S3, second is Wasabi S3?
That involves 2 different endpoints and 2 different sets of credentials.
Preferable without storage data locally first, talking about 1+ million files.
amazon-web-services amazon-s3
edited Nov 19 '18 at 22:32
John Rotenstein
68.3k775119
asked Nov 19 '18 at 19:19
HyperDevilHyperDevil
89742239
add a comment |
Does anybody have a solution to sync a bucket between 2 different s3 providers?
For example one is Amazon S3, second is Wasabi S3?
That involves 2 different endpoints and 2 different sets of credentials.
Preferable without storage data locally first, talking about 1+ million files.
amazon-web-services amazon-s3
edited Nov 19 '18 at 22:32
John Rotenstein
68.3k775119
asked Nov 19 '18 at 19:19
HyperDevilHyperDevil
89742239
add a comment |
Does anybody have a solution to sync a bucket between 2 different s3 providers?
For example one is Amazon S3, second is Wasabi S3?
That involves 2 different endpoints and 2 different sets of credentials.
Preferable without storage data locally first, talking about 1+ million files.
amazon-web-services amazon-s3
edited Nov 19 '18 at 22:32
John Rotenstein
68.3k775119
asked Nov 19 '18 at 19:19
HyperDevilHyperDevil
89742239
Does anybody have a solution to sync a bucket between 2 different s3 providers?
For example one is Amazon S3, second is Wasabi S3?
That involves 2 different endpoints and 2 different sets of credentials.
Preferable without storage data locally first, talking about 1+ million files.
amazon-web-services amazon-s3
amazon-web-services amazon-s3
edited Nov 19 '18 at 22:32
John Rotenstein
68.3k775119
asked Nov 19 '18 at 19:19
HyperDevilHyperDevil
89742239
edited Nov 19 '18 at 22:32
John Rotenstein
68.3k775119
asked Nov 19 '18 at 19:19
HyperDevilHyperDevil
89742239
edited Nov 19 '18 at 22:32
John Rotenstein
68.3k775119
edited Nov 19 '18 at 22:32
John Rotenstein
68.3k775119
edited Nov 19 '18 at 22:32
John Rotenstein
68.3k775119
68.3k775119
asked Nov 19 '18 at 19:19
HyperDevilHyperDevil
89742239
asked Nov 19 '18 at 19:19
HyperDevilHyperDevil
89742239
asked Nov 19 '18 at 19:19
HyperDevilHyperDevil
89742239
89742239
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
This would not be possible.
In Amazon S3, it is possible to copy directly between two buckets, even in different regions, because the S3 services in each region communicate with each other.
This wouldn't be possible with a non-Amazon service.
Therefore, your best bet would be to launch an EC2 instance, use aws s3 sync to download the files, then use aws s3 sync (with a different endpoint & credentials) to connect to Wasabi.
It is possible that the S3-compatible providers might offer an easier method, but it appears that Wasabi recommends the 'download and copy' method: How do I copy files from AWS S3 to Wasabi? – Wasabi Knowledge Base
answered Nov 19 '18 at 22:35
John RotensteinJohn Rotenstein
68.3k775119
Thanks. i was afraid of this, so this keeps me locked in to AWS for now. The sync command after 10 days does not ever finish because of the insane amount of objects.
– HyperDevil
Nov 21 '18 at 8:33
Is this the sync to download, or to upload? Another method is to Export Data from Amazon S3 with Snowball - AWS Snowball, but then you'd have to upload from your own computer to Wasabi.
– John Rotenstein
Nov 21 '18 at 9:41
This is to sync 2 buckets for backup purposes.
– HyperDevil
Nov 23 '18 at 7:05
If you wish to have a second copy of the data (well, Amazon S3 already keeps copies in 3 data centers, but if you want a copy in a different location), then you can use Amazon S3 Cross-Region Replication that will automatically copy the data to a different region (where it would again be kept in 3 different data centers).
– John Rotenstein
Nov 23 '18 at 7:37
It has to do with regulatory requirements, one should not only depend on only one technical provider.
– HyperDevil
Nov 26 '18 at 13:45
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53381254%2faws-cli-aws-sync-between-2-different-s3-providers%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
This would not be possible.
In Amazon S3, it is possible to copy directly between two buckets, even in different regions, because the S3 services in each region communicate with each other.
This wouldn't be possible with a non-Amazon service.
Therefore, your best bet would be to launch an EC2 instance, use aws s3 sync to download the files, then use aws s3 sync (with a different endpoint & credentials) to connect to Wasabi.
It is possible that the S3-compatible providers might offer an easier method, but it appears that Wasabi recommends the 'download and copy' method: How do I copy files from AWS S3 to Wasabi? – Wasabi Knowledge Base
answered Nov 19 '18 at 22:35
John RotensteinJohn Rotenstein
68.3k775119
Thanks. i was afraid of this, so this keeps me locked in to AWS for now. The sync command after 10 days does not ever finish because of the insane amount of objects.
– HyperDevil
Nov 21 '18 at 8:33
Is this the sync to download, or to upload? Another method is to Export Data from Amazon S3 with Snowball - AWS Snowball, but then you'd have to upload from your own computer to Wasabi.
– John Rotenstein
Nov 21 '18 at 9:41
This is to sync 2 buckets for backup purposes.
– HyperDevil
Nov 23 '18 at 7:05
If you wish to have a second copy of the data (well, Amazon S3 already keeps copies in 3 data centers, but if you want a copy in a different location), then you can use Amazon S3 Cross-Region Replication that will automatically copy the data to a different region (where it would again be kept in 3 different data centers).
– John Rotenstein
Nov 23 '18 at 7:37
It has to do with regulatory requirements, one should not only depend on only one technical provider.
– HyperDevil
Nov 26 '18 at 13:45
add a comment |
This would not be possible.
In Amazon S3, it is possible to copy directly between two buckets, even in different regions, because the S3 services in each region communicate with each other.
This wouldn't be possible with a non-Amazon service.
Therefore, your best bet would be to launch an EC2 instance, use aws s3 sync to download the files, then use aws s3 sync (with a different endpoint & credentials) to connect to Wasabi.
It is possible that the S3-compatible providers might offer an easier method, but it appears that Wasabi recommends the 'download and copy' method: How do I copy files from AWS S3 to Wasabi? – Wasabi Knowledge Base
answered Nov 19 '18 at 22:35
John RotensteinJohn Rotenstein
68.3k775119
Thanks. i was afraid of this, so this keeps me locked in to AWS for now. The sync command after 10 days does not ever finish because of the insane amount of objects.
– HyperDevil
Nov 21 '18 at 8:33
Is this the sync to download, or to upload? Another method is to Export Data from Amazon S3 with Snowball - AWS Snowball, but then you'd have to upload from your own computer to Wasabi.
– John Rotenstein
Nov 21 '18 at 9:41
This is to sync 2 buckets for backup purposes.
– HyperDevil
Nov 23 '18 at 7:05
If you wish to have a second copy of the data (well, Amazon S3 already keeps copies in 3 data centers, but if you want a copy in a different location), then you can use Amazon S3 Cross-Region Replication that will automatically copy the data to a different region (where it would again be kept in 3 different data centers).
– John Rotenstein
Nov 23 '18 at 7:37
It has to do with regulatory requirements, one should not only depend on only one technical provider.
– HyperDevil
Nov 26 '18 at 13:45
add a comment |
This would not be possible.
In Amazon S3, it is possible to copy directly between two buckets, even in different regions, because the S3 services in each region communicate with each other.
This wouldn't be possible with a non-Amazon service.
Therefore, your best bet would be to launch an EC2 instance, use aws s3 sync to download the files, then use aws s3 sync (with a different endpoint & credentials) to connect to Wasabi.
It is possible that the S3-compatible providers might offer an easier method, but it appears that Wasabi recommends the 'download and copy' method: How do I copy files from AWS S3 to Wasabi? – Wasabi Knowledge Base
answered Nov 19 '18 at 22:35
John RotensteinJohn Rotenstein
68.3k775119
This would not be possible.
In Amazon S3, it is possible to copy directly between two buckets, even in different regions, because the S3 services in each region communicate with each other.
This wouldn't be possible with a non-Amazon service.
Therefore, your best bet would be to launch an EC2 instance, use aws s3 sync to download the files, then use aws s3 sync (with a different endpoint & credentials) to connect to Wasabi.
It is possible that the S3-compatible providers might offer an easier method, but it appears that Wasabi recommends the 'download and copy' method: How do I copy files from AWS S3 to Wasabi? – Wasabi Knowledge Base
answered Nov 19 '18 at 22:35
John RotensteinJohn Rotenstein
68.3k775119
answered Nov 19 '18 at 22:35
John RotensteinJohn Rotenstein
68.3k775119
answered Nov 19 '18 at 22:35
John RotensteinJohn Rotenstein
68.3k775119
answered Nov 19 '18 at 22:35
John RotensteinJohn Rotenstein
68.3k775119
68.3k775119
Thanks. i was afraid of this, so this keeps me locked in to AWS for now. The sync command after 10 days does not ever finish because of the insane amount of objects.
– HyperDevil
Nov 21 '18 at 8:33
Is this the sync to download, or to upload? Another method is to Export Data from Amazon S3 with Snowball - AWS Snowball, but then you'd have to upload from your own computer to Wasabi.
– John Rotenstein
Nov 21 '18 at 9:41
This is to sync 2 buckets for backup purposes.
– HyperDevil
Nov 23 '18 at 7:05
If you wish to have a second copy of the data (well, Amazon S3 already keeps copies in 3 data centers, but if you want a copy in a different location), then you can use Amazon S3 Cross-Region Replication that will automatically copy the data to a different region (where it would again be kept in 3 different data centers).
– John Rotenstein
Nov 23 '18 at 7:37
It has to do with regulatory requirements, one should not only depend on only one technical provider.
– HyperDevil
Nov 26 '18 at 13:45
add a comment |
Thanks. i was afraid of this, so this keeps me locked in to AWS for now. The sync command after 10 days does not ever finish because of the insane amount of objects.
– HyperDevil
Nov 21 '18 at 8:33
Is this the sync to download, or to upload? Another method is to Export Data from Amazon S3 with Snowball - AWS Snowball, but then you'd have to upload from your own computer to Wasabi.
– John Rotenstein
Nov 21 '18 at 9:41
This is to sync 2 buckets for backup purposes.
– HyperDevil
Nov 23 '18 at 7:05
If you wish to have a second copy of the data (well, Amazon S3 already keeps copies in 3 data centers, but if you want a copy in a different location), then you can use Amazon S3 Cross-Region Replication that will automatically copy the data to a different region (where it would again be kept in 3 different data centers).
– John Rotenstein
Nov 23 '18 at 7:37
It has to do with regulatory requirements, one should not only depend on only one technical provider.
– HyperDevil
Nov 26 '18 at 13:45
Thanks. i was afraid of this, so this keeps me locked in to AWS for now. The sync command after 10 days does not ever finish because of the insane amount of objects.
– HyperDevil
Nov 21 '18 at 8:33
Thanks. i was afraid of this, so this keeps me locked in to AWS for now. The sync command after 10 days does not ever finish because of the insane amount of objects.
– HyperDevil
Nov 21 '18 at 8:33
Is this the sync to download, or to upload? Another method is to Export Data from Amazon S3 with Snowball - AWS Snowball, but then you'd have to upload from your own computer to Wasabi.
– John Rotenstein
Nov 21 '18 at 9:41
Is this the sync to download, or to upload? Another method is to Export Data from Amazon S3 with Snowball - AWS Snowball, but then you'd have to upload from your own computer to Wasabi.
– John Rotenstein
Nov 21 '18 at 9:41
This is to sync 2 buckets for backup purposes.
– HyperDevil
Nov 23 '18 at 7:05
This is to sync 2 buckets for backup purposes.
– HyperDevil
Nov 23 '18 at 7:05
If you wish to have a second copy of the data (well, Amazon S3 already keeps copies in 3 data centers, but if you want a copy in a different location), then you can use Amazon S3 Cross-Region Replication that will automatically copy the data to a different region (where it would again be kept in 3 different data centers).
– John Rotenstein
Nov 23 '18 at 7:37
If you wish to have a second copy of the data (well, Amazon S3 already keeps copies in 3 data centers, but if you want a copy in a different location), then you can use Amazon S3 Cross-Region Replication that will automatically copy the data to a different region (where it would again be kept in 3 different data centers).
– John Rotenstein
Nov 23 '18 at 7:37
It has to do with regulatory requirements, one should not only depend on only one technical provider.
– HyperDevil
Nov 26 '18 at 13:45
It has to do with regulatory requirements, one should not only depend on only one technical provider.
– HyperDevil
Nov 26 '18 at 13:45
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53381254%2faws-cli-aws-sync-between-2-different-s3-providers%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
