Mixing
Download via http and streaming upload to Google Cloud storage using nodejs
I am fighting to get a download to directly stream to GCS without saving to the file system, see below snippet.
const { Storage } = require('@google-cloud/storage');
const http = require('http');
const storage = new Storage();
const fs = require("fs");
const bucketName = 'BUCKETNAMEHERE';
const blobName = 'image.jpg';
const bucket = storage.bucket(bucketName);
const blob = bucket.file(blobName);
const streamDownload = () => {
http.get("http://i3.ytimg.com/vi/J---aiyznGQ/mqdefault.jpg")
.pipe(blob.createWriteStream({
metadata: {
contentType: 'image/jpg'
}
}))
.on("error", (err) => {
console.error(`error occurred`);
})
.on('finish', () => {
console.info(`success`);
});
};
On finish is never triggered. No output whatsoever.
I can stream http.get to a local file without problems, so that part seems to be OK.
What also works is streaming from the local filesystem to GCS, like below :
const streamFs = () => {
fs.createReadStream('/path/to/mqdefault.jpg')
.pipe(blob.createWriteStream({
metadata: {
contentType: 'image/jpg'
}
}))
.on("error", (err) => {
console.error(`error occurred`);
})
.on('finish', () => {
console.info(`success`);
});
};
The second snippet logs 'success' and the file is present on the bucket.
Both http.get and fs.createReadStream create a read stream.
What am I doing wrong here ?
node.js google-cloud-platform google-cloud-storage
asked Jan 2 at 23:01
komakoma
5,75021841
add a comment |
I am fighting to get a download to directly stream to GCS without saving to the file system, see below snippet.
const { Storage } = require('@google-cloud/storage');
const http = require('http');
const storage = new Storage();
const fs = require("fs");
const bucketName = 'BUCKETNAMEHERE';
const blobName = 'image.jpg';
const bucket = storage.bucket(bucketName);
const blob = bucket.file(blobName);
const streamDownload = () => {
http.get("http://i3.ytimg.com/vi/J---aiyznGQ/mqdefault.jpg")
.pipe(blob.createWriteStream({
metadata: {
contentType: 'image/jpg'
}
}))
.on("error", (err) => {
console.error(`error occurred`);
})
.on('finish', () => {
console.info(`success`);
});
};
On finish is never triggered. No output whatsoever.
I can stream http.get to a local file without problems, so that part seems to be OK.
What also works is streaming from the local filesystem to GCS, like below :
const streamFs = () => {
fs.createReadStream('/path/to/mqdefault.jpg')
.pipe(blob.createWriteStream({
metadata: {
contentType: 'image/jpg'
}
}))
.on("error", (err) => {
console.error(`error occurred`);
})
.on('finish', () => {
console.info(`success`);
});
};
The second snippet logs 'success' and the file is present on the bucket.
Both http.get and fs.createReadStream create a read stream.
What am I doing wrong here ?
node.js google-cloud-platform google-cloud-storage
asked Jan 2 at 23:01
komakoma
5,75021841
add a comment |
I am fighting to get a download to directly stream to GCS without saving to the file system, see below snippet.
const { Storage } = require('@google-cloud/storage');
const http = require('http');
const storage = new Storage();
const fs = require("fs");
const bucketName = 'BUCKETNAMEHERE';
const blobName = 'image.jpg';
const bucket = storage.bucket(bucketName);
const blob = bucket.file(blobName);
const streamDownload = () => {
http.get("http://i3.ytimg.com/vi/J---aiyznGQ/mqdefault.jpg")
.pipe(blob.createWriteStream({
metadata: {
contentType: 'image/jpg'
}
}))
.on("error", (err) => {
console.error(`error occurred`);
})
.on('finish', () => {
console.info(`success`);
});
};
On finish is never triggered. No output whatsoever.
I can stream http.get to a local file without problems, so that part seems to be OK.
What also works is streaming from the local filesystem to GCS, like below :
const streamFs = () => {
fs.createReadStream('/path/to/mqdefault.jpg')
.pipe(blob.createWriteStream({
metadata: {
contentType: 'image/jpg'
}
}))
.on("error", (err) => {
console.error(`error occurred`);
})
.on('finish', () => {
console.info(`success`);
});
};
The second snippet logs 'success' and the file is present on the bucket.
Both http.get and fs.createReadStream create a read stream.
What am I doing wrong here ?
node.js google-cloud-platform google-cloud-storage
asked Jan 2 at 23:01
komakoma
5,75021841
I am fighting to get a download to directly stream to GCS without saving to the file system, see below snippet.
const { Storage } = require('@google-cloud/storage');
const http = require('http');
const storage = new Storage();
const fs = require("fs");
const bucketName = 'BUCKETNAMEHERE';
const blobName = 'image.jpg';
const bucket = storage.bucket(bucketName);
const blob = bucket.file(blobName);
const streamDownload = () => {
http.get("http://i3.ytimg.com/vi/J---aiyznGQ/mqdefault.jpg")
.pipe(blob.createWriteStream({
metadata: {
contentType: 'image/jpg'
}
}))
.on("error", (err) => {
console.error(`error occurred`);
})
.on('finish', () => {
console.info(`success`);
});
};
On finish is never triggered. No output whatsoever.
I can stream http.get to a local file without problems, so that part seems to be OK.
What also works is streaming from the local filesystem to GCS, like below :
const streamFs = () => {
fs.createReadStream('/path/to/mqdefault.jpg')
.pipe(blob.createWriteStream({
metadata: {
contentType: 'image/jpg'
}
}))
.on("error", (err) => {
console.error(`error occurred`);
})
.on('finish', () => {
console.info(`success`);
});
};
The second snippet logs 'success' and the file is present on the bucket.
Both http.get and fs.createReadStream create a read stream.
What am I doing wrong here ?
node.js google-cloud-platform google-cloud-storage
node.js google-cloud-platform google-cloud-storage
asked Jan 2 at 23:01
komakoma
5,75021841
asked Jan 2 at 23:01
komakoma
5,75021841
edited Jan 2 at 23:34
koma
asked Jan 2 at 23:01
komakoma
5,75021841
asked Jan 2 at 23:01
komakoma
5,75021841
asked Jan 2 at 23:01
komakoma
5,75021841
5,75021841
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
the request library works with promises whereas the http lib needs a callback :
const streamDownload = () => {
var stream = blob.createWriteStream({
metadata: {
contentType: 'image/jpg'
}
})
http.get("http://i3.ytimg.com/vi/J---aiyznGQ/mqdefault.jpg", function(res){
stream.on("error", (err) => {
console.error(`error occurred`);
})
stream.on('finish', () => {
console.info(`success`);
});
res.pipe(stream)
})
}
edited Jan 3 at 9:54
koma
5,75021841
answered Jan 3 at 7:55
ThomasThomas
443310
add a comment |
Switching over to the request library does work :
const request = require('request');
const streamDownload = () => {
request.get("http://i3.ytimg.com/vi/J---aiyznGQ/mqdefault.jpg")
.pipe(blob.createWriteStream({
metadata: {
contentType: 'image/jpg'
}
}))
.on("error", (err) => {
console.error(`error occurred`);
})
.on('finish', () => {
console.info(`success`);
});
};
Still unsure why http library does not.
answered Jan 2 at 23:47
komakoma
5,75021841
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54014305%2fdownload-via-http-and-streaming-upload-to-google-cloud-storage-using-nodejs%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
the request library works with promises whereas the http lib needs a callback :
const streamDownload = () => {
var stream = blob.createWriteStream({
metadata: {
contentType: 'image/jpg'
}
})
http.get("http://i3.ytimg.com/vi/J---aiyznGQ/mqdefault.jpg", function(res){
stream.on("error", (err) => {
console.error(`error occurred`);
})
stream.on('finish', () => {
console.info(`success`);
});
res.pipe(stream)
})
}
edited Jan 3 at 9:54
koma
5,75021841
answered Jan 3 at 7:55
ThomasThomas
443310
add a comment |
the request library works with promises whereas the http lib needs a callback :
const streamDownload = () => {
var stream = blob.createWriteStream({
metadata: {
contentType: 'image/jpg'
}
})
http.get("http://i3.ytimg.com/vi/J---aiyznGQ/mqdefault.jpg", function(res){
stream.on("error", (err) => {
console.error(`error occurred`);
})
stream.on('finish', () => {
console.info(`success`);
});
res.pipe(stream)
})
}
edited Jan 3 at 9:54
koma
5,75021841
answered Jan 3 at 7:55
ThomasThomas
443310
add a comment |
the request library works with promises whereas the http lib needs a callback :
const streamDownload = () => {
var stream = blob.createWriteStream({
metadata: {
contentType: 'image/jpg'
}
})
http.get("http://i3.ytimg.com/vi/J---aiyznGQ/mqdefault.jpg", function(res){
stream.on("error", (err) => {
console.error(`error occurred`);
})
stream.on('finish', () => {
console.info(`success`);
});
res.pipe(stream)
})
}
edited Jan 3 at 9:54
koma
5,75021841
answered Jan 3 at 7:55
ThomasThomas
443310
the request library works with promises whereas the http lib needs a callback :
const streamDownload = () => {
var stream = blob.createWriteStream({
metadata: {
contentType: 'image/jpg'
}
})
http.get("http://i3.ytimg.com/vi/J---aiyznGQ/mqdefault.jpg", function(res){
stream.on("error", (err) => {
console.error(`error occurred`);
})
stream.on('finish', () => {
console.info(`success`);
});
res.pipe(stream)
})
}
edited Jan 3 at 9:54
koma
5,75021841
answered Jan 3 at 7:55
ThomasThomas
443310
edited Jan 3 at 9:54
koma
5,75021841
edited Jan 3 at 9:54
koma
5,75021841
edited Jan 3 at 9:54
koma
5,75021841
5,75021841
answered Jan 3 at 7:55
ThomasThomas
443310
answered Jan 3 at 7:55
ThomasThomas
443310
answered Jan 3 at 7:55
ThomasThomas
443310
443310
add a comment |
add a comment |
Switching over to the request library does work :
const request = require('request');
const streamDownload = () => {
request.get("http://i3.ytimg.com/vi/J---aiyznGQ/mqdefault.jpg")
.pipe(blob.createWriteStream({
metadata: {
contentType: 'image/jpg'
}
}))
.on("error", (err) => {
console.error(`error occurred`);
})
.on('finish', () => {
console.info(`success`);
});
};
Still unsure why http library does not.
answered Jan 2 at 23:47
komakoma
5,75021841
add a comment |
Switching over to the request library does work :
const request = require('request');
const streamDownload = () => {
request.get("http://i3.ytimg.com/vi/J---aiyznGQ/mqdefault.jpg")
.pipe(blob.createWriteStream({
metadata: {
contentType: 'image/jpg'
}
}))
.on("error", (err) => {
console.error(`error occurred`);
})
.on('finish', () => {
console.info(`success`);
});
};
Still unsure why http library does not.
answered Jan 2 at 23:47
komakoma
5,75021841
add a comment |
Switching over to the request library does work :
const request = require('request');
const streamDownload = () => {
request.get("http://i3.ytimg.com/vi/J---aiyznGQ/mqdefault.jpg")
.pipe(blob.createWriteStream({
metadata: {
contentType: 'image/jpg'
}
}))
.on("error", (err) => {
console.error(`error occurred`);
})
.on('finish', () => {
console.info(`success`);
});
};
Still unsure why http library does not.
answered Jan 2 at 23:47
komakoma
5,75021841
Switching over to the request library does work :
const request = require('request');
const streamDownload = () => {
request.get("http://i3.ytimg.com/vi/J---aiyznGQ/mqdefault.jpg")
.pipe(blob.createWriteStream({
metadata: {
contentType: 'image/jpg'
}
}))
.on("error", (err) => {
console.error(`error occurred`);
})
.on('finish', () => {
console.info(`success`);
});
};
Still unsure why http library does not.
answered Jan 2 at 23:47
komakoma
5,75021841
answered Jan 2 at 23:47
komakoma
5,75021841
answered Jan 2 at 23:47
komakoma
5,75021841
answered Jan 2 at 23:47
komakoma
5,75021841
5,75021841
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54014305%2fdownload-via-http-and-streaming-upload-to-google-cloud-storage-using-nodejs%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
