BigQuery retrieve data from sftp
I have an internet facing sftp server that has regularly csv files update. Is there anyway command to have BigQuery to retrieve data from this sftp and put it into tables. Alternatively, any API or python library that support this?
google-bigquery sftp asked Nov 20 at 20:03
Kai Zhang
83
add a comment |
I have an internet facing sftp server that has regularly csv files update. Is there anyway command to have BigQuery to retrieve data from this sftp and put it into tables. Alternatively, any API or python library that support this?
google-bigquery sftp asked Nov 20 at 20:03
Kai Zhang
83
add a comment |
I have an internet facing sftp server that has regularly csv files update. Is there anyway command to have BigQuery to retrieve data from this sftp and put it into tables. Alternatively, any API or python library that support this?
google-bigquery sftp asked Nov 20 at 20:03
Kai Zhang
83
I have an internet facing sftp server that has regularly csv files update. Is there anyway command to have BigQuery to retrieve data from this sftp and put it into tables. Alternatively, any API or python library that support this?
google-bigquery sftp google-bigquery sftp asked Nov 20 at 20:03
Kai Zhang
83
asked Nov 20 at 20:03
Kai Zhang
83
asked Nov 20 at 20:03
Kai Zhang
83
asked Nov 20 at 20:03
Kai Zhang
83
asked Nov 20 at 20:03
Kai Zhang
83
83
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
As for BigQuery - there's no integration I know of with SFTP.
You'll need to either:
- Create/find a script that reads from SFTP and pushes to GCS.
- Add a HTTPS service to the SFTP server, so your data can be read with the GCS transfer service (https://cloud.google.com/storage-transfer/docs/)
answered Nov 20 at 20:59
Felipe Hoffa
20.8k249106
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53400716%2fbigquery-retrieve-data-from-sftp%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
As for BigQuery - there's no integration I know of with SFTP.
You'll need to either:
- Create/find a script that reads from SFTP and pushes to GCS.
- Add a HTTPS service to the SFTP server, so your data can be read with the GCS transfer service (https://cloud.google.com/storage-transfer/docs/)
answered Nov 20 at 20:59
Felipe Hoffa
20.8k249106
add a comment |
As for BigQuery - there's no integration I know of with SFTP.
You'll need to either:
- Create/find a script that reads from SFTP and pushes to GCS.
- Add a HTTPS service to the SFTP server, so your data can be read with the GCS transfer service (https://cloud.google.com/storage-transfer/docs/)
answered Nov 20 at 20:59
Felipe Hoffa
20.8k249106
add a comment |
As for BigQuery - there's no integration I know of with SFTP.
You'll need to either:
- Create/find a script that reads from SFTP and pushes to GCS.
- Add a HTTPS service to the SFTP server, so your data can be read with the GCS transfer service (https://cloud.google.com/storage-transfer/docs/)
answered Nov 20 at 20:59
Felipe Hoffa
20.8k249106
As for BigQuery - there's no integration I know of with SFTP.
You'll need to either:
- Create/find a script that reads from SFTP and pushes to GCS.
- Add a HTTPS service to the SFTP server, so your data can be read with the GCS transfer service (https://cloud.google.com/storage-transfer/docs/)
answered Nov 20 at 20:59
Felipe Hoffa
20.8k249106
answered Nov 20 at 20:59
Felipe Hoffa
20.8k249106
answered Nov 20 at 20:59
Felipe Hoffa
20.8k249106
answered Nov 20 at 20:59
Felipe Hoffa
20.8k249106
20.8k249106
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53400716%2fbigquery-retrieve-data-from-sftp%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
