Requesting size of Stream from consumer, before loading all the data
Hy all,
I have a problem with the camel component I am developing, where I'm not sure how to implement it in a way, that goes in line with the concepts of camel.
The producer I'm developing talks to the http api for our server, which is used to send messages with attachments.
Those attachments can potentially be very big, which is why the server expects the total filesize before any upload is done.
Currently the producer only accepts io.Files, nio.Paths and GenericFile, because there I can read the file size, before I upload the file.
Of course this is not a good way to do things, because it requires the (big) file to be available locally.
Connecting, for a example, a ftp server as the consumer would mean, that I have to download each file locally so I can upload it afterwards.
The obvious solution is using streams to access and upload the data, but with this I do not know how big the file is, before I'm done uploading, which is not an option, I need the size in advance.
My question now is, what are best practices to stream files through camel and also make the consumer give me the filesize in advance.
Greets
Chris
apache-camel
asked Nov 21 '18 at 9:30
Chris
384
add a comment |
Hy all,
I have a problem with the camel component I am developing, where I'm not sure how to implement it in a way, that goes in line with the concepts of camel.
The producer I'm developing talks to the http api for our server, which is used to send messages with attachments.
Those attachments can potentially be very big, which is why the server expects the total filesize before any upload is done.
Currently the producer only accepts io.Files, nio.Paths and GenericFile, because there I can read the file size, before I upload the file.
Of course this is not a good way to do things, because it requires the (big) file to be available locally.
Connecting, for a example, a ftp server as the consumer would mean, that I have to download each file locally so I can upload it afterwards.
The obvious solution is using streams to access and upload the data, but with this I do not know how big the file is, before I'm done uploading, which is not an option, I need the size in advance.
My question now is, what are best practices to stream files through camel and also make the consumer give me the filesize in advance.
Greets
Chris
apache-camel
asked Nov 21 '18 at 9:30
Chris
384
add a comment |
Hy all,
I have a problem with the camel component I am developing, where I'm not sure how to implement it in a way, that goes in line with the concepts of camel.
The producer I'm developing talks to the http api for our server, which is used to send messages with attachments.
Those attachments can potentially be very big, which is why the server expects the total filesize before any upload is done.
Currently the producer only accepts io.Files, nio.Paths and GenericFile, because there I can read the file size, before I upload the file.
Of course this is not a good way to do things, because it requires the (big) file to be available locally.
Connecting, for a example, a ftp server as the consumer would mean, that I have to download each file locally so I can upload it afterwards.
The obvious solution is using streams to access and upload the data, but with this I do not know how big the file is, before I'm done uploading, which is not an option, I need the size in advance.
My question now is, what are best practices to stream files through camel and also make the consumer give me the filesize in advance.
Greets
Chris
apache-camel
asked Nov 21 '18 at 9:30
Chris
384
Hy all,
I have a problem with the camel component I am developing, where I'm not sure how to implement it in a way, that goes in line with the concepts of camel.
The producer I'm developing talks to the http api for our server, which is used to send messages with attachments.
Those attachments can potentially be very big, which is why the server expects the total filesize before any upload is done.
Currently the producer only accepts io.Files, nio.Paths and GenericFile, because there I can read the file size, before I upload the file.
Of course this is not a good way to do things, because it requires the (big) file to be available locally.
Connecting, for a example, a ftp server as the consumer would mean, that I have to download each file locally so I can upload it afterwards.
The obvious solution is using streams to access and upload the data, but with this I do not know how big the file is, before I'm done uploading, which is not an option, I need the size in advance.
My question now is, what are best practices to stream files through camel and also make the consumer give me the filesize in advance.
Greets
Chris
apache-camel
apache-camel
asked Nov 21 '18 at 9:30
Chris
384
asked Nov 21 '18 at 9:30
Chris
384
asked Nov 21 '18 at 9:30
Chris
384
asked Nov 21 '18 at 9:30
Chris
384
asked Nov 21 '18 at 9:30
Chris
384
384
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
For File/FTP consumer, the exchange in header has a key CamelFileLength (Exchange.FILE_LENGTH) which return the file size in remote ftp server from consumer's scan result.
Unlike the file size obtain from local, the file size in key CamelFileLength might differ from actual file size your application received
- The ASCII mode will potentially change the linefeed when there is OS differ
- The file size might change between consumer scan action and consumer pick actionn
answered Nov 22 '18 at 2:48
hk6279
1,01111024
So if I understand you correctly, this would result in two issues: 1. I can not rely on the CamelFileSize to be there at all 2. The file size in my application (which is the only one relevant to me) is basically wrong If so, this unfortunatelly does not help me with my problem =(
– Chris
Nov 26 '18 at 7:48
1
@Chris The CamelFileSize is reliable if you accept delay receive like using readlock or receive single file (exist in folder at least minutes) in each batch.
– hk6279
Nov 27 '18 at 2:24
Well as long as the CamelFileSize is the correct value, when I have actually read the whole file, this is fine. But all this is only the case for the file component, isn't it? I can't rely on this header, when I receive a file from a different component, or am I wrong here?
– Chris
Nov 28 '18 at 10:56
@Chris The answer's point 2 also apply to file component. As state in above comment, the headerCamelFileSizeis reliable under restricted condition.
– hk6279
Nov 29 '18 at 4:01
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53408933%2frequesting-size-of-stream-from-consumer-before-loading-all-the-data%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
For File/FTP consumer, the exchange in header has a key CamelFileLength (Exchange.FILE_LENGTH) which return the file size in remote ftp server from consumer's scan result.
Unlike the file size obtain from local, the file size in key CamelFileLength might differ from actual file size your application received
- The ASCII mode will potentially change the linefeed when there is OS differ
- The file size might change between consumer scan action and consumer pick actionn
answered Nov 22 '18 at 2:48
hk6279
1,01111024
So if I understand you correctly, this would result in two issues: 1. I can not rely on the CamelFileSize to be there at all 2. The file size in my application (which is the only one relevant to me) is basically wrong If so, this unfortunatelly does not help me with my problem =(
– Chris
Nov 26 '18 at 7:48
1
@Chris The CamelFileSize is reliable if you accept delay receive like using readlock or receive single file (exist in folder at least minutes) in each batch.
– hk6279
Nov 27 '18 at 2:24
Well as long as the CamelFileSize is the correct value, when I have actually read the whole file, this is fine. But all this is only the case for the file component, isn't it? I can't rely on this header, when I receive a file from a different component, or am I wrong here?
– Chris
Nov 28 '18 at 10:56
@Chris The answer's point 2 also apply to file component. As state in above comment, the headerCamelFileSizeis reliable under restricted condition.
– hk6279
Nov 29 '18 at 4:01
add a comment |
For File/FTP consumer, the exchange in header has a key CamelFileLength (Exchange.FILE_LENGTH) which return the file size in remote ftp server from consumer's scan result.
Unlike the file size obtain from local, the file size in key CamelFileLength might differ from actual file size your application received
- The ASCII mode will potentially change the linefeed when there is OS differ
- The file size might change between consumer scan action and consumer pick actionn
answered Nov 22 '18 at 2:48
hk6279
1,01111024
So if I understand you correctly, this would result in two issues: 1. I can not rely on the CamelFileSize to be there at all 2. The file size in my application (which is the only one relevant to me) is basically wrong If so, this unfortunatelly does not help me with my problem =(
– Chris
Nov 26 '18 at 7:48
1
@Chris The CamelFileSize is reliable if you accept delay receive like using readlock or receive single file (exist in folder at least minutes) in each batch.
– hk6279
Nov 27 '18 at 2:24
Well as long as the CamelFileSize is the correct value, when I have actually read the whole file, this is fine. But all this is only the case for the file component, isn't it? I can't rely on this header, when I receive a file from a different component, or am I wrong here?
– Chris
Nov 28 '18 at 10:56
@Chris The answer's point 2 also apply to file component. As state in above comment, the headerCamelFileSizeis reliable under restricted condition.
– hk6279
Nov 29 '18 at 4:01
add a comment |
For File/FTP consumer, the exchange in header has a key CamelFileLength (Exchange.FILE_LENGTH) which return the file size in remote ftp server from consumer's scan result.
Unlike the file size obtain from local, the file size in key CamelFileLength might differ from actual file size your application received
- The ASCII mode will potentially change the linefeed when there is OS differ
- The file size might change between consumer scan action and consumer pick actionn
answered Nov 22 '18 at 2:48
hk6279
1,01111024
For File/FTP consumer, the exchange in header has a key CamelFileLength (Exchange.FILE_LENGTH) which return the file size in remote ftp server from consumer's scan result.
Unlike the file size obtain from local, the file size in key CamelFileLength might differ from actual file size your application received
- The ASCII mode will potentially change the linefeed when there is OS differ
- The file size might change between consumer scan action and consumer pick actionn
answered Nov 22 '18 at 2:48
hk6279
1,01111024
answered Nov 22 '18 at 2:48
hk6279
1,01111024
answered Nov 22 '18 at 2:48
hk6279
1,01111024
answered Nov 22 '18 at 2:48
hk6279
1,01111024
1,01111024
So if I understand you correctly, this would result in two issues: 1. I can not rely on the CamelFileSize to be there at all 2. The file size in my application (which is the only one relevant to me) is basically wrong If so, this unfortunatelly does not help me with my problem =(
– Chris
Nov 26 '18 at 7:48
1
@Chris The CamelFileSize is reliable if you accept delay receive like using readlock or receive single file (exist in folder at least minutes) in each batch.
– hk6279
Nov 27 '18 at 2:24
Well as long as the CamelFileSize is the correct value, when I have actually read the whole file, this is fine. But all this is only the case for the file component, isn't it? I can't rely on this header, when I receive a file from a different component, or am I wrong here?
– Chris
Nov 28 '18 at 10:56
@Chris The answer's point 2 also apply to file component. As state in above comment, the headerCamelFileSizeis reliable under restricted condition.
– hk6279
Nov 29 '18 at 4:01
add a comment |
So if I understand you correctly, this would result in two issues: 1. I can not rely on the CamelFileSize to be there at all 2. The file size in my application (which is the only one relevant to me) is basically wrong If so, this unfortunatelly does not help me with my problem =(
– Chris
Nov 26 '18 at 7:48
1
@Chris The CamelFileSize is reliable if you accept delay receive like using readlock or receive single file (exist in folder at least minutes) in each batch.
– hk6279
Nov 27 '18 at 2:24
Well as long as the CamelFileSize is the correct value, when I have actually read the whole file, this is fine. But all this is only the case for the file component, isn't it? I can't rely on this header, when I receive a file from a different component, or am I wrong here?
– Chris
Nov 28 '18 at 10:56
@Chris The answer's point 2 also apply to file component. As state in above comment, the headerCamelFileSizeis reliable under restricted condition.
– hk6279
Nov 29 '18 at 4:01
So if I understand you correctly, this would result in two issues: 1. I can not rely on the CamelFileSize to be there at all 2. The file size in my application (which is the only one relevant to me) is basically wrong If so, this unfortunatelly does not help me with my problem =(
– Chris
Nov 26 '18 at 7:48
So if I understand you correctly, this would result in two issues: 1. I can not rely on the CamelFileSize to be there at all 2. The file size in my application (which is the only one relevant to me) is basically wrong If so, this unfortunatelly does not help me with my problem =(
– Chris
Nov 26 '18 at 7:48
1
1
@Chris The CamelFileSize is reliable if you accept delay receive like using readlock or receive single file (exist in folder at least minutes) in each batch.
– hk6279
Nov 27 '18 at 2:24
@Chris The CamelFileSize is reliable if you accept delay receive like using readlock or receive single file (exist in folder at least minutes) in each batch.
– hk6279
Nov 27 '18 at 2:24
Well as long as the CamelFileSize is the correct value, when I have actually read the whole file, this is fine. But all this is only the case for the file component, isn't it? I can't rely on this header, when I receive a file from a different component, or am I wrong here?
– Chris
Nov 28 '18 at 10:56
Well as long as the CamelFileSize is the correct value, when I have actually read the whole file, this is fine. But all this is only the case for the file component, isn't it? I can't rely on this header, when I receive a file from a different component, or am I wrong here?
– Chris
Nov 28 '18 at 10:56
@Chris The answer's point 2 also apply to file component. As state in above comment, the header
CamelFileSize is reliable under restricted condition.– hk6279
Nov 29 '18 at 4:01
@Chris The answer's point 2 also apply to file component. As state in above comment, the header
CamelFileSize is reliable under restricted condition.– hk6279
Nov 29 '18 at 4:01
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53408933%2frequesting-size-of-stream-from-consumer-before-loading-all-the-data%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
