tensorflow: can reshape() create a copy?
In v4 of their API, torch has introduced reshape(), to align more with the style of numpy. Previously, changing the shape of a torch tensor was done with view().
I wondered whether view() was going to be deprecated now and looked at the docs. Turns out that reshape() is not just a numpy-friendly alias for view(), actually, it has a different semantic. Torch tries to give you contiguous memory where possible. If the new view dimensions violate a contiguity constraint, you have to explicitly call contiguous() before view(). Reshape will work even if this constraint is violated, but will silently make a copy of the data.
This is the same behaviour as in numpy, where reshape can produce copies, too.
A question on view() vs reshape() in torch is here: What's the difference between reshape and view in pytorch?
if you need a copy use clone() if you need the same storage use
view(). The semantics of reshape() are that it may or may not share
the storage and you don't know beforehand.
Up until now, torch only offered view(). Maybe to intentionally force their developers to care about memory layout. Which makes me wonder how reshape() works in tensorflow.
In torch, the distinction between a view and a copy could produce complicated bugs. You assume that tensors share data, but they don't.
In tensorflow this problem shouldn't exist. A tensorflow tensor is symbolic and doesn't hold a value. A reshape is just an Op in the tensorflow graph. While the graph is evaluated, data in placeholders and variables don't change, so it is clear what data you are working on.
But I don't know if this could hurt performance. Copying a huge tensor can be very expensive. Do I have to be careful when using reshape(), not to duplicate memory ?
python numpy tensorflow reshape torch
asked Nov 20 at 17:49
lhk
6,46585588
|
show 2 more comments
In v4 of their API, torch has introduced reshape(), to align more with the style of numpy. Previously, changing the shape of a torch tensor was done with view().
I wondered whether view() was going to be deprecated now and looked at the docs. Turns out that reshape() is not just a numpy-friendly alias for view(), actually, it has a different semantic. Torch tries to give you contiguous memory where possible. If the new view dimensions violate a contiguity constraint, you have to explicitly call contiguous() before view(). Reshape will work even if this constraint is violated, but will silently make a copy of the data.
This is the same behaviour as in numpy, where reshape can produce copies, too.
A question on view() vs reshape() in torch is here: What's the difference between reshape and view in pytorch?
if you need a copy use clone() if you need the same storage use
view(). The semantics of reshape() are that it may or may not share
the storage and you don't know beforehand.
Up until now, torch only offered view(). Maybe to intentionally force their developers to care about memory layout. Which makes me wonder how reshape() works in tensorflow.
In torch, the distinction between a view and a copy could produce complicated bugs. You assume that tensors share data, but they don't.
In tensorflow this problem shouldn't exist. A tensorflow tensor is symbolic and doesn't hold a value. A reshape is just an Op in the tensorflow graph. While the graph is evaluated, data in placeholders and variables don't change, so it is clear what data you are working on.
But I don't know if this could hurt performance. Copying a huge tensor can be very expensive. Do I have to be careful when using reshape(), not to duplicate memory ?
python numpy tensorflow reshape torch
asked Nov 20 at 17:49
lhk
6,46585588
Tensors in Tensorflow are always contiguous.reshape-ing contiguous array always result in contiguous array. Therefore,reshapein Tensorflow never have to create copy (not sure if it actually does it or not). On the contrary, operations like transpose always makes copy.
– ZisIsNotZis
Nov 21 at 4:25
But since Tensorflow is a "computation graph compiler", it do have some automatic optimization in the graph that might optimize out the copying. I did some google but did't find detail of that.
– ZisIsNotZis
Nov 21 at 4:31
@ZisIsNotZis "Tensors in tensorflow are always contiguous", I don't understand this. If you slice a tensor with a stride [::2], the result will not be contiguous. And such a slice is definitely possible in tensorflow. Does tensorflow implicitly call contiguous() after every op that breaks this invariant? That would not be efficient. I guess this only applies when running a session and evaluating a tensor. But still, how do you keep it contiguous?
– lhk
Nov 21 at 8:27
Actually I read about it [here][tensorflow.org/api_docs/python/tf/transpose ]. At the bottom of the page, it sais Tensorflow do not support strides. Therefore, it have to be contiguous. If you look at [here][stackoverflow.com/questions/50779869/… ], the answer also sais the tensorflow always make copy unless dim-0-aligned.
– ZisIsNotZis
Nov 21 at 8:35
But this shouldn't be a that big problem since if you are using Tensorflow, you are usually doing much slower jobs like matrix multiplication and etc. In this sense, copying is the fastest operation you can do with some data, which takes relatively negligible time.
– ZisIsNotZis
Nov 21 at 8:46
|
show 2 more comments
In v4 of their API, torch has introduced reshape(), to align more with the style of numpy. Previously, changing the shape of a torch tensor was done with view().
I wondered whether view() was going to be deprecated now and looked at the docs. Turns out that reshape() is not just a numpy-friendly alias for view(), actually, it has a different semantic. Torch tries to give you contiguous memory where possible. If the new view dimensions violate a contiguity constraint, you have to explicitly call contiguous() before view(). Reshape will work even if this constraint is violated, but will silently make a copy of the data.
This is the same behaviour as in numpy, where reshape can produce copies, too.
A question on view() vs reshape() in torch is here: What's the difference between reshape and view in pytorch?
if you need a copy use clone() if you need the same storage use
view(). The semantics of reshape() are that it may or may not share
the storage and you don't know beforehand.
Up until now, torch only offered view(). Maybe to intentionally force their developers to care about memory layout. Which makes me wonder how reshape() works in tensorflow.
In torch, the distinction between a view and a copy could produce complicated bugs. You assume that tensors share data, but they don't.
In tensorflow this problem shouldn't exist. A tensorflow tensor is symbolic and doesn't hold a value. A reshape is just an Op in the tensorflow graph. While the graph is evaluated, data in placeholders and variables don't change, so it is clear what data you are working on.
But I don't know if this could hurt performance. Copying a huge tensor can be very expensive. Do I have to be careful when using reshape(), not to duplicate memory ?
python numpy tensorflow reshape torch
asked Nov 20 at 17:49
lhk
6,46585588
In v4 of their API, torch has introduced reshape(), to align more with the style of numpy. Previously, changing the shape of a torch tensor was done with view().
I wondered whether view() was going to be deprecated now and looked at the docs. Turns out that reshape() is not just a numpy-friendly alias for view(), actually, it has a different semantic. Torch tries to give you contiguous memory where possible. If the new view dimensions violate a contiguity constraint, you have to explicitly call contiguous() before view(). Reshape will work even if this constraint is violated, but will silently make a copy of the data.
This is the same behaviour as in numpy, where reshape can produce copies, too.
A question on view() vs reshape() in torch is here: What's the difference between reshape and view in pytorch?
if you need a copy use clone() if you need the same storage use
view(). The semantics of reshape() are that it may or may not share
the storage and you don't know beforehand.
Up until now, torch only offered view(). Maybe to intentionally force their developers to care about memory layout. Which makes me wonder how reshape() works in tensorflow.
In torch, the distinction between a view and a copy could produce complicated bugs. You assume that tensors share data, but they don't.
In tensorflow this problem shouldn't exist. A tensorflow tensor is symbolic and doesn't hold a value. A reshape is just an Op in the tensorflow graph. While the graph is evaluated, data in placeholders and variables don't change, so it is clear what data you are working on.
But I don't know if this could hurt performance. Copying a huge tensor can be very expensive. Do I have to be careful when using reshape(), not to duplicate memory ?
python numpy tensorflow reshape torch
python numpy tensorflow reshape torch
asked Nov 20 at 17:49
lhk
6,46585588
asked Nov 20 at 17:49
lhk
6,46585588
asked Nov 20 at 17:49
lhk
6,46585588
asked Nov 20 at 17:49
lhk
6,46585588
asked Nov 20 at 17:49
lhk
6,46585588
6,46585588
Tensors in Tensorflow are always contiguous.reshape-ing contiguous array always result in contiguous array. Therefore,reshapein Tensorflow never have to create copy (not sure if it actually does it or not). On the contrary, operations like transpose always makes copy.
– ZisIsNotZis
Nov 21 at 4:25
But since Tensorflow is a "computation graph compiler", it do have some automatic optimization in the graph that might optimize out the copying. I did some google but did't find detail of that.
– ZisIsNotZis
Nov 21 at 4:31
@ZisIsNotZis "Tensors in tensorflow are always contiguous", I don't understand this. If you slice a tensor with a stride [::2], the result will not be contiguous. And such a slice is definitely possible in tensorflow. Does tensorflow implicitly call contiguous() after every op that breaks this invariant? That would not be efficient. I guess this only applies when running a session and evaluating a tensor. But still, how do you keep it contiguous?
– lhk
Nov 21 at 8:27
Actually I read about it [here][tensorflow.org/api_docs/python/tf/transpose ]. At the bottom of the page, it sais Tensorflow do not support strides. Therefore, it have to be contiguous. If you look at [here][stackoverflow.com/questions/50779869/… ], the answer also sais the tensorflow always make copy unless dim-0-aligned.
– ZisIsNotZis
Nov 21 at 8:35
But this shouldn't be a that big problem since if you are using Tensorflow, you are usually doing much slower jobs like matrix multiplication and etc. In this sense, copying is the fastest operation you can do with some data, which takes relatively negligible time.
– ZisIsNotZis
Nov 21 at 8:46
|
show 2 more comments
Tensors in Tensorflow are always contiguous.reshape-ing contiguous array always result in contiguous array. Therefore,reshapein Tensorflow never have to create copy (not sure if it actually does it or not). On the contrary, operations like transpose always makes copy.
– ZisIsNotZis
Nov 21 at 4:25
But since Tensorflow is a "computation graph compiler", it do have some automatic optimization in the graph that might optimize out the copying. I did some google but did't find detail of that.
– ZisIsNotZis
Nov 21 at 4:31
@ZisIsNotZis "Tensors in tensorflow are always contiguous", I don't understand this. If you slice a tensor with a stride [::2], the result will not be contiguous. And such a slice is definitely possible in tensorflow. Does tensorflow implicitly call contiguous() after every op that breaks this invariant? That would not be efficient. I guess this only applies when running a session and evaluating a tensor. But still, how do you keep it contiguous?
– lhk
Nov 21 at 8:27
Actually I read about it [here][tensorflow.org/api_docs/python/tf/transpose ]. At the bottom of the page, it sais Tensorflow do not support strides. Therefore, it have to be contiguous. If you look at [here][stackoverflow.com/questions/50779869/… ], the answer also sais the tensorflow always make copy unless dim-0-aligned.
– ZisIsNotZis
Nov 21 at 8:35
But this shouldn't be a that big problem since if you are using Tensorflow, you are usually doing much slower jobs like matrix multiplication and etc. In this sense, copying is the fastest operation you can do with some data, which takes relatively negligible time.
– ZisIsNotZis
Nov 21 at 8:46
Tensors in Tensorflow are always contiguous. reshape-ing contiguous array always result in contiguous array. Therefore, reshape in Tensorflow never have to create copy (not sure if it actually does it or not). On the contrary, operations like transpose always makes copy.– ZisIsNotZis
Nov 21 at 4:25
Tensors in Tensorflow are always contiguous. reshape-ing contiguous array always result in contiguous array. Therefore, reshape in Tensorflow never have to create copy (not sure if it actually does it or not). On the contrary, operations like transpose always makes copy.– ZisIsNotZis
Nov 21 at 4:25
But since Tensorflow is a "computation graph compiler", it do have some automatic optimization in the graph that might optimize out the copying. I did some google but did't find detail of that.
– ZisIsNotZis
Nov 21 at 4:31
But since Tensorflow is a "computation graph compiler", it do have some automatic optimization in the graph that might optimize out the copying. I did some google but did't find detail of that.
– ZisIsNotZis
Nov 21 at 4:31
@ZisIsNotZis "Tensors in tensorflow are always contiguous", I don't understand this. If you slice a tensor with a stride [::2], the result will not be contiguous. And such a slice is definitely possible in tensorflow. Does tensorflow implicitly call contiguous() after every op that breaks this invariant? That would not be efficient. I guess this only applies when running a session and evaluating a tensor. But still, how do you keep it contiguous?
– lhk
Nov 21 at 8:27
@ZisIsNotZis "Tensors in tensorflow are always contiguous", I don't understand this. If you slice a tensor with a stride [::2], the result will not be contiguous. And such a slice is definitely possible in tensorflow. Does tensorflow implicitly call contiguous() after every op that breaks this invariant? That would not be efficient. I guess this only applies when running a session and evaluating a tensor. But still, how do you keep it contiguous?
– lhk
Nov 21 at 8:27
Actually I read about it [here][tensorflow.org/api_docs/python/tf/transpose ]. At the bottom of the page, it sais Tensorflow do not support strides. Therefore, it have to be contiguous. If you look at [here][stackoverflow.com/questions/50779869/… ], the answer also sais the tensorflow always make copy unless dim-0-aligned.
– ZisIsNotZis
Nov 21 at 8:35
Actually I read about it [here][tensorflow.org/api_docs/python/tf/transpose ]. At the bottom of the page, it sais Tensorflow do not support strides. Therefore, it have to be contiguous. If you look at [here][stackoverflow.com/questions/50779869/… ], the answer also sais the tensorflow always make copy unless dim-0-aligned.
– ZisIsNotZis
Nov 21 at 8:35
But this shouldn't be a that big problem since if you are using Tensorflow, you are usually doing much slower jobs like matrix multiplication and etc. In this sense, copying is the fastest operation you can do with some data, which takes relatively negligible time.
– ZisIsNotZis
Nov 21 at 8:46
But this shouldn't be a that big problem since if you are using Tensorflow, you are usually doing much slower jobs like matrix multiplication and etc. In this sense, copying is the fastest operation you can do with some data, which takes relatively negligible time.
– ZisIsNotZis
Nov 21 at 8:46
|
show 2 more comments
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53398721%2ftensorflow-can-reshape-create-a-copy%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53398721%2ftensorflow-can-reshape-create-a-copy%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Tensors in Tensorflow are always contiguous.reshape-ing contiguous array always result in contiguous array. Therefore,reshapein Tensorflow never have to create copy (not sure if it actually does it or not). On the contrary, operations like transpose always makes copy.– ZisIsNotZis
Nov 21 at 4:25
But since Tensorflow is a "computation graph compiler", it do have some automatic optimization in the graph that might optimize out the copying. I did some google but did't find detail of that.
– ZisIsNotZis
Nov 21 at 4:31
@ZisIsNotZis "Tensors in tensorflow are always contiguous", I don't understand this. If you slice a tensor with a stride [::2], the result will not be contiguous. And such a slice is definitely possible in tensorflow. Does tensorflow implicitly call contiguous() after every op that breaks this invariant? That would not be efficient. I guess this only applies when running a session and evaluating a tensor. But still, how do you keep it contiguous?
– lhk
Nov 21 at 8:27
Actually I read about it [here][tensorflow.org/api_docs/python/tf/transpose ]. At the bottom of the page, it sais Tensorflow do not support strides. Therefore, it have to be contiguous. If you look at [here][stackoverflow.com/questions/50779869/… ], the answer also sais the tensorflow always make copy unless dim-0-aligned.
– ZisIsNotZis
Nov 21 at 8:35
But this shouldn't be a that big problem since if you are using Tensorflow, you are usually doing much slower jobs like matrix multiplication and etc. In this sense, copying is the fastest operation you can do with some data, which takes relatively negligible time.
– ZisIsNotZis
Nov 21 at 8:46