keras layer and shared memory
up vote
-1
down vote
favorite
I am using a very heavy layer in different keras models (a word embedding layer). Since I want to run all models in parallel, I would need to have the layer loaded in memory once for each running process. This is why I was thinking in using a shared memory block for that layer which could be used by all the processes and only one copy of the layer would be loaded in memory at the same time. The layer is not trainable, so only read operations should be done on it. Is it possible? If so, does Keras include something to facilitate this task? Thanks!
keras shared-memory
asked Nov 19 at 12:15
Rodrigo Serna Pérez
23
add a comment |
up vote
-1
down vote
favorite
I am using a very heavy layer in different keras models (a word embedding layer). Since I want to run all models in parallel, I would need to have the layer loaded in memory once for each running process. This is why I was thinking in using a shared memory block for that layer which could be used by all the processes and only one copy of the layer would be loaded in memory at the same time. The layer is not trainable, so only read operations should be done on it. Is it possible? If so, does Keras include something to facilitate this task? Thanks!
keras shared-memory
asked Nov 19 at 12:15
Rodrigo Serna Pérez
23
add a comment |
up vote
-1
down vote
favorite
up vote
-1
down vote
favorite
I am using a very heavy layer in different keras models (a word embedding layer). Since I want to run all models in parallel, I would need to have the layer loaded in memory once for each running process. This is why I was thinking in using a shared memory block for that layer which could be used by all the processes and only one copy of the layer would be loaded in memory at the same time. The layer is not trainable, so only read operations should be done on it. Is it possible? If so, does Keras include something to facilitate this task? Thanks!
keras shared-memory
asked Nov 19 at 12:15
Rodrigo Serna Pérez
23
I am using a very heavy layer in different keras models (a word embedding layer). Since I want to run all models in parallel, I would need to have the layer loaded in memory once for each running process. This is why I was thinking in using a shared memory block for that layer which could be used by all the processes and only one copy of the layer would be loaded in memory at the same time. The layer is not trainable, so only read operations should be done on it. Is it possible? If so, does Keras include something to facilitate this task? Thanks!
keras shared-memory
keras shared-memory
asked Nov 19 at 12:15
Rodrigo Serna Pérez
23
asked Nov 19 at 12:15
Rodrigo Serna Pérez
23
asked Nov 19 at 12:15
Rodrigo Serna Pérez
23
asked Nov 19 at 12:15
Rodrigo Serna Pérez
23
asked Nov 19 at 12:15
Rodrigo Serna Pérez
23
23
add a comment |
add a comment |
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53374437%2fkeras-layer-and-shared-memory%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
