How to define per-layer learning rate in mxnet.gluon?
I know that it is possible to freeze layers in a network for example to train only the last layers of a pre-trained model.
However, I want to know is there any way to apply certain learning rates to different layers. For example, in pytorch it would be:
optimizer = torch.optim.Adam([
{'params': paras['conv1'], 'lr': learning_rate / 10},
{'params': paras['middle'], 'lr': learning_rate / 3},
{'params': paras['fc'], 'lr': learning_rate }
], lr=learning_rate)
Interfaces of gluon and torch are pretty much the same. Any idea how I can do this in gluon?
machine-learning deep-learning mxnet
edited Nov 25 '18 at 13:32
desertnaut
19.4k74076
asked Nov 25 '18 at 12:44
JumpyWarlockJumpyWarlock
216
add a comment |
I know that it is possible to freeze layers in a network for example to train only the last layers of a pre-trained model.
However, I want to know is there any way to apply certain learning rates to different layers. For example, in pytorch it would be:
optimizer = torch.optim.Adam([
{'params': paras['conv1'], 'lr': learning_rate / 10},
{'params': paras['middle'], 'lr': learning_rate / 3},
{'params': paras['fc'], 'lr': learning_rate }
], lr=learning_rate)
Interfaces of gluon and torch are pretty much the same. Any idea how I can do this in gluon?
machine-learning deep-learning mxnet
edited Nov 25 '18 at 13:32
desertnaut
19.4k74076
asked Nov 25 '18 at 12:44
JumpyWarlockJumpyWarlock
216
add a comment |
I know that it is possible to freeze layers in a network for example to train only the last layers of a pre-trained model.
However, I want to know is there any way to apply certain learning rates to different layers. For example, in pytorch it would be:
optimizer = torch.optim.Adam([
{'params': paras['conv1'], 'lr': learning_rate / 10},
{'params': paras['middle'], 'lr': learning_rate / 3},
{'params': paras['fc'], 'lr': learning_rate }
], lr=learning_rate)
Interfaces of gluon and torch are pretty much the same. Any idea how I can do this in gluon?
machine-learning deep-learning mxnet
edited Nov 25 '18 at 13:32
desertnaut
19.4k74076
asked Nov 25 '18 at 12:44
JumpyWarlockJumpyWarlock
216
I know that it is possible to freeze layers in a network for example to train only the last layers of a pre-trained model.
However, I want to know is there any way to apply certain learning rates to different layers. For example, in pytorch it would be:
optimizer = torch.optim.Adam([
{'params': paras['conv1'], 'lr': learning_rate / 10},
{'params': paras['middle'], 'lr': learning_rate / 3},
{'params': paras['fc'], 'lr': learning_rate }
], lr=learning_rate)
Interfaces of gluon and torch are pretty much the same. Any idea how I can do this in gluon?
machine-learning deep-learning mxnet
machine-learning deep-learning mxnet
edited Nov 25 '18 at 13:32
desertnaut
19.4k74076
asked Nov 25 '18 at 12:44
JumpyWarlockJumpyWarlock
216
edited Nov 25 '18 at 13:32
desertnaut
19.4k74076
asked Nov 25 '18 at 12:44
JumpyWarlockJumpyWarlock
216
edited Nov 25 '18 at 13:32
desertnaut
19.4k74076
edited Nov 25 '18 at 13:32
desertnaut
19.4k74076
edited Nov 25 '18 at 13:32
desertnaut
19.4k74076
19.4k74076
asked Nov 25 '18 at 12:44
JumpyWarlockJumpyWarlock
216
asked Nov 25 '18 at 12:44
JumpyWarlockJumpyWarlock
216
asked Nov 25 '18 at 12:44
JumpyWarlockJumpyWarlock
216
216
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
You can adjust the learning rate in each layer by modifying lr_mult:
for key, value in model.collect_params().items():
print value.lr_mult
edited Nov 29 '18 at 20:43
willk
1,181722
answered Nov 29 '18 at 17:05
NRauschmayrNRauschmayr
211
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53467556%2fhow-to-define-per-layer-learning-rate-in-mxnet-gluon%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
You can adjust the learning rate in each layer by modifying lr_mult:
for key, value in model.collect_params().items():
print value.lr_mult
edited Nov 29 '18 at 20:43
willk
1,181722
answered Nov 29 '18 at 17:05
NRauschmayrNRauschmayr
211
add a comment |
You can adjust the learning rate in each layer by modifying lr_mult:
for key, value in model.collect_params().items():
print value.lr_mult
edited Nov 29 '18 at 20:43
willk
1,181722
answered Nov 29 '18 at 17:05
NRauschmayrNRauschmayr
211
add a comment |
You can adjust the learning rate in each layer by modifying lr_mult:
for key, value in model.collect_params().items():
print value.lr_mult
edited Nov 29 '18 at 20:43
willk
1,181722
answered Nov 29 '18 at 17:05
NRauschmayrNRauschmayr
211
You can adjust the learning rate in each layer by modifying lr_mult:
for key, value in model.collect_params().items():
print value.lr_mult
edited Nov 29 '18 at 20:43
willk
1,181722
answered Nov 29 '18 at 17:05
NRauschmayrNRauschmayr
211
edited Nov 29 '18 at 20:43
willk
1,181722
edited Nov 29 '18 at 20:43
willk
1,181722
edited Nov 29 '18 at 20:43
willk
1,181722
1,181722
answered Nov 29 '18 at 17:05
NRauschmayrNRauschmayr
211
answered Nov 29 '18 at 17:05
NRauschmayrNRauschmayr
211
answered Nov 29 '18 at 17:05
NRauschmayrNRauschmayr
211
211
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53467556%2fhow-to-define-per-layer-learning-rate-in-mxnet-gluon%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
