Keras model has very good loss after 1 epoch but doesn't really get better with more epochs
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
Hello I just wanted to ask this theoretical question.
What could be the causes of a model that has already a very good loss (0.004 on normalized data) after one single epoch but this loss doesn't really decrease over time (after 10 epochs it's still 0.0032).
Shouldn't it normally decrease way more over time?
The dataset is pretty big with bit more than a million datapoints and I didn't expect this very good loss just after 1 epoch.
So what could I change about this model or what am I doing wrong? (it's a densely connected NN predicting regression with adam and mse)
tensorflow machine-learning keras
asked Nov 26 '18 at 20:26
Dirkx SenneDirkx Senne
5810
add a comment |
Hello I just wanted to ask this theoretical question.
What could be the causes of a model that has already a very good loss (0.004 on normalized data) after one single epoch but this loss doesn't really decrease over time (after 10 epochs it's still 0.0032).
Shouldn't it normally decrease way more over time?
The dataset is pretty big with bit more than a million datapoints and I didn't expect this very good loss just after 1 epoch.
So what could I change about this model or what am I doing wrong? (it's a densely connected NN predicting regression with adam and mse)
tensorflow machine-learning keras
asked Nov 26 '18 at 20:26
Dirkx SenneDirkx Senne
5810
Sorry but this is not a programming question.
– Matias Valdenegro
Nov 26 '18 at 23:05
add a comment |
Hello I just wanted to ask this theoretical question.
What could be the causes of a model that has already a very good loss (0.004 on normalized data) after one single epoch but this loss doesn't really decrease over time (after 10 epochs it's still 0.0032).
Shouldn't it normally decrease way more over time?
The dataset is pretty big with bit more than a million datapoints and I didn't expect this very good loss just after 1 epoch.
So what could I change about this model or what am I doing wrong? (it's a densely connected NN predicting regression with adam and mse)
tensorflow machine-learning keras
asked Nov 26 '18 at 20:26
Dirkx SenneDirkx Senne
5810
Hello I just wanted to ask this theoretical question.
What could be the causes of a model that has already a very good loss (0.004 on normalized data) after one single epoch but this loss doesn't really decrease over time (after 10 epochs it's still 0.0032).
Shouldn't it normally decrease way more over time?
The dataset is pretty big with bit more than a million datapoints and I didn't expect this very good loss just after 1 epoch.
So what could I change about this model or what am I doing wrong? (it's a densely connected NN predicting regression with adam and mse)
tensorflow machine-learning keras
tensorflow machine-learning keras
asked Nov 26 '18 at 20:26
Dirkx SenneDirkx Senne
5810
asked Nov 26 '18 at 20:26
Dirkx SenneDirkx Senne
5810
asked Nov 26 '18 at 20:26
Dirkx SenneDirkx Senne
5810
asked Nov 26 '18 at 20:26
Dirkx SenneDirkx Senne
5810
asked Nov 26 '18 at 20:26
Dirkx SenneDirkx Senne
5810
5810
Sorry but this is not a programming question.
– Matias Valdenegro
Nov 26 '18 at 23:05
add a comment |
Sorry but this is not a programming question.
– Matias Valdenegro
Nov 26 '18 at 23:05
Sorry but this is not a programming question.
– Matias Valdenegro
Nov 26 '18 at 23:05
Sorry but this is not a programming question.
– Matias Valdenegro
Nov 26 '18 at 23:05
add a comment |
1 Answer
1
active
oldest
votes
There are multiple possibilities, but the problem needs some clarification.
Could you specify the range of your target?
0.004 might sound low as a loss, but it's not if your target ranges from 0 to 0.0001 for example.
What are the metrics of your validation & test data set? Loss on itself does not say much without knowing the validation loss.
Guessing that the 0.004 is too good to be true, your model might be over fitting.
Try implementing dropout to avoid over fitting.
In case your model is not over fitting, it might be the case that Adam is overshooting a (local) minima. Try lowering its learning rate, or try sgd with custom hyper-parameters. This does take a lot of tuning.
There is a free course on Coursera called Machine Learning by Stanford. This covers theory on these concepts (and more) in a good way.
answered Nov 26 '18 at 22:21
deKeijzerdeKeijzer
8712
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53488524%2fkeras-model-has-very-good-loss-after-1-epoch-but-doesnt-really-get-better-with%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
There are multiple possibilities, but the problem needs some clarification.
Could you specify the range of your target?
0.004 might sound low as a loss, but it's not if your target ranges from 0 to 0.0001 for example.
What are the metrics of your validation & test data set? Loss on itself does not say much without knowing the validation loss.
Guessing that the 0.004 is too good to be true, your model might be over fitting.
Try implementing dropout to avoid over fitting.
In case your model is not over fitting, it might be the case that Adam is overshooting a (local) minima. Try lowering its learning rate, or try sgd with custom hyper-parameters. This does take a lot of tuning.
There is a free course on Coursera called Machine Learning by Stanford. This covers theory on these concepts (and more) in a good way.
answered Nov 26 '18 at 22:21
deKeijzerdeKeijzer
8712
add a comment |
There are multiple possibilities, but the problem needs some clarification.
Could you specify the range of your target?
0.004 might sound low as a loss, but it's not if your target ranges from 0 to 0.0001 for example.
What are the metrics of your validation & test data set? Loss on itself does not say much without knowing the validation loss.
Guessing that the 0.004 is too good to be true, your model might be over fitting.
Try implementing dropout to avoid over fitting.
In case your model is not over fitting, it might be the case that Adam is overshooting a (local) minima. Try lowering its learning rate, or try sgd with custom hyper-parameters. This does take a lot of tuning.
There is a free course on Coursera called Machine Learning by Stanford. This covers theory on these concepts (and more) in a good way.
answered Nov 26 '18 at 22:21
deKeijzerdeKeijzer
8712
add a comment |
There are multiple possibilities, but the problem needs some clarification.
Could you specify the range of your target?
0.004 might sound low as a loss, but it's not if your target ranges from 0 to 0.0001 for example.
What are the metrics of your validation & test data set? Loss on itself does not say much without knowing the validation loss.
Guessing that the 0.004 is too good to be true, your model might be over fitting.
Try implementing dropout to avoid over fitting.
In case your model is not over fitting, it might be the case that Adam is overshooting a (local) minima. Try lowering its learning rate, or try sgd with custom hyper-parameters. This does take a lot of tuning.
There is a free course on Coursera called Machine Learning by Stanford. This covers theory on these concepts (and more) in a good way.
answered Nov 26 '18 at 22:21
deKeijzerdeKeijzer
8712
There are multiple possibilities, but the problem needs some clarification.
Could you specify the range of your target?
0.004 might sound low as a loss, but it's not if your target ranges from 0 to 0.0001 for example.
What are the metrics of your validation & test data set? Loss on itself does not say much without knowing the validation loss.
Guessing that the 0.004 is too good to be true, your model might be over fitting.
Try implementing dropout to avoid over fitting.
In case your model is not over fitting, it might be the case that Adam is overshooting a (local) minima. Try lowering its learning rate, or try sgd with custom hyper-parameters. This does take a lot of tuning.
There is a free course on Coursera called Machine Learning by Stanford. This covers theory on these concepts (and more) in a good way.
answered Nov 26 '18 at 22:21
deKeijzerdeKeijzer
8712
answered Nov 26 '18 at 22:21
deKeijzerdeKeijzer
8712
answered Nov 26 '18 at 22:21
deKeijzerdeKeijzer
8712
answered Nov 26 '18 at 22:21
deKeijzerdeKeijzer
8712
8712
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53488524%2fkeras-model-has-very-good-loss-after-1-epoch-but-doesnt-really-get-better-with%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Sorry but this is not a programming question.
– Matias Valdenegro
Nov 26 '18 at 23:05