Does collect return a list snapshot if run on a parallel stream?
up vote
2
down vote
favorite
I have a unit test that started to fail on Circle CI only. It fails on the last line in this (Kotlin) example:
generator.generateNames(50) // returns List<String>
.parallelStream()
.map { name ->
val playerId = "${name.firstName.toLowerCase()}"
Player(playerId = playerId)
}.collect(Collectors.toList()).last()
throwing: Caused by: java.util.NoSuchElementException.
It works always on my local machine or on Circle CI if I do not use a parallel stream. My theory is that the collect call returns a List snapshot (it actually doesn't block until the List is completely filled) and that CI doesn't have enough CPU to collect a single element in other threads?
However, my stream is ordered and so is the Collector right? Is this even collecting in parallel?
kotlin java-8 java-stream
asked 2 days ago
reikje
1,11711325
add a comment |
up vote
2
down vote
favorite
I have a unit test that started to fail on Circle CI only. It fails on the last line in this (Kotlin) example:
generator.generateNames(50) // returns List<String>
.parallelStream()
.map { name ->
val playerId = "${name.firstName.toLowerCase()}"
Player(playerId = playerId)
}.collect(Collectors.toList()).last()
throwing: Caused by: java.util.NoSuchElementException.
It works always on my local machine or on Circle CI if I do not use a parallel stream. My theory is that the collect call returns a List snapshot (it actually doesn't block until the List is completely filled) and that CI doesn't have enough CPU to collect a single element in other threads?
However, my stream is ordered and so is the Collector right? Is this even collecting in parallel?
kotlin java-8 java-stream
asked 2 days ago
reikje
1,11711325
Does the returned list definitely contain an element?.last()will throwNoSuchElementExceptionif it is empty
– Eamon Scullion
2 days ago
Yes. Like I said running the code unchanged on my dev machine always work. On CI it works if I turn the parallel stream into a sequential one.
– reikje
2 days ago
2
Not very sure of what Kotlin does, but if making things sequential seems to be the only solution currently. Then I would suggest trying tocollect(Collectors.toList())asPlayerlist and break the part of fetching the last into two steps. Something like this in java :List<Player> playerList = generator.generateNames(50) // returns List<String> .parallelStream() .map(name -> new Player(name.getFirstName.toLowerCase()) }.collect(Collectors.toList()); Player lastplayer = playerList.get(playerList.size()-1);.. note that this shouldn't work either on Circle CI for your code.
– nullpointer
2 days ago
3
I don’t understand, in which regard “Circle CI” is relevant in this operation at all. The only thing that doesn’t look standard Java or trivial operation, isgenerator.generateNames(50). If this method returns one of the standardListimplementations, there should be no problem. But perhaps, it returns some kind of lazy list implementation that is not thread safe?
– Holger
2 days ago
What flavour and version of a JDK are you using on both machines?
– Jayson Minard
2 days ago
add a comment |
up vote
2
down vote
favorite
up vote
2
down vote
favorite
I have a unit test that started to fail on Circle CI only. It fails on the last line in this (Kotlin) example:
generator.generateNames(50) // returns List<String>
.parallelStream()
.map { name ->
val playerId = "${name.firstName.toLowerCase()}"
Player(playerId = playerId)
}.collect(Collectors.toList()).last()
throwing: Caused by: java.util.NoSuchElementException.
It works always on my local machine or on Circle CI if I do not use a parallel stream. My theory is that the collect call returns a List snapshot (it actually doesn't block until the List is completely filled) and that CI doesn't have enough CPU to collect a single element in other threads?
However, my stream is ordered and so is the Collector right? Is this even collecting in parallel?
kotlin java-8 java-stream
asked 2 days ago
reikje
1,11711325
I have a unit test that started to fail on Circle CI only. It fails on the last line in this (Kotlin) example:
generator.generateNames(50) // returns List<String>
.parallelStream()
.map { name ->
val playerId = "${name.firstName.toLowerCase()}"
Player(playerId = playerId)
}.collect(Collectors.toList()).last()
throwing: Caused by: java.util.NoSuchElementException.
It works always on my local machine or on Circle CI if I do not use a parallel stream. My theory is that the collect call returns a List snapshot (it actually doesn't block until the List is completely filled) and that CI doesn't have enough CPU to collect a single element in other threads?
However, my stream is ordered and so is the Collector right? Is this even collecting in parallel?
kotlin java-8 java-stream
kotlin java-8 java-stream
asked 2 days ago
reikje
1,11711325
asked 2 days ago
reikje
1,11711325
asked 2 days ago
reikje
1,11711325
asked 2 days ago
reikje
1,11711325
asked 2 days ago
reikje
1,11711325
1,11711325
Does the returned list definitely contain an element?.last()will throwNoSuchElementExceptionif it is empty
– Eamon Scullion
2 days ago
Yes. Like I said running the code unchanged on my dev machine always work. On CI it works if I turn the parallel stream into a sequential one.
– reikje
2 days ago
2
Not very sure of what Kotlin does, but if making things sequential seems to be the only solution currently. Then I would suggest trying tocollect(Collectors.toList())asPlayerlist and break the part of fetching the last into two steps. Something like this in java :List<Player> playerList = generator.generateNames(50) // returns List<String> .parallelStream() .map(name -> new Player(name.getFirstName.toLowerCase()) }.collect(Collectors.toList()); Player lastplayer = playerList.get(playerList.size()-1);.. note that this shouldn't work either on Circle CI for your code.
– nullpointer
2 days ago
3
I don’t understand, in which regard “Circle CI” is relevant in this operation at all. The only thing that doesn’t look standard Java or trivial operation, isgenerator.generateNames(50). If this method returns one of the standardListimplementations, there should be no problem. But perhaps, it returns some kind of lazy list implementation that is not thread safe?
– Holger
2 days ago
What flavour and version of a JDK are you using on both machines?
– Jayson Minard
2 days ago
add a comment |
Does the returned list definitely contain an element?.last()will throwNoSuchElementExceptionif it is empty
– Eamon Scullion
2 days ago
Yes. Like I said running the code unchanged on my dev machine always work. On CI it works if I turn the parallel stream into a sequential one.
– reikje
2 days ago
2
Not very sure of what Kotlin does, but if making things sequential seems to be the only solution currently. Then I would suggest trying tocollect(Collectors.toList())asPlayerlist and break the part of fetching the last into two steps. Something like this in java :List<Player> playerList = generator.generateNames(50) // returns List<String> .parallelStream() .map(name -> new Player(name.getFirstName.toLowerCase()) }.collect(Collectors.toList()); Player lastplayer = playerList.get(playerList.size()-1);.. note that this shouldn't work either on Circle CI for your code.
– nullpointer
2 days ago
3
I don’t understand, in which regard “Circle CI” is relevant in this operation at all. The only thing that doesn’t look standard Java or trivial operation, isgenerator.generateNames(50). If this method returns one of the standardListimplementations, there should be no problem. But perhaps, it returns some kind of lazy list implementation that is not thread safe?
– Holger
2 days ago
What flavour and version of a JDK are you using on both machines?
– Jayson Minard
2 days ago
Does the returned list definitely contain an element?
.last() will throw NoSuchElementException if it is empty– Eamon Scullion
2 days ago
Does the returned list definitely contain an element?
.last() will throw NoSuchElementException if it is empty– Eamon Scullion
2 days ago
Yes. Like I said running the code unchanged on my dev machine always work. On CI it works if I turn the parallel stream into a sequential one.
– reikje
2 days ago
Yes. Like I said running the code unchanged on my dev machine always work. On CI it works if I turn the parallel stream into a sequential one.
– reikje
2 days ago
2
2
Not very sure of what Kotlin does, but if making things sequential seems to be the only solution currently. Then I would suggest trying to
collect(Collectors.toList()) as Player list and break the part of fetching the last into two steps. Something like this in java : List<Player> playerList = generator.generateNames(50) // returns List<String> .parallelStream() .map(name -> new Player(name.getFirstName.toLowerCase()) }.collect(Collectors.toList()); Player lastplayer = playerList.get(playerList.size()-1); .. note that this shouldn't work either on Circle CI for your code.– nullpointer
2 days ago
Not very sure of what Kotlin does, but if making things sequential seems to be the only solution currently. Then I would suggest trying to
collect(Collectors.toList()) as Player list and break the part of fetching the last into two steps. Something like this in java : List<Player> playerList = generator.generateNames(50) // returns List<String> .parallelStream() .map(name -> new Player(name.getFirstName.toLowerCase()) }.collect(Collectors.toList()); Player lastplayer = playerList.get(playerList.size()-1); .. note that this shouldn't work either on Circle CI for your code.– nullpointer
2 days ago
3
3
I don’t understand, in which regard “Circle CI” is relevant in this operation at all. The only thing that doesn’t look standard Java or trivial operation, is
generator.generateNames(50). If this method returns one of the standard List implementations, there should be no problem. But perhaps, it returns some kind of lazy list implementation that is not thread safe?– Holger
2 days ago
I don’t understand, in which regard “Circle CI” is relevant in this operation at all. The only thing that doesn’t look standard Java or trivial operation, is
generator.generateNames(50). If this method returns one of the standard List implementations, there should be no problem. But perhaps, it returns some kind of lazy list implementation that is not thread safe?– Holger
2 days ago
What flavour and version of a JDK are you using on both machines?
– Jayson Minard
2 days ago
What flavour and version of a JDK are you using on both machines?
– Jayson Minard
2 days ago
add a comment |
1 Answer
1
active
oldest
votes
up vote
3
down vote
accepted
The exception you are getting probably has a message in it, and not just the name of the exception. That message is likely telling you the error. For example, the last part of your code calls the Kotlin extension function last() which in the implementation:
public fun <T> List<T>.last(): T {
if (isEmpty())
throw NoSuchElementException("List is empty.")
return this[lastIndex]
}
So if you are seeing "List is empty" message in the stack trace for java.util.NoSuchElementException then that is the cause.
Also, if you share the stack trace you can actually see what is throwing the exception. But looking at your code this is the only likely candidate.
The question then is, "why is the final list empty?!" ... is generateNames(50) working differently in this environment? The problem is not with collect(Collectors.toList()) which provides a synchronous result.
answered 2 days ago
Jayson Minard
35.7k13103170
1
After some investigation, it turned out that running a parallel stream exhausted a JedisPool. I excluded that line above because I thought it would be not relevant. It was a bit unfortunate that the exhausted pool also came back with aNoSuchElementException. That got me to the wrong track. But basically you answered my question: whencollect(Collectors.toList())returns, theListis always fully populated - also when using parallel streams.
– reikje
2 days ago
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
3
down vote
accepted
The exception you are getting probably has a message in it, and not just the name of the exception. That message is likely telling you the error. For example, the last part of your code calls the Kotlin extension function last() which in the implementation:
public fun <T> List<T>.last(): T {
if (isEmpty())
throw NoSuchElementException("List is empty.")
return this[lastIndex]
}
So if you are seeing "List is empty" message in the stack trace for java.util.NoSuchElementException then that is the cause.
Also, if you share the stack trace you can actually see what is throwing the exception. But looking at your code this is the only likely candidate.
The question then is, "why is the final list empty?!" ... is generateNames(50) working differently in this environment? The problem is not with collect(Collectors.toList()) which provides a synchronous result.
answered 2 days ago
Jayson Minard
35.7k13103170
1
After some investigation, it turned out that running a parallel stream exhausted a JedisPool. I excluded that line above because I thought it would be not relevant. It was a bit unfortunate that the exhausted pool also came back with aNoSuchElementException. That got me to the wrong track. But basically you answered my question: whencollect(Collectors.toList())returns, theListis always fully populated - also when using parallel streams.
– reikje
2 days ago
add a comment |
up vote
3
down vote
accepted
The exception you are getting probably has a message in it, and not just the name of the exception. That message is likely telling you the error. For example, the last part of your code calls the Kotlin extension function last() which in the implementation:
public fun <T> List<T>.last(): T {
if (isEmpty())
throw NoSuchElementException("List is empty.")
return this[lastIndex]
}
So if you are seeing "List is empty" message in the stack trace for java.util.NoSuchElementException then that is the cause.
Also, if you share the stack trace you can actually see what is throwing the exception. But looking at your code this is the only likely candidate.
The question then is, "why is the final list empty?!" ... is generateNames(50) working differently in this environment? The problem is not with collect(Collectors.toList()) which provides a synchronous result.
answered 2 days ago
Jayson Minard
35.7k13103170
1
After some investigation, it turned out that running a parallel stream exhausted a JedisPool. I excluded that line above because I thought it would be not relevant. It was a bit unfortunate that the exhausted pool also came back with aNoSuchElementException. That got me to the wrong track. But basically you answered my question: whencollect(Collectors.toList())returns, theListis always fully populated - also when using parallel streams.
– reikje
2 days ago
add a comment |
up vote
3
down vote
accepted
up vote
3
down vote
accepted
The exception you are getting probably has a message in it, and not just the name of the exception. That message is likely telling you the error. For example, the last part of your code calls the Kotlin extension function last() which in the implementation:
public fun <T> List<T>.last(): T {
if (isEmpty())
throw NoSuchElementException("List is empty.")
return this[lastIndex]
}
So if you are seeing "List is empty" message in the stack trace for java.util.NoSuchElementException then that is the cause.
Also, if you share the stack trace you can actually see what is throwing the exception. But looking at your code this is the only likely candidate.
The question then is, "why is the final list empty?!" ... is generateNames(50) working differently in this environment? The problem is not with collect(Collectors.toList()) which provides a synchronous result.
answered 2 days ago
Jayson Minard
35.7k13103170
The exception you are getting probably has a message in it, and not just the name of the exception. That message is likely telling you the error. For example, the last part of your code calls the Kotlin extension function last() which in the implementation:
public fun <T> List<T>.last(): T {
if (isEmpty())
throw NoSuchElementException("List is empty.")
return this[lastIndex]
}
So if you are seeing "List is empty" message in the stack trace for java.util.NoSuchElementException then that is the cause.
Also, if you share the stack trace you can actually see what is throwing the exception. But looking at your code this is the only likely candidate.
The question then is, "why is the final list empty?!" ... is generateNames(50) working differently in this environment? The problem is not with collect(Collectors.toList()) which provides a synchronous result.
answered 2 days ago
Jayson Minard
35.7k13103170
edited 2 days ago
answered 2 days ago
Jayson Minard
35.7k13103170
answered 2 days ago
Jayson Minard
35.7k13103170
answered 2 days ago
Jayson Minard
35.7k13103170
35.7k13103170
1
After some investigation, it turned out that running a parallel stream exhausted a JedisPool. I excluded that line above because I thought it would be not relevant. It was a bit unfortunate that the exhausted pool also came back with aNoSuchElementException. That got me to the wrong track. But basically you answered my question: whencollect(Collectors.toList())returns, theListis always fully populated - also when using parallel streams.
– reikje
2 days ago
add a comment |
1
After some investigation, it turned out that running a parallel stream exhausted a JedisPool. I excluded that line above because I thought it would be not relevant. It was a bit unfortunate that the exhausted pool also came back with aNoSuchElementException. That got me to the wrong track. But basically you answered my question: whencollect(Collectors.toList())returns, theListis always fully populated - also when using parallel streams.
– reikje
2 days ago
1
1
After some investigation, it turned out that running a parallel stream exhausted a JedisPool. I excluded that line above because I thought it would be not relevant. It was a bit unfortunate that the exhausted pool also came back with a
NoSuchElementException. That got me to the wrong track. But basically you answered my question: when collect(Collectors.toList()) returns, the List is always fully populated - also when using parallel streams.– reikje
2 days ago
After some investigation, it turned out that running a parallel stream exhausted a JedisPool. I excluded that line above because I thought it would be not relevant. It was a bit unfortunate that the exhausted pool also came back with a
NoSuchElementException. That got me to the wrong track. But basically you answered my question: when collect(Collectors.toList()) returns, the List is always fully populated - also when using parallel streams.– reikje
2 days ago
add a comment |
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53373038%2fdoes-collect-return-a-list-snapshot-if-run-on-a-parallel-stream%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Does the returned list definitely contain an element?
.last()will throwNoSuchElementExceptionif it is empty– Eamon Scullion
2 days ago
Yes. Like I said running the code unchanged on my dev machine always work. On CI it works if I turn the parallel stream into a sequential one.
– reikje
2 days ago
2
Not very sure of what Kotlin does, but if making things sequential seems to be the only solution currently. Then I would suggest trying to
collect(Collectors.toList())asPlayerlist and break the part of fetching the last into two steps. Something like this in java :List<Player> playerList = generator.generateNames(50) // returns List<String> .parallelStream() .map(name -> new Player(name.getFirstName.toLowerCase()) }.collect(Collectors.toList()); Player lastplayer = playerList.get(playerList.size()-1);.. note that this shouldn't work either on Circle CI for your code.– nullpointer
2 days ago
3
I don’t understand, in which regard “Circle CI” is relevant in this operation at all. The only thing that doesn’t look standard Java or trivial operation, is
generator.generateNames(50). If this method returns one of the standardListimplementations, there should be no problem. But perhaps, it returns some kind of lazy list implementation that is not thread safe?– Holger
2 days ago
What flavour and version of a JDK are you using on both machines?
– Jayson Minard
2 days ago