Thread waiting for sun.misc.Unsafe.park in spark createDataFrame
up vote
0
down vote
favorite
I am using spark to filter out data from a JavaRDD dataframe.
It has around 2900000 record of which I need to filter 29000.
There is no memory issue as I am using a super server with enought memory.
Below is the code
JavaRDD<SomeClass> sample = dataframe.javaRDD().map{..........};
Dataset<Row> data= spark.createDataFrame(sample.rdd(), SomeClass.class).filter(col("column1").isNotNull());
It just hangs on 2nd line for hours. When I tried checking the debug point by pausing the debug I was able to find the code was stuck at sun.misc.Unsafe.park.
I checked similar answers none helped.
There is no CPU utilization as well. And there can't be any memory issue as I am using super server.
apache-spark threadpool deadlock
asked Nov 19 at 14:57
Mehaboob Khan
102113
add a comment |
up vote
0
down vote
favorite
I am using spark to filter out data from a JavaRDD dataframe.
It has around 2900000 record of which I need to filter 29000.
There is no memory issue as I am using a super server with enought memory.
Below is the code
JavaRDD<SomeClass> sample = dataframe.javaRDD().map{..........};
Dataset<Row> data= spark.createDataFrame(sample.rdd(), SomeClass.class).filter(col("column1").isNotNull());
It just hangs on 2nd line for hours. When I tried checking the debug point by pausing the debug I was able to find the code was stuck at sun.misc.Unsafe.park.
I checked similar answers none helped.
There is no CPU utilization as well. And there can't be any memory issue as I am using super server.
apache-spark threadpool deadlock
asked Nov 19 at 14:57
Mehaboob Khan
102113
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I am using spark to filter out data from a JavaRDD dataframe.
It has around 2900000 record of which I need to filter 29000.
There is no memory issue as I am using a super server with enought memory.
Below is the code
JavaRDD<SomeClass> sample = dataframe.javaRDD().map{..........};
Dataset<Row> data= spark.createDataFrame(sample.rdd(), SomeClass.class).filter(col("column1").isNotNull());
It just hangs on 2nd line for hours. When I tried checking the debug point by pausing the debug I was able to find the code was stuck at sun.misc.Unsafe.park.
I checked similar answers none helped.
There is no CPU utilization as well. And there can't be any memory issue as I am using super server.
apache-spark threadpool deadlock
asked Nov 19 at 14:57
Mehaboob Khan
102113
I am using spark to filter out data from a JavaRDD dataframe.
It has around 2900000 record of which I need to filter 29000.
There is no memory issue as I am using a super server with enought memory.
Below is the code
JavaRDD<SomeClass> sample = dataframe.javaRDD().map{..........};
Dataset<Row> data= spark.createDataFrame(sample.rdd(), SomeClass.class).filter(col("column1").isNotNull());
It just hangs on 2nd line for hours. When I tried checking the debug point by pausing the debug I was able to find the code was stuck at sun.misc.Unsafe.park.
I checked similar answers none helped.
There is no CPU utilization as well. And there can't be any memory issue as I am using super server.
apache-spark threadpool deadlock
apache-spark threadpool deadlock
asked Nov 19 at 14:57
Mehaboob Khan
102113
asked Nov 19 at 14:57
Mehaboob Khan
102113
asked Nov 19 at 14:57
Mehaboob Khan
102113
asked Nov 19 at 14:57
Mehaboob Khan
102113
asked Nov 19 at 14:57
Mehaboob Khan
102113
102113
add a comment |
add a comment |
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53377284%2fthread-waiting-for-sun-misc-unsafe-park-in-spark-createdataframe%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
