Using Capybara to interact with lazy loading elements

Currently I have trouble in with interacting with the elements within the page using lazy loading. I need to select the list of items (which mostly from top till the bottom of page, up to like 100 items). With the lazy loading implemented, i could only select a portion of it, like 1/3 of that since they split all the items into 3 different portions then it will only load the the first portion. I found out using a tricky Javascript could help to scroll it to the bottom like page.execute_script("window.scrollTo(0,100000)"), then i could probably get all the items.

but should be saving that as the last option.

So, my question is Does Capybara support to interact with Lazy Loading stuff like that . Also what should I do to get the whole items, without using that Javascript ?

asked Nov 24 '18 at 16:19

Dinh Luong

Are you testing an app, or just scraping data from a site?

– Thomas Walpole
Nov 24 '18 at 16:57

add a comment |

but should be saving that as the last option.

So, my question is Does Capybara support to interact with Lazy Loading stuff like that . Also what should I do to get the whole items, without using that Javascript ?

asked Nov 24 '18 at 16:19

Dinh Luong

Are you testing an app, or just scraping data from a site?

– Thomas Walpole
Nov 24 '18 at 16:57

add a comment |

but should be saving that as the last option.

So, my question is Does Capybara support to interact with Lazy Loading stuff like that . Also what should I do to get the whole items, without using that Javascript ?

asked Nov 24 '18 at 16:19

Dinh Luong

but should be saving that as the last option.

So, my question is Does Capybara support to interact with Lazy Loading stuff like that . Also what should I do to get the whole items, without using that Javascript ?

ruby capybara

asked Nov 24 '18 at 16:19

Dinh Luong

asked Nov 24 '18 at 16:19

Dinh Luong

asked Nov 24 '18 at 16:19

Dinh Luong

asked Nov 24 '18 at 16:19

Dinh Luong

asked Nov 24 '18 at 16:19

Dinh Luong

Are you testing an app, or just scraping data from a site?

– Thomas Walpole
Nov 24 '18 at 16:57

add a comment |

Are you testing an app, or just scraping data from a site?

– Thomas Walpole
Nov 24 '18 at 16:57

Are you testing an app, or just scraping data from a site?

– Thomas Walpole
Nov 24 '18 at 16:57

add a comment |

1 Answer
1

active

oldest

votes

If you're just scraping data from a site then the easiest solution is going to be just using execute_script to scroll the page. If however you're testing an app behaves correctly then you want to stay away from execute_script since it can allow you to do things a user never could which may invalidate your tests. Instead use hover to move the mouse pointer over an element on the page that would trigger loading the next portion. For instance if you have a list of items showing up

<ul id="my_list_of_items">

  <li class="item"></li>

  <li class="item"></li>

  ...

  <li class="item"></li>

</ul>

and the next group of items are loaded into the page when the bottom of the list is scrolled into view then do something like

find('.item:last-child').hover

This will cause the last item element on the page to be scrolled into view and the move pointer moved over it, which would then trigger loading of more items.

answered Nov 25 '18 at 19:33

Thomas Walpole

31.3k33052

Thanks for your help @Thomas , actually I just wanna scrape data from website then get about like 10 samples from the list of data I get. It was fine before by just using execute_script then i got the things i want. Unfortunately, right now with the lazy loading, it must have been scrolled over and over from the top to bottom until the JS script finish then i could have the whole data i want. Using your solution is not applicable in this case cause the selector i wanna test is one of the attribute like data-test-id = ... (which has been set for testing purpose), was not the fixed class/id

– Dinh Luong
Dec 3 '18 at 15:35

Beside that, i managed to use the execute_script to get the data i want. However, since it was async, so before the JS finished (~ 40 secs) , it has the data that i use for assertion, which triggered the next step to run when it wasn't really finishing. So i wonder if there's something that i could use to implicit wait at this stage or even overriding the default waiting time of Capybara just for waiting for JS script running to be finished ? Thanks

– Dinh Luong
Dec 3 '18 at 15:42

@DinhLuong it would still apply, you just need to change the selector to something that would match the last item in the list - #list > li:last-child, etc - However since you’re scraping execute_script is easier and just as valid.

– Thomas Walpole
Dec 3 '18 at 15:46

add a comment |

Your Answer

StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});

}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53460068%2fusing-capybara-to-interact-with-lazy-loading-elements%23new-answer', 'question_page');
}
);

Post as a guest

Name

Required, but never shown

1 Answer
1

active

oldest

votes

1 Answer
1

active

oldest

votes

<ul id="my_list_of_items">

  <li class="item"></li>

  <li class="item"></li>

  ...

  <li class="item"></li>

</ul>

and the next group of items are loaded into the page when the bottom of the list is scrolled into view then do something like

find('.item:last-child').hover

This will cause the last item element on the page to be scrolled into view and the move pointer moved over it, which would then trigger loading of more items.

answered Nov 25 '18 at 19:33

Thomas Walpole

31.3k33052

Thanks for your help @Thomas , actually I just wanna scrape data from website then get about like 10 samples from the list of data I get. It was fine before by just using execute_script then i got the things i want. Unfortunately, right now with the lazy loading, it must have been scrolled over and over from the top to bottom until the JS script finish then i could have the whole data i want. Using your solution is not applicable in this case cause the selector i wanna test is one of the attribute like data-test-id = ... (which has been set for testing purpose), was not the fixed class/id

– Dinh Luong
Dec 3 '18 at 15:35

Beside that, i managed to use the execute_script to get the data i want. However, since it was async, so before the JS finished (~ 40 secs) , it has the data that i use for assertion, which triggered the next step to run when it wasn't really finishing. So i wonder if there's something that i could use to implicit wait at this stage or even overriding the default waiting time of Capybara just for waiting for JS script running to be finished ? Thanks

– Dinh Luong
Dec 3 '18 at 15:42

@DinhLuong it would still apply, you just need to change the selector to something that would match the last item in the list - #list > li:last-child, etc - However since you’re scraping execute_script is easier and just as valid.

– Thomas Walpole
Dec 3 '18 at 15:46

add a comment |

<ul id="my_list_of_items">

  <li class="item"></li>

  <li class="item"></li>

  ...

  <li class="item"></li>

</ul>

and the next group of items are loaded into the page when the bottom of the list is scrolled into view then do something like

find('.item:last-child').hover

This will cause the last item element on the page to be scrolled into view and the move pointer moved over it, which would then trigger loading of more items.

answered Nov 25 '18 at 19:33

Thomas Walpole

31.3k33052

Thanks for your help @Thomas , actually I just wanna scrape data from website then get about like 10 samples from the list of data I get. It was fine before by just using execute_script then i got the things i want. Unfortunately, right now with the lazy loading, it must have been scrolled over and over from the top to bottom until the JS script finish then i could have the whole data i want. Using your solution is not applicable in this case cause the selector i wanna test is one of the attribute like data-test-id = ... (which has been set for testing purpose), was not the fixed class/id

– Dinh Luong
Dec 3 '18 at 15:35

Beside that, i managed to use the execute_script to get the data i want. However, since it was async, so before the JS finished (~ 40 secs) , it has the data that i use for assertion, which triggered the next step to run when it wasn't really finishing. So i wonder if there's something that i could use to implicit wait at this stage or even overriding the default waiting time of Capybara just for waiting for JS script running to be finished ? Thanks

– Dinh Luong
Dec 3 '18 at 15:42

@DinhLuong it would still apply, you just need to change the selector to something that would match the last item in the list - #list > li:last-child, etc - However since you’re scraping execute_script is easier and just as valid.

– Thomas Walpole
Dec 3 '18 at 15:46

add a comment |

<ul id="my_list_of_items">

  <li class="item"></li>

  <li class="item"></li>

  ...

  <li class="item"></li>

</ul>

and the next group of items are loaded into the page when the bottom of the list is scrolled into view then do something like

find('.item:last-child').hover

This will cause the last item element on the page to be scrolled into view and the move pointer moved over it, which would then trigger loading of more items.

answered Nov 25 '18 at 19:33

Thomas Walpole

31.3k33052

<ul id="my_list_of_items">

  <li class="item"></li>

  <li class="item"></li>

  ...

  <li class="item"></li>

</ul>

and the next group of items are loaded into the page when the bottom of the list is scrolled into view then do something like

find('.item:last-child').hover

This will cause the last item element on the page to be scrolled into view and the move pointer moved over it, which would then trigger loading of more items.

answered Nov 25 '18 at 19:33

Thomas Walpole

31.3k33052

answered Nov 25 '18 at 19:33

Thomas Walpole

31.3k33052

answered Nov 25 '18 at 19:33

Thomas Walpole

31.3k33052

answered Nov 25 '18 at 19:33

Thomas Walpole

31.3k33052

Thanks for your help @Thomas , actually I just wanna scrape data from website then get about like 10 samples from the list of data I get. It was fine before by just using execute_script then i got the things i want. Unfortunately, right now with the lazy loading, it must have been scrolled over and over from the top to bottom until the JS script finish then i could have the whole data i want. Using your solution is not applicable in this case cause the selector i wanna test is one of the attribute like data-test-id = ... (which has been set for testing purpose), was not the fixed class/id

– Dinh Luong
Dec 3 '18 at 15:35

Beside that, i managed to use the execute_script to get the data i want. However, since it was async, so before the JS finished (~ 40 secs) , it has the data that i use for assertion, which triggered the next step to run when it wasn't really finishing. So i wonder if there's something that i could use to implicit wait at this stage or even overriding the default waiting time of Capybara just for waiting for JS script running to be finished ? Thanks

– Dinh Luong
Dec 3 '18 at 15:42

@DinhLuong it would still apply, you just need to change the selector to something that would match the last item in the list - #list > li:last-child, etc - However since you’re scraping execute_script is easier and just as valid.

– Thomas Walpole
Dec 3 '18 at 15:46

add a comment |

Thanks for your help @Thomas , actually I just wanna scrape data from website then get about like 10 samples from the list of data I get. It was fine before by just using execute_script then i got the things i want. Unfortunately, right now with the lazy loading, it must have been scrolled over and over from the top to bottom until the JS script finish then i could have the whole data i want. Using your solution is not applicable in this case cause the selector i wanna test is one of the attribute like data-test-id = ... (which has been set for testing purpose), was not the fixed class/id

– Dinh Luong
Dec 3 '18 at 15:35

Beside that, i managed to use the execute_script to get the data i want. However, since it was async, so before the JS finished (~ 40 secs) , it has the data that i use for assertion, which triggered the next step to run when it wasn't really finishing. So i wonder if there's something that i could use to implicit wait at this stage or even overriding the default waiting time of Capybara just for waiting for JS script running to be finished ? Thanks

– Dinh Luong
Dec 3 '18 at 15:42

@DinhLuong it would still apply, you just need to change the selector to something that would match the last item in the list - #list > li:last-child, etc - However since you’re scraping execute_script is easier and just as valid.

– Thomas Walpole
Dec 3 '18 at 15:46

Thanks for your help @Thomas , actually I just wanna scrape data from website then get about like 10 samples from the list of data I get. It was fine before by just using execute_script then i got the things i want. Unfortunately, right now with the lazy loading, it must have been scrolled over and over from the top to bottom until the JS script finish then i could have the whole data i want. Using your solution is not applicable in this case cause the selector i wanna test is one of the attribute like data-test-id = ... (which has been set for testing purpose), was not the fixed class/id

– Dinh Luong
Dec 3 '18 at 15:35

Beside that, i managed to use the execute_script to get the data i want. However, since it was async, so before the JS finished (~ 40 secs) , it has the data that i use for assertion, which triggered the next step to run when it wasn't really finishing. So i wonder if there's something that i could use to implicit wait at this stage or even overriding the default waiting time of Capybara just for waiting for JS script running to be finished ? Thanks

– Dinh Luong
Dec 3 '18 at 15:42

@DinhLuong it would still apply, you just need to change the selector to something that would match the last item in the list - #list > li:last-child, etc - However since you’re scraping execute_script is easier and just as valid.

– Thomas Walpole
Dec 3 '18 at 15:46

add a comment |

draft saved

draft discarded

Thanks for contributing an answer to Stack Overflow!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

搜尋此網誌

Ytukyg