Mixing
Clicking multiple items on one page using selenium
My main purpose is to go to this specific website, to click each of the products, have enough time to scrape the data from the clicked product, then go back to click another product from the page until all the products are clicked through and scraped (The scraping code I have not included).
My code opens up chrome to redirect to my desired website, generates a list of links to click by class_name. This is the part I am stuck on, I would believe I need a for-loop to iterate through the list of links to click and go back to the original. But, I can't figure out why this won't work.
Here is my code:
import csv
import time
from selenium import webdriver
import selenium.webdriver.chrome.service as service
import requests
from bs4 import BeautifulSoup
url = "https://www.vatainc.com/infusion/adult-infusion.html?limit=all"
service = service.Service('path to chromedriver')
service.start()
capabilities = {'chrome.binary': 'path to chrome'}
driver = webdriver.Remote(service.service_url, capabilities)
driver.get(url)
time.sleep(2)
links = driver.find_elements_by_class_name('product-name')
for link in links:
link.click()
driver.back()
link.click()
python selenium beautifulsoup
asked Nov 20 '18 at 17:11
Jonathan Jonathan
165
add a comment |
My main purpose is to go to this specific website, to click each of the products, have enough time to scrape the data from the clicked product, then go back to click another product from the page until all the products are clicked through and scraped (The scraping code I have not included).
My code opens up chrome to redirect to my desired website, generates a list of links to click by class_name. This is the part I am stuck on, I would believe I need a for-loop to iterate through the list of links to click and go back to the original. But, I can't figure out why this won't work.
Here is my code:
import csv
import time
from selenium import webdriver
import selenium.webdriver.chrome.service as service
import requests
from bs4 import BeautifulSoup
url = "https://www.vatainc.com/infusion/adult-infusion.html?limit=all"
service = service.Service('path to chromedriver')
service.start()
capabilities = {'chrome.binary': 'path to chrome'}
driver = webdriver.Remote(service.service_url, capabilities)
driver.get(url)
time.sleep(2)
links = driver.find_elements_by_class_name('product-name')
for link in links:
link.click()
driver.back()
link.click()
python selenium beautifulsoup
asked Nov 20 '18 at 17:11
Jonathan Jonathan
165
Haven't tested your code, but there should be a mistake in the for loop, the second link.click() should not be there I guess...
– kaihami
Nov 20 '18 at 17:30
Yes, I had that before. It gives the error 'Message: stale element reference: element is not attached to the page document'. I'm not sure why though, I have multiple elements in the links variable..
– Jonathan
Nov 20 '18 at 17:33
Regarding your "stale element" error: stackoverflow.com/questions/44630912/… After the first link clicked, you don't have access to the previous page, you need to re-evaluate (find) the links to click.
– Clément Denoix
Nov 20 '18 at 17:39
@ClémentDenoix That is a good point, I am curious to how I would find the links again. I tried for link in links: link.click() driver.back() time.sleep(2) driver.find_elements_by_class_name('product-name') ...................................... But still the same error stale reference error..
– Jonathan
Nov 20 '18 at 17:56
add a comment |
My main purpose is to go to this specific website, to click each of the products, have enough time to scrape the data from the clicked product, then go back to click another product from the page until all the products are clicked through and scraped (The scraping code I have not included).
My code opens up chrome to redirect to my desired website, generates a list of links to click by class_name. This is the part I am stuck on, I would believe I need a for-loop to iterate through the list of links to click and go back to the original. But, I can't figure out why this won't work.
Here is my code:
import csv
import time
from selenium import webdriver
import selenium.webdriver.chrome.service as service
import requests
from bs4 import BeautifulSoup
url = "https://www.vatainc.com/infusion/adult-infusion.html?limit=all"
service = service.Service('path to chromedriver')
service.start()
capabilities = {'chrome.binary': 'path to chrome'}
driver = webdriver.Remote(service.service_url, capabilities)
driver.get(url)
time.sleep(2)
links = driver.find_elements_by_class_name('product-name')
for link in links:
link.click()
driver.back()
link.click()
python selenium beautifulsoup
asked Nov 20 '18 at 17:11
Jonathan Jonathan
165
My main purpose is to go to this specific website, to click each of the products, have enough time to scrape the data from the clicked product, then go back to click another product from the page until all the products are clicked through and scraped (The scraping code I have not included).
My code opens up chrome to redirect to my desired website, generates a list of links to click by class_name. This is the part I am stuck on, I would believe I need a for-loop to iterate through the list of links to click and go back to the original. But, I can't figure out why this won't work.
Here is my code:
import csv
import time
from selenium import webdriver
import selenium.webdriver.chrome.service as service
import requests
from bs4 import BeautifulSoup
url = "https://www.vatainc.com/infusion/adult-infusion.html?limit=all"
service = service.Service('path to chromedriver')
service.start()
capabilities = {'chrome.binary': 'path to chrome'}
driver = webdriver.Remote(service.service_url, capabilities)
driver.get(url)
time.sleep(2)
links = driver.find_elements_by_class_name('product-name')
for link in links:
link.click()
driver.back()
link.click()
python selenium beautifulsoup
python selenium beautifulsoup
asked Nov 20 '18 at 17:11
Jonathan Jonathan
165
asked Nov 20 '18 at 17:11
Jonathan Jonathan
165
edited Nov 20 '18 at 17:19
Jonathan
asked Nov 20 '18 at 17:11
Jonathan Jonathan
165
asked Nov 20 '18 at 17:11
Jonathan Jonathan
165
asked Nov 20 '18 at 17:11
Jonathan Jonathan
165
165
Haven't tested your code, but there should be a mistake in the for loop, the second link.click() should not be there I guess...
– kaihami
Nov 20 '18 at 17:30
Yes, I had that before. It gives the error 'Message: stale element reference: element is not attached to the page document'. I'm not sure why though, I have multiple elements in the links variable..
– Jonathan
Nov 20 '18 at 17:33
Regarding your "stale element" error: stackoverflow.com/questions/44630912/… After the first link clicked, you don't have access to the previous page, you need to re-evaluate (find) the links to click.
– Clément Denoix
Nov 20 '18 at 17:39
@ClémentDenoix That is a good point, I am curious to how I would find the links again. I tried for link in links: link.click() driver.back() time.sleep(2) driver.find_elements_by_class_name('product-name') ...................................... But still the same error stale reference error..
– Jonathan
Nov 20 '18 at 17:56
add a comment |
Haven't tested your code, but there should be a mistake in the for loop, the second link.click() should not be there I guess...
– kaihami
Nov 20 '18 at 17:30
Yes, I had that before. It gives the error 'Message: stale element reference: element is not attached to the page document'. I'm not sure why though, I have multiple elements in the links variable..
– Jonathan
Nov 20 '18 at 17:33
Regarding your "stale element" error: stackoverflow.com/questions/44630912/… After the first link clicked, you don't have access to the previous page, you need to re-evaluate (find) the links to click.
– Clément Denoix
Nov 20 '18 at 17:39
@ClémentDenoix That is a good point, I am curious to how I would find the links again. I tried for link in links: link.click() driver.back() time.sleep(2) driver.find_elements_by_class_name('product-name') ...................................... But still the same error stale reference error..
– Jonathan
Nov 20 '18 at 17:56
Haven't tested your code, but there should be a mistake in the for loop, the second link.click() should not be there I guess...
– kaihami
Nov 20 '18 at 17:30
Haven't tested your code, but there should be a mistake in the for loop, the second link.click() should not be there I guess...
– kaihami
Nov 20 '18 at 17:30
Yes, I had that before. It gives the error 'Message: stale element reference: element is not attached to the page document'. I'm not sure why though, I have multiple elements in the links variable..
– Jonathan
Nov 20 '18 at 17:33
Yes, I had that before. It gives the error 'Message: stale element reference: element is not attached to the page document'. I'm not sure why though, I have multiple elements in the links variable..
– Jonathan
Nov 20 '18 at 17:33
Regarding your "stale element" error: stackoverflow.com/questions/44630912/… After the first link clicked, you don't have access to the previous page, you need to re-evaluate (find) the links to click.
– Clément Denoix
Nov 20 '18 at 17:39
Regarding your "stale element" error: stackoverflow.com/questions/44630912/… After the first link clicked, you don't have access to the previous page, you need to re-evaluate (find) the links to click.
– Clément Denoix
Nov 20 '18 at 17:39
@ClémentDenoix That is a good point, I am curious to how I would find the links again. I tried for link in links: link.click() driver.back() time.sleep(2) driver.find_elements_by_class_name('product-name') ...................................... But still the same error stale reference error..
– Jonathan
Nov 20 '18 at 17:56
@ClémentDenoix That is a good point, I am curious to how I would find the links again. I tried for link in links: link.click() driver.back() time.sleep(2) driver.find_elements_by_class_name('product-name') ...................................... But still the same error stale reference error..
– Jonathan
Nov 20 '18 at 17:56
add a comment |
1 Answer
1
active
oldest
votes
I have another solution to your problem.
When I tested your code it showed a strange behaviour. Fixed all problems that I had using xpath.
url = "https://www.vatainc.com/infusion/adult-infusion.html?limit=all"
driver.get(url)
links = [x.get_attribute('href') for x in driver.find_elements_by_xpath("//*[contains(@class, 'product-name')]/a")]
htmls =
for link in links:
driver.get(link)
htmls.append(driver.page_source)
Instead of going back and forward I saved all links (named as links) and iterate over this list.
answered Nov 20 '18 at 18:21
kaihamikaihami
187311
Thank you, what did you define 'x' as ? I am getting an error, when trying to run.
– Jonathan
Nov 20 '18 at 18:27
Sorry, it was only for debugging. Edited my answer
– kaihami
Nov 20 '18 at 18:32
Wow, that solution, even is able to generate clicks on links that are not in display to the current page view! Thank you
– Jonathan
Nov 20 '18 at 18:36
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53398126%2fclicking-multiple-items-on-one-page-using-selenium%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
I have another solution to your problem.
When I tested your code it showed a strange behaviour. Fixed all problems that I had using xpath.
url = "https://www.vatainc.com/infusion/adult-infusion.html?limit=all"
driver.get(url)
links = [x.get_attribute('href') for x in driver.find_elements_by_xpath("//*[contains(@class, 'product-name')]/a")]
htmls =
for link in links:
driver.get(link)
htmls.append(driver.page_source)
Instead of going back and forward I saved all links (named as links) and iterate over this list.
answered Nov 20 '18 at 18:21
kaihamikaihami
187311
Thank you, what did you define 'x' as ? I am getting an error, when trying to run.
– Jonathan
Nov 20 '18 at 18:27
Sorry, it was only for debugging. Edited my answer
– kaihami
Nov 20 '18 at 18:32
Wow, that solution, even is able to generate clicks on links that are not in display to the current page view! Thank you
– Jonathan
Nov 20 '18 at 18:36
add a comment |
I have another solution to your problem.
When I tested your code it showed a strange behaviour. Fixed all problems that I had using xpath.
url = "https://www.vatainc.com/infusion/adult-infusion.html?limit=all"
driver.get(url)
links = [x.get_attribute('href') for x in driver.find_elements_by_xpath("//*[contains(@class, 'product-name')]/a")]
htmls =
for link in links:
driver.get(link)
htmls.append(driver.page_source)
Instead of going back and forward I saved all links (named as links) and iterate over this list.
answered Nov 20 '18 at 18:21
kaihamikaihami
187311
Thank you, what did you define 'x' as ? I am getting an error, when trying to run.
– Jonathan
Nov 20 '18 at 18:27
Sorry, it was only for debugging. Edited my answer
– kaihami
Nov 20 '18 at 18:32
Wow, that solution, even is able to generate clicks on links that are not in display to the current page view! Thank you
– Jonathan
Nov 20 '18 at 18:36
add a comment |
I have another solution to your problem.
When I tested your code it showed a strange behaviour. Fixed all problems that I had using xpath.
url = "https://www.vatainc.com/infusion/adult-infusion.html?limit=all"
driver.get(url)
links = [x.get_attribute('href') for x in driver.find_elements_by_xpath("//*[contains(@class, 'product-name')]/a")]
htmls =
for link in links:
driver.get(link)
htmls.append(driver.page_source)
Instead of going back and forward I saved all links (named as links) and iterate over this list.
answered Nov 20 '18 at 18:21
kaihamikaihami
187311
I have another solution to your problem.
When I tested your code it showed a strange behaviour. Fixed all problems that I had using xpath.
url = "https://www.vatainc.com/infusion/adult-infusion.html?limit=all"
driver.get(url)
links = [x.get_attribute('href') for x in driver.find_elements_by_xpath("//*[contains(@class, 'product-name')]/a")]
htmls =
for link in links:
driver.get(link)
htmls.append(driver.page_source)
Instead of going back and forward I saved all links (named as links) and iterate over this list.
answered Nov 20 '18 at 18:21
kaihamikaihami
187311
edited Nov 20 '18 at 18:31
answered Nov 20 '18 at 18:21
kaihamikaihami
187311
answered Nov 20 '18 at 18:21
kaihamikaihami
187311
answered Nov 20 '18 at 18:21
kaihamikaihami
187311
187311
Thank you, what did you define 'x' as ? I am getting an error, when trying to run.
– Jonathan
Nov 20 '18 at 18:27
Sorry, it was only for debugging. Edited my answer
– kaihami
Nov 20 '18 at 18:32
Wow, that solution, even is able to generate clicks on links that are not in display to the current page view! Thank you
– Jonathan
Nov 20 '18 at 18:36
add a comment |
Thank you, what did you define 'x' as ? I am getting an error, when trying to run.
– Jonathan
Nov 20 '18 at 18:27
Sorry, it was only for debugging. Edited my answer
– kaihami
Nov 20 '18 at 18:32
Wow, that solution, even is able to generate clicks on links that are not in display to the current page view! Thank you
– Jonathan
Nov 20 '18 at 18:36
Thank you, what did you define 'x' as ? I am getting an error, when trying to run.
– Jonathan
Nov 20 '18 at 18:27
Thank you, what did you define 'x' as ? I am getting an error, when trying to run.
– Jonathan
Nov 20 '18 at 18:27
Sorry, it was only for debugging. Edited my answer
– kaihami
Nov 20 '18 at 18:32
Sorry, it was only for debugging. Edited my answer
– kaihami
Nov 20 '18 at 18:32
Wow, that solution, even is able to generate clicks on links that are not in display to the current page view! Thank you
– Jonathan
Nov 20 '18 at 18:36
Wow, that solution, even is able to generate clicks on links that are not in display to the current page view! Thank you
– Jonathan
Nov 20 '18 at 18:36
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53398126%2fclicking-multiple-items-on-one-page-using-selenium%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Haven't tested your code, but there should be a mistake in the for loop, the second link.click() should not be there I guess...
– kaihami
Nov 20 '18 at 17:30
Yes, I had that before. It gives the error 'Message: stale element reference: element is not attached to the page document'. I'm not sure why though, I have multiple elements in the links variable..
– Jonathan
Nov 20 '18 at 17:33
Regarding your "stale element" error: stackoverflow.com/questions/44630912/… After the first link clicked, you don't have access to the previous page, you need to re-evaluate (find) the links to click.
– Clément Denoix
Nov 20 '18 at 17:39
@ClémentDenoix That is a good point, I am curious to how I would find the links again. I tried for link in links: link.click() driver.back() time.sleep(2) driver.find_elements_by_class_name('product-name') ...................................... But still the same error stale reference error..
– Jonathan
Nov 20 '18 at 17:56