Mixing
How to pass a web element into the BeautifulSoup
I am getting web element like this
elements = browser.find_elements_by_xpath("//*[contains(text(), 'Open Until')]")
Now I have to pass this element to soup to find it next & previous sibling. I am trying this
soup = BeautifulSoup(elements,'html.parser')
What should i write
??? soup = BeautifulSoup(elements.source,'html.parser') ???
Please Suggest
web-scraping beautifulsoup request
asked Nov 19 '18 at 14:30
aftab qaisrani
194
add a comment |
I am getting web element like this
elements = browser.find_elements_by_xpath("//*[contains(text(), 'Open Until')]")
Now I have to pass this element to soup to find it next & previous sibling. I am trying this
soup = BeautifulSoup(elements,'html.parser')
What should i write
??? soup = BeautifulSoup(elements.source,'html.parser') ???
Please Suggest
web-scraping beautifulsoup request
asked Nov 19 '18 at 14:30
aftab qaisrani
194
add a comment |
I am getting web element like this
elements = browser.find_elements_by_xpath("//*[contains(text(), 'Open Until')]")
Now I have to pass this element to soup to find it next & previous sibling. I am trying this
soup = BeautifulSoup(elements,'html.parser')
What should i write
??? soup = BeautifulSoup(elements.source,'html.parser') ???
Please Suggest
web-scraping beautifulsoup request
asked Nov 19 '18 at 14:30
aftab qaisrani
194
I am getting web element like this
elements = browser.find_elements_by_xpath("//*[contains(text(), 'Open Until')]")
Now I have to pass this element to soup to find it next & previous sibling. I am trying this
soup = BeautifulSoup(elements,'html.parser')
What should i write
??? soup = BeautifulSoup(elements.source,'html.parser') ???
Please Suggest
web-scraping beautifulsoup request
web-scraping beautifulsoup request
asked Nov 19 '18 at 14:30
aftab qaisrani
194
asked Nov 19 '18 at 14:30
aftab qaisrani
194
asked Nov 19 '18 at 14:30
aftab qaisrani
194
asked Nov 19 '18 at 14:30
aftab qaisrani
194
asked Nov 19 '18 at 14:30
aftab qaisrani
194
194
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
This should be a comment, but I am not able to add one.
So it should be:
soup = BeautifulSoup(elements.parent.page_source,'html.parser')
or
create soup directly from browser:
soup = BeautifulSoup(browser.page_source,'html.parser')
and then search and get your elements in soup
There is no information about this in https://selenium-python.readthedocs.io/locating-elements.html or https://saucelabs.com/resources/articles/selenium-tips-css-selectors
answered Nov 19 '18 at 14:49
Sonu Panchal
132
I am getting the parent element through , rather i need it previous & next sibling. Can you please tell about the specific solution.
– aftab qaisrani
Nov 20 '18 at 5:11
add a comment |
you don't need to mix it and you can't, selenium also has method to get prev and next sibling, example
from selenium import webdriver
driver = webdriver.Chrome()
driver.get('https://140f670e-5774-43b5-a1a5-c993f66fa51d.htmlpasta.com/')
element = driver.find_element_by_xpath("//*[contains(text(), 'Open Until')]")
prevSibling = element.find_element_by_xpath('.//preceding-sibling::*')
nextSibling = element.find_element_by_xpath('.//following-sibling::*')
print(prevSibling.tag_name + ': ' + prevSibling.text)
print(element.tag_name + ': ' + element.text)
print(nextSibling.tag_name + ': ' + nextSibling.text)
driver.close()
answered Nov 19 '18 at 16:10
ewwink
10.3k22236
I couldn't understand your answer. Can you please clarify it. I have both <p> elements above & below the "Open Until" element. What should i exactly write in x_Path.
– aftab qaisrani
Nov 20 '18 at 5:08
prevSibling = element.find_element_by_xpath('.//preceding-sibling::p') nextSibling = element.find_element_by_xpath('.//following-sibling::p') I have written in this way but unable to find it.
– aftab qaisrani
Nov 20 '18 at 5:18
post your html snippet
– ewwink
Nov 20 '18 at 7:54
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53376789%2fhow-to-pass-a-web-element-into-the-beautifulsoup%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
This should be a comment, but I am not able to add one.
So it should be:
soup = BeautifulSoup(elements.parent.page_source,'html.parser')
or
create soup directly from browser:
soup = BeautifulSoup(browser.page_source,'html.parser')
and then search and get your elements in soup
There is no information about this in https://selenium-python.readthedocs.io/locating-elements.html or https://saucelabs.com/resources/articles/selenium-tips-css-selectors
answered Nov 19 '18 at 14:49
Sonu Panchal
132
I am getting the parent element through , rather i need it previous & next sibling. Can you please tell about the specific solution.
– aftab qaisrani
Nov 20 '18 at 5:11
add a comment |
This should be a comment, but I am not able to add one.
So it should be:
soup = BeautifulSoup(elements.parent.page_source,'html.parser')
or
create soup directly from browser:
soup = BeautifulSoup(browser.page_source,'html.parser')
and then search and get your elements in soup
There is no information about this in https://selenium-python.readthedocs.io/locating-elements.html or https://saucelabs.com/resources/articles/selenium-tips-css-selectors
answered Nov 19 '18 at 14:49
Sonu Panchal
132
I am getting the parent element through , rather i need it previous & next sibling. Can you please tell about the specific solution.
– aftab qaisrani
Nov 20 '18 at 5:11
add a comment |
This should be a comment, but I am not able to add one.
So it should be:
soup = BeautifulSoup(elements.parent.page_source,'html.parser')
or
create soup directly from browser:
soup = BeautifulSoup(browser.page_source,'html.parser')
and then search and get your elements in soup
There is no information about this in https://selenium-python.readthedocs.io/locating-elements.html or https://saucelabs.com/resources/articles/selenium-tips-css-selectors
answered Nov 19 '18 at 14:49
Sonu Panchal
132
This should be a comment, but I am not able to add one.
So it should be:
soup = BeautifulSoup(elements.parent.page_source,'html.parser')
or
create soup directly from browser:
soup = BeautifulSoup(browser.page_source,'html.parser')
and then search and get your elements in soup
There is no information about this in https://selenium-python.readthedocs.io/locating-elements.html or https://saucelabs.com/resources/articles/selenium-tips-css-selectors
answered Nov 19 '18 at 14:49
Sonu Panchal
132
answered Nov 19 '18 at 14:49
Sonu Panchal
132
answered Nov 19 '18 at 14:49
Sonu Panchal
132
answered Nov 19 '18 at 14:49
Sonu Panchal
132
132
I am getting the parent element through , rather i need it previous & next sibling. Can you please tell about the specific solution.
– aftab qaisrani
Nov 20 '18 at 5:11
add a comment |
I am getting the parent element through , rather i need it previous & next sibling. Can you please tell about the specific solution.
– aftab qaisrani
Nov 20 '18 at 5:11
I am getting the parent element through , rather i need it previous & next sibling. Can you please tell about the specific solution.
– aftab qaisrani
Nov 20 '18 at 5:11
I am getting the parent element through , rather i need it previous & next sibling. Can you please tell about the specific solution.
– aftab qaisrani
Nov 20 '18 at 5:11
add a comment |
you don't need to mix it and you can't, selenium also has method to get prev and next sibling, example
from selenium import webdriver
driver = webdriver.Chrome()
driver.get('https://140f670e-5774-43b5-a1a5-c993f66fa51d.htmlpasta.com/')
element = driver.find_element_by_xpath("//*[contains(text(), 'Open Until')]")
prevSibling = element.find_element_by_xpath('.//preceding-sibling::*')
nextSibling = element.find_element_by_xpath('.//following-sibling::*')
print(prevSibling.tag_name + ': ' + prevSibling.text)
print(element.tag_name + ': ' + element.text)
print(nextSibling.tag_name + ': ' + nextSibling.text)
driver.close()
answered Nov 19 '18 at 16:10
ewwink
10.3k22236
I couldn't understand your answer. Can you please clarify it. I have both <p> elements above & below the "Open Until" element. What should i exactly write in x_Path.
– aftab qaisrani
Nov 20 '18 at 5:08
prevSibling = element.find_element_by_xpath('.//preceding-sibling::p') nextSibling = element.find_element_by_xpath('.//following-sibling::p') I have written in this way but unable to find it.
– aftab qaisrani
Nov 20 '18 at 5:18
post your html snippet
– ewwink
Nov 20 '18 at 7:54
add a comment |
you don't need to mix it and you can't, selenium also has method to get prev and next sibling, example
from selenium import webdriver
driver = webdriver.Chrome()
driver.get('https://140f670e-5774-43b5-a1a5-c993f66fa51d.htmlpasta.com/')
element = driver.find_element_by_xpath("//*[contains(text(), 'Open Until')]")
prevSibling = element.find_element_by_xpath('.//preceding-sibling::*')
nextSibling = element.find_element_by_xpath('.//following-sibling::*')
print(prevSibling.tag_name + ': ' + prevSibling.text)
print(element.tag_name + ': ' + element.text)
print(nextSibling.tag_name + ': ' + nextSibling.text)
driver.close()
answered Nov 19 '18 at 16:10
ewwink
10.3k22236
I couldn't understand your answer. Can you please clarify it. I have both <p> elements above & below the "Open Until" element. What should i exactly write in x_Path.
– aftab qaisrani
Nov 20 '18 at 5:08
prevSibling = element.find_element_by_xpath('.//preceding-sibling::p') nextSibling = element.find_element_by_xpath('.//following-sibling::p') I have written in this way but unable to find it.
– aftab qaisrani
Nov 20 '18 at 5:18
post your html snippet
– ewwink
Nov 20 '18 at 7:54
add a comment |
you don't need to mix it and you can't, selenium also has method to get prev and next sibling, example
from selenium import webdriver
driver = webdriver.Chrome()
driver.get('https://140f670e-5774-43b5-a1a5-c993f66fa51d.htmlpasta.com/')
element = driver.find_element_by_xpath("//*[contains(text(), 'Open Until')]")
prevSibling = element.find_element_by_xpath('.//preceding-sibling::*')
nextSibling = element.find_element_by_xpath('.//following-sibling::*')
print(prevSibling.tag_name + ': ' + prevSibling.text)
print(element.tag_name + ': ' + element.text)
print(nextSibling.tag_name + ': ' + nextSibling.text)
driver.close()
answered Nov 19 '18 at 16:10
ewwink
10.3k22236
you don't need to mix it and you can't, selenium also has method to get prev and next sibling, example
from selenium import webdriver
driver = webdriver.Chrome()
driver.get('https://140f670e-5774-43b5-a1a5-c993f66fa51d.htmlpasta.com/')
element = driver.find_element_by_xpath("//*[contains(text(), 'Open Until')]")
prevSibling = element.find_element_by_xpath('.//preceding-sibling::*')
nextSibling = element.find_element_by_xpath('.//following-sibling::*')
print(prevSibling.tag_name + ': ' + prevSibling.text)
print(element.tag_name + ': ' + element.text)
print(nextSibling.tag_name + ': ' + nextSibling.text)
driver.close()
answered Nov 19 '18 at 16:10
ewwink
10.3k22236
answered Nov 19 '18 at 16:10
ewwink
10.3k22236
answered Nov 19 '18 at 16:10
ewwink
10.3k22236
answered Nov 19 '18 at 16:10
ewwink
10.3k22236
10.3k22236
I couldn't understand your answer. Can you please clarify it. I have both <p> elements above & below the "Open Until" element. What should i exactly write in x_Path.
– aftab qaisrani
Nov 20 '18 at 5:08
prevSibling = element.find_element_by_xpath('.//preceding-sibling::p') nextSibling = element.find_element_by_xpath('.//following-sibling::p') I have written in this way but unable to find it.
– aftab qaisrani
Nov 20 '18 at 5:18
post your html snippet
– ewwink
Nov 20 '18 at 7:54
add a comment |
I couldn't understand your answer. Can you please clarify it. I have both <p> elements above & below the "Open Until" element. What should i exactly write in x_Path.
– aftab qaisrani
Nov 20 '18 at 5:08
prevSibling = element.find_element_by_xpath('.//preceding-sibling::p') nextSibling = element.find_element_by_xpath('.//following-sibling::p') I have written in this way but unable to find it.
– aftab qaisrani
Nov 20 '18 at 5:18
post your html snippet
– ewwink
Nov 20 '18 at 7:54
I couldn't understand your answer. Can you please clarify it. I have both <p> elements above & below the "Open Until" element. What should i exactly write in x_Path.
– aftab qaisrani
Nov 20 '18 at 5:08
I couldn't understand your answer. Can you please clarify it. I have both <p> elements above & below the "Open Until" element. What should i exactly write in x_Path.
– aftab qaisrani
Nov 20 '18 at 5:08
prevSibling = element.find_element_by_xpath('.//preceding-sibling::p') nextSibling = element.find_element_by_xpath('.//following-sibling::p') I have written in this way but unable to find it.
– aftab qaisrani
Nov 20 '18 at 5:18
prevSibling = element.find_element_by_xpath('.//preceding-sibling::p') nextSibling = element.find_element_by_xpath('.//following-sibling::p') I have written in this way but unable to find it.
– aftab qaisrani
Nov 20 '18 at 5:18
post your html snippet
– ewwink
Nov 20 '18 at 7:54
post your html snippet
– ewwink
Nov 20 '18 at 7:54
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53376789%2fhow-to-pass-a-web-element-into-the-beautifulsoup%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
