Mixing
Scrape data with load more button from ratemyprofessor
I encountered a problem when I tried to scrape the review text from ratemyprofessor.com (http://www.ratemyprofessors.com/ShowRatings.jsp?tid=860968#). I am currently using BeautifulSoup and requests.
I would love to get all the review contents, while the data after clicking "Load More" is inaccessible. I have tried different ways that have been posted on StackOverflow and Reddit, unfortunately, none of them works for me.
The load more button under inspection: onclick="javascript:mtvn.btg.Controller.sendLinkEvent({ linkName:'PROF:LoadMore', linkType:'o' } );"
I would greatly appreciate if anyone could help me with this problem. Thank you.
python web-scraping beautifulsoup
asked Nov 19 '18 at 21:15
ChloeChloe
32
add a comment |
I encountered a problem when I tried to scrape the review text from ratemyprofessor.com (http://www.ratemyprofessors.com/ShowRatings.jsp?tid=860968#). I am currently using BeautifulSoup and requests.
I would love to get all the review contents, while the data after clicking "Load More" is inaccessible. I have tried different ways that have been posted on StackOverflow and Reddit, unfortunately, none of them works for me.
The load more button under inspection: onclick="javascript:mtvn.btg.Controller.sendLinkEvent({ linkName:'PROF:LoadMore', linkType:'o' } );"
I would greatly appreciate if anyone could help me with this problem. Thank you.
python web-scraping beautifulsoup
asked Nov 19 '18 at 21:15
ChloeChloe
32
Is that data JS generated?
– SuperStew
Nov 19 '18 at 21:15
@SuperStew I'm not sure... This is in the element inspect : onclick="javascript:mtvn.btg.Controller.sendLinkEvent({ linkName:'PROF:LoadMore', linkType:'o' } );"
– Chloe
Nov 19 '18 at 21:49
I'm afraid this violates the terms of service: ratemyprofessors.com/TermsOfUse_us.jsp#section6 unless you have prior permission.
– QHarr
Nov 19 '18 at 22:22
add a comment |
I encountered a problem when I tried to scrape the review text from ratemyprofessor.com (http://www.ratemyprofessors.com/ShowRatings.jsp?tid=860968#). I am currently using BeautifulSoup and requests.
I would love to get all the review contents, while the data after clicking "Load More" is inaccessible. I have tried different ways that have been posted on StackOverflow and Reddit, unfortunately, none of them works for me.
The load more button under inspection: onclick="javascript:mtvn.btg.Controller.sendLinkEvent({ linkName:'PROF:LoadMore', linkType:'o' } );"
I would greatly appreciate if anyone could help me with this problem. Thank you.
python web-scraping beautifulsoup
asked Nov 19 '18 at 21:15
ChloeChloe
32
I encountered a problem when I tried to scrape the review text from ratemyprofessor.com (http://www.ratemyprofessors.com/ShowRatings.jsp?tid=860968#). I am currently using BeautifulSoup and requests.
I would love to get all the review contents, while the data after clicking "Load More" is inaccessible. I have tried different ways that have been posted on StackOverflow and Reddit, unfortunately, none of them works for me.
The load more button under inspection: onclick="javascript:mtvn.btg.Controller.sendLinkEvent({ linkName:'PROF:LoadMore', linkType:'o' } );"
I would greatly appreciate if anyone could help me with this problem. Thank you.
python web-scraping beautifulsoup
python web-scraping beautifulsoup
asked Nov 19 '18 at 21:15
ChloeChloe
32
asked Nov 19 '18 at 21:15
ChloeChloe
32
edited Nov 19 '18 at 21:50
Chloe
asked Nov 19 '18 at 21:15
ChloeChloe
32
asked Nov 19 '18 at 21:15
ChloeChloe
32
asked Nov 19 '18 at 21:15
ChloeChloe
32
32
Is that data JS generated?
– SuperStew
Nov 19 '18 at 21:15
@SuperStew I'm not sure... This is in the element inspect : onclick="javascript:mtvn.btg.Controller.sendLinkEvent({ linkName:'PROF:LoadMore', linkType:'o' } );"
– Chloe
Nov 19 '18 at 21:49
I'm afraid this violates the terms of service: ratemyprofessors.com/TermsOfUse_us.jsp#section6 unless you have prior permission.
– QHarr
Nov 19 '18 at 22:22
add a comment |
Is that data JS generated?
– SuperStew
Nov 19 '18 at 21:15
@SuperStew I'm not sure... This is in the element inspect : onclick="javascript:mtvn.btg.Controller.sendLinkEvent({ linkName:'PROF:LoadMore', linkType:'o' } );"
– Chloe
Nov 19 '18 at 21:49
I'm afraid this violates the terms of service: ratemyprofessors.com/TermsOfUse_us.jsp#section6 unless you have prior permission.
– QHarr
Nov 19 '18 at 22:22
Is that data JS generated?
– SuperStew
Nov 19 '18 at 21:15
Is that data JS generated?
– SuperStew
Nov 19 '18 at 21:15
@SuperStew I'm not sure... This is in the element inspect : onclick="javascript:mtvn.btg.Controller.sendLinkEvent({ linkName:'PROF:LoadMore', linkType:'o' } );"
– Chloe
Nov 19 '18 at 21:49
@SuperStew I'm not sure... This is in the element inspect : onclick="javascript:mtvn.btg.Controller.sendLinkEvent({ linkName:'PROF:LoadMore', linkType:'o' } );"
– Chloe
Nov 19 '18 at 21:49
I'm afraid this violates the terms of service: ratemyprofessors.com/TermsOfUse_us.jsp#section6 unless you have prior permission.
– QHarr
Nov 19 '18 at 22:22
I'm afraid this violates the terms of service: ratemyprofessors.com/TermsOfUse_us.jsp#section6 unless you have prior permission.
– QHarr
Nov 19 '18 at 22:22
add a comment |
2 Answers
2
active
oldest
votes
You need to use the chrome network tab so see what request is made when you click load more.
In this case it's:
http://www.ratemyprofessors.com/paginate/professors/ratings?tid=860968&filter=&courseCode=&page=2
answered Nov 19 '18 at 22:55
pguardiariopguardiario
36k979114
Thank you so much!!! But I still don't know how to get these data...
– Chloe
Nov 21 '18 at 6:30
Just make the request and load the response with lxml/bs4 or whatever other html parser you use,.
– pguardiario
Nov 21 '18 at 6:49
I am using the following codes. I managed to load the page, but did not get the contents in your link. :read_mores = driver.find_elements_by_xpath('//*[@data-teach-id='+ tid + ']') for read_more in read_mores: driver.execute_script("arguments[0].scrollIntoView();", read_more) driver.execute_script("$(arguments[0]).click();", read_more) soup = BeautifulSoup(driver.page_source, 'html.parser')
– Chloe
Nov 21 '18 at 6:54
Are you usingselenium? You tagged your question withbeautiful soup.
– pguardiario
Nov 21 '18 at 6:56
Oh yeah. I didn't know how to load pages with BS, therefore I used Selenium to load pages and use BS to parse it. Is there any way I can do it with BS only? I am very new to this area and have a lot of questions. Thanks for your patience!
– Chloe
Nov 21 '18 at 7:01
|
show 2 more comments
This appears to a JS website. I think you'll need to use something like Selenium to scrape this. By using Selenium you could direct the web browser to scroll to the end and capture all the data you are looking for that way.
answered Nov 19 '18 at 21:29
Owais ArshadOwais Arshad
7519
Thanks a lot! I have tried to use use the function "location_once_scrolled_into_view", but it raises the error: Element <a id="myProfloadMore" class="content" href="#"> could not be scrolled into view
– Chloe
Nov 19 '18 at 21:40
Try something like this: stackoverflow.com/questions/33094727/…
– Owais Arshad
Nov 20 '18 at 3:28
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53382746%2fscrape-data-with-load-more-button-from-ratemyprofessor%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
You need to use the chrome network tab so see what request is made when you click load more.
In this case it's:
http://www.ratemyprofessors.com/paginate/professors/ratings?tid=860968&filter=&courseCode=&page=2
answered Nov 19 '18 at 22:55
pguardiariopguardiario
36k979114
Thank you so much!!! But I still don't know how to get these data...
– Chloe
Nov 21 '18 at 6:30
Just make the request and load the response with lxml/bs4 or whatever other html parser you use,.
– pguardiario
Nov 21 '18 at 6:49
I am using the following codes. I managed to load the page, but did not get the contents in your link. :read_mores = driver.find_elements_by_xpath('//*[@data-teach-id='+ tid + ']') for read_more in read_mores: driver.execute_script("arguments[0].scrollIntoView();", read_more) driver.execute_script("$(arguments[0]).click();", read_more) soup = BeautifulSoup(driver.page_source, 'html.parser')
– Chloe
Nov 21 '18 at 6:54
Are you usingselenium? You tagged your question withbeautiful soup.
– pguardiario
Nov 21 '18 at 6:56
Oh yeah. I didn't know how to load pages with BS, therefore I used Selenium to load pages and use BS to parse it. Is there any way I can do it with BS only? I am very new to this area and have a lot of questions. Thanks for your patience!
– Chloe
Nov 21 '18 at 7:01
|
show 2 more comments
You need to use the chrome network tab so see what request is made when you click load more.
In this case it's:
http://www.ratemyprofessors.com/paginate/professors/ratings?tid=860968&filter=&courseCode=&page=2
answered Nov 19 '18 at 22:55
pguardiariopguardiario
36k979114
Thank you so much!!! But I still don't know how to get these data...
– Chloe
Nov 21 '18 at 6:30
Just make the request and load the response with lxml/bs4 or whatever other html parser you use,.
– pguardiario
Nov 21 '18 at 6:49
I am using the following codes. I managed to load the page, but did not get the contents in your link. :read_mores = driver.find_elements_by_xpath('//*[@data-teach-id='+ tid + ']') for read_more in read_mores: driver.execute_script("arguments[0].scrollIntoView();", read_more) driver.execute_script("$(arguments[0]).click();", read_more) soup = BeautifulSoup(driver.page_source, 'html.parser')
– Chloe
Nov 21 '18 at 6:54
Are you usingselenium? You tagged your question withbeautiful soup.
– pguardiario
Nov 21 '18 at 6:56
Oh yeah. I didn't know how to load pages with BS, therefore I used Selenium to load pages and use BS to parse it. Is there any way I can do it with BS only? I am very new to this area and have a lot of questions. Thanks for your patience!
– Chloe
Nov 21 '18 at 7:01
|
show 2 more comments
You need to use the chrome network tab so see what request is made when you click load more.
In this case it's:
http://www.ratemyprofessors.com/paginate/professors/ratings?tid=860968&filter=&courseCode=&page=2
answered Nov 19 '18 at 22:55
pguardiariopguardiario
36k979114
You need to use the chrome network tab so see what request is made when you click load more.
In this case it's:
http://www.ratemyprofessors.com/paginate/professors/ratings?tid=860968&filter=&courseCode=&page=2
answered Nov 19 '18 at 22:55
pguardiariopguardiario
36k979114
answered Nov 19 '18 at 22:55
pguardiariopguardiario
36k979114
answered Nov 19 '18 at 22:55
pguardiariopguardiario
36k979114
answered Nov 19 '18 at 22:55
pguardiariopguardiario
36k979114
36k979114
Thank you so much!!! But I still don't know how to get these data...
– Chloe
Nov 21 '18 at 6:30
Just make the request and load the response with lxml/bs4 or whatever other html parser you use,.
– pguardiario
Nov 21 '18 at 6:49
I am using the following codes. I managed to load the page, but did not get the contents in your link. :read_mores = driver.find_elements_by_xpath('//*[@data-teach-id='+ tid + ']') for read_more in read_mores: driver.execute_script("arguments[0].scrollIntoView();", read_more) driver.execute_script("$(arguments[0]).click();", read_more) soup = BeautifulSoup(driver.page_source, 'html.parser')
– Chloe
Nov 21 '18 at 6:54
Are you usingselenium? You tagged your question withbeautiful soup.
– pguardiario
Nov 21 '18 at 6:56
Oh yeah. I didn't know how to load pages with BS, therefore I used Selenium to load pages and use BS to parse it. Is there any way I can do it with BS only? I am very new to this area and have a lot of questions. Thanks for your patience!
– Chloe
Nov 21 '18 at 7:01
|
show 2 more comments
Thank you so much!!! But I still don't know how to get these data...
– Chloe
Nov 21 '18 at 6:30
Just make the request and load the response with lxml/bs4 or whatever other html parser you use,.
– pguardiario
Nov 21 '18 at 6:49
I am using the following codes. I managed to load the page, but did not get the contents in your link. :read_mores = driver.find_elements_by_xpath('//*[@data-teach-id='+ tid + ']') for read_more in read_mores: driver.execute_script("arguments[0].scrollIntoView();", read_more) driver.execute_script("$(arguments[0]).click();", read_more) soup = BeautifulSoup(driver.page_source, 'html.parser')
– Chloe
Nov 21 '18 at 6:54
Are you usingselenium? You tagged your question withbeautiful soup.
– pguardiario
Nov 21 '18 at 6:56
Oh yeah. I didn't know how to load pages with BS, therefore I used Selenium to load pages and use BS to parse it. Is there any way I can do it with BS only? I am very new to this area and have a lot of questions. Thanks for your patience!
– Chloe
Nov 21 '18 at 7:01
Thank you so much!!! But I still don't know how to get these data...
– Chloe
Nov 21 '18 at 6:30
Thank you so much!!! But I still don't know how to get these data...
– Chloe
Nov 21 '18 at 6:30
Just make the request and load the response with lxml/bs4 or whatever other html parser you use,.
– pguardiario
Nov 21 '18 at 6:49
Just make the request and load the response with lxml/bs4 or whatever other html parser you use,.
– pguardiario
Nov 21 '18 at 6:49
I am using the following codes. I managed to load the page, but did not get the contents in your link. :
read_mores = driver.find_elements_by_xpath('//*[@data-teach-id='+ tid + ']') for read_more in read_mores: driver.execute_script("arguments[0].scrollIntoView();", read_more) driver.execute_script("$(arguments[0]).click();", read_more) soup = BeautifulSoup(driver.page_source, 'html.parser') – Chloe
Nov 21 '18 at 6:54
I am using the following codes. I managed to load the page, but did not get the contents in your link. :
read_mores = driver.find_elements_by_xpath('//*[@data-teach-id='+ tid + ']') for read_more in read_mores: driver.execute_script("arguments[0].scrollIntoView();", read_more) driver.execute_script("$(arguments[0]).click();", read_more) soup = BeautifulSoup(driver.page_source, 'html.parser') – Chloe
Nov 21 '18 at 6:54
Are you using
selenium? You tagged your question with beautiful soup.– pguardiario
Nov 21 '18 at 6:56
Are you using
selenium? You tagged your question with beautiful soup.– pguardiario
Nov 21 '18 at 6:56
Oh yeah. I didn't know how to load pages with BS, therefore I used Selenium to load pages and use BS to parse it. Is there any way I can do it with BS only? I am very new to this area and have a lot of questions. Thanks for your patience!
– Chloe
Nov 21 '18 at 7:01
Oh yeah. I didn't know how to load pages with BS, therefore I used Selenium to load pages and use BS to parse it. Is there any way I can do it with BS only? I am very new to this area and have a lot of questions. Thanks for your patience!
– Chloe
Nov 21 '18 at 7:01
|
show 2 more comments
This appears to a JS website. I think you'll need to use something like Selenium to scrape this. By using Selenium you could direct the web browser to scroll to the end and capture all the data you are looking for that way.
answered Nov 19 '18 at 21:29
Owais ArshadOwais Arshad
7519
Thanks a lot! I have tried to use use the function "location_once_scrolled_into_view", but it raises the error: Element <a id="myProfloadMore" class="content" href="#"> could not be scrolled into view
– Chloe
Nov 19 '18 at 21:40
Try something like this: stackoverflow.com/questions/33094727/…
– Owais Arshad
Nov 20 '18 at 3:28
add a comment |
This appears to a JS website. I think you'll need to use something like Selenium to scrape this. By using Selenium you could direct the web browser to scroll to the end and capture all the data you are looking for that way.
answered Nov 19 '18 at 21:29
Owais ArshadOwais Arshad
7519
Thanks a lot! I have tried to use use the function "location_once_scrolled_into_view", but it raises the error: Element <a id="myProfloadMore" class="content" href="#"> could not be scrolled into view
– Chloe
Nov 19 '18 at 21:40
Try something like this: stackoverflow.com/questions/33094727/…
– Owais Arshad
Nov 20 '18 at 3:28
add a comment |
This appears to a JS website. I think you'll need to use something like Selenium to scrape this. By using Selenium you could direct the web browser to scroll to the end and capture all the data you are looking for that way.
answered Nov 19 '18 at 21:29
Owais ArshadOwais Arshad
7519
This appears to a JS website. I think you'll need to use something like Selenium to scrape this. By using Selenium you could direct the web browser to scroll to the end and capture all the data you are looking for that way.
answered Nov 19 '18 at 21:29
Owais ArshadOwais Arshad
7519
answered Nov 19 '18 at 21:29
Owais ArshadOwais Arshad
7519
answered Nov 19 '18 at 21:29
Owais ArshadOwais Arshad
7519
answered Nov 19 '18 at 21:29
Owais ArshadOwais Arshad
7519
7519
Thanks a lot! I have tried to use use the function "location_once_scrolled_into_view", but it raises the error: Element <a id="myProfloadMore" class="content" href="#"> could not be scrolled into view
– Chloe
Nov 19 '18 at 21:40
Try something like this: stackoverflow.com/questions/33094727/…
– Owais Arshad
Nov 20 '18 at 3:28
add a comment |
Thanks a lot! I have tried to use use the function "location_once_scrolled_into_view", but it raises the error: Element <a id="myProfloadMore" class="content" href="#"> could not be scrolled into view
– Chloe
Nov 19 '18 at 21:40
Try something like this: stackoverflow.com/questions/33094727/…
– Owais Arshad
Nov 20 '18 at 3:28
Thanks a lot! I have tried to use use the function "location_once_scrolled_into_view", but it raises the error: Element <a id="myProfloadMore" class="content" href="#"> could not be scrolled into view
– Chloe
Nov 19 '18 at 21:40
Thanks a lot! I have tried to use use the function "location_once_scrolled_into_view", but it raises the error: Element <a id="myProfloadMore" class="content" href="#"> could not be scrolled into view
– Chloe
Nov 19 '18 at 21:40
Try something like this: stackoverflow.com/questions/33094727/…
– Owais Arshad
Nov 20 '18 at 3:28
Try something like this: stackoverflow.com/questions/33094727/…
– Owais Arshad
Nov 20 '18 at 3:28
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53382746%2fscrape-data-with-load-more-button-from-ratemyprofessor%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Is that data JS generated?
– SuperStew
Nov 19 '18 at 21:15
@SuperStew I'm not sure... This is in the element inspect : onclick="javascript:mtvn.btg.Controller.sendLinkEvent({ linkName:'PROF:LoadMore', linkType:'o' } );"
– Chloe
Nov 19 '18 at 21:49
I'm afraid this violates the terms of service: ratemyprofessors.com/TermsOfUse_us.jsp#section6 unless you have prior permission.
– QHarr
Nov 19 '18 at 22:22