How to click a random link from google search results through Selenium and Python












0















Currently using Selenium+ Python and this code:



browser.get('http://www.google.com')
search = browser.find_element_by_name('q')
search.send_keys("how to search the internet")
search.send_keys(Keys.RETURN) # hit return after you enter search text
time.sleep(5) # sleep for 5 seconds so you can see the results
browser.execute_script("window.scrollTo(0, 873)")
time.sleep(2)
browser.find_element(By.XPATH, '(//h3)[3]/a').click()`


as my code.



What Happens: Goes to Google and types in the words and hits search. Scrolls down a bit and then it clicks the first link



What I want: I want to be able to click a random link from the search result page



How to do this?



Edit: This is what I meant when I said unnecessary:
Unnecessary 1: https://imgur.com/a/70qz89x



Unnecessary 2: https://imgur.com/a/8WWvcnC



These are the only results I want:
This: https://imgur.com/a/eTatFV9










share|improve this question




















  • 1





    You'll need to collect the links from the page and then make a random choice from that list. See the random package documentation for the choice method.

    – Prune
    Aug 22 '18 at 22:57











  • Thanks for the help. I figured out how to do collect all the links from the page. However there are some links that are unnecessary. Is it possible to exclude them?

    – Kunwar Sodhi
    Aug 23 '18 at 2:39











  • If you got unnecessary links then your selector is not perfect. Use more specific selector to get rid of them

    – Andersson
    Aug 23 '18 at 4:58











  • Hi @KunwarSodhi, what exactly do you mean by unnecessary links? Do you want to edit your post and give us an example of what exactly you need - which links exactly, and we could provide a detailed solution.

    – Shlomi Bazel
    Aug 23 '18 at 8:21
















0















Currently using Selenium+ Python and this code:



browser.get('http://www.google.com')
search = browser.find_element_by_name('q')
search.send_keys("how to search the internet")
search.send_keys(Keys.RETURN) # hit return after you enter search text
time.sleep(5) # sleep for 5 seconds so you can see the results
browser.execute_script("window.scrollTo(0, 873)")
time.sleep(2)
browser.find_element(By.XPATH, '(//h3)[3]/a').click()`


as my code.



What Happens: Goes to Google and types in the words and hits search. Scrolls down a bit and then it clicks the first link



What I want: I want to be able to click a random link from the search result page



How to do this?



Edit: This is what I meant when I said unnecessary:
Unnecessary 1: https://imgur.com/a/70qz89x



Unnecessary 2: https://imgur.com/a/8WWvcnC



These are the only results I want:
This: https://imgur.com/a/eTatFV9










share|improve this question




















  • 1





    You'll need to collect the links from the page and then make a random choice from that list. See the random package documentation for the choice method.

    – Prune
    Aug 22 '18 at 22:57











  • Thanks for the help. I figured out how to do collect all the links from the page. However there are some links that are unnecessary. Is it possible to exclude them?

    – Kunwar Sodhi
    Aug 23 '18 at 2:39











  • If you got unnecessary links then your selector is not perfect. Use more specific selector to get rid of them

    – Andersson
    Aug 23 '18 at 4:58











  • Hi @KunwarSodhi, what exactly do you mean by unnecessary links? Do you want to edit your post and give us an example of what exactly you need - which links exactly, and we could provide a detailed solution.

    – Shlomi Bazel
    Aug 23 '18 at 8:21














0












0








0








Currently using Selenium+ Python and this code:



browser.get('http://www.google.com')
search = browser.find_element_by_name('q')
search.send_keys("how to search the internet")
search.send_keys(Keys.RETURN) # hit return after you enter search text
time.sleep(5) # sleep for 5 seconds so you can see the results
browser.execute_script("window.scrollTo(0, 873)")
time.sleep(2)
browser.find_element(By.XPATH, '(//h3)[3]/a').click()`


as my code.



What Happens: Goes to Google and types in the words and hits search. Scrolls down a bit and then it clicks the first link



What I want: I want to be able to click a random link from the search result page



How to do this?



Edit: This is what I meant when I said unnecessary:
Unnecessary 1: https://imgur.com/a/70qz89x



Unnecessary 2: https://imgur.com/a/8WWvcnC



These are the only results I want:
This: https://imgur.com/a/eTatFV9










share|improve this question
















Currently using Selenium+ Python and this code:



browser.get('http://www.google.com')
search = browser.find_element_by_name('q')
search.send_keys("how to search the internet")
search.send_keys(Keys.RETURN) # hit return after you enter search text
time.sleep(5) # sleep for 5 seconds so you can see the results
browser.execute_script("window.scrollTo(0, 873)")
time.sleep(2)
browser.find_element(By.XPATH, '(//h3)[3]/a').click()`


as my code.



What Happens: Goes to Google and types in the words and hits search. Scrolls down a bit and then it clicks the first link



What I want: I want to be able to click a random link from the search result page



How to do this?



Edit: This is what I meant when I said unnecessary:
Unnecessary 1: https://imgur.com/a/70qz89x



Unnecessary 2: https://imgur.com/a/8WWvcnC



These are the only results I want:
This: https://imgur.com/a/eTatFV9







python selenium selenium-webdriver xpath webdriver






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Aug 24 '18 at 15:14









DebanjanB

43.3k104384




43.3k104384










asked Aug 22 '18 at 22:54









Kunwar SodhiKunwar Sodhi

8817




8817








  • 1





    You'll need to collect the links from the page and then make a random choice from that list. See the random package documentation for the choice method.

    – Prune
    Aug 22 '18 at 22:57











  • Thanks for the help. I figured out how to do collect all the links from the page. However there are some links that are unnecessary. Is it possible to exclude them?

    – Kunwar Sodhi
    Aug 23 '18 at 2:39











  • If you got unnecessary links then your selector is not perfect. Use more specific selector to get rid of them

    – Andersson
    Aug 23 '18 at 4:58











  • Hi @KunwarSodhi, what exactly do you mean by unnecessary links? Do you want to edit your post and give us an example of what exactly you need - which links exactly, and we could provide a detailed solution.

    – Shlomi Bazel
    Aug 23 '18 at 8:21














  • 1





    You'll need to collect the links from the page and then make a random choice from that list. See the random package documentation for the choice method.

    – Prune
    Aug 22 '18 at 22:57











  • Thanks for the help. I figured out how to do collect all the links from the page. However there are some links that are unnecessary. Is it possible to exclude them?

    – Kunwar Sodhi
    Aug 23 '18 at 2:39











  • If you got unnecessary links then your selector is not perfect. Use more specific selector to get rid of them

    – Andersson
    Aug 23 '18 at 4:58











  • Hi @KunwarSodhi, what exactly do you mean by unnecessary links? Do you want to edit your post and give us an example of what exactly you need - which links exactly, and we could provide a detailed solution.

    – Shlomi Bazel
    Aug 23 '18 at 8:21








1




1





You'll need to collect the links from the page and then make a random choice from that list. See the random package documentation for the choice method.

– Prune
Aug 22 '18 at 22:57





You'll need to collect the links from the page and then make a random choice from that list. See the random package documentation for the choice method.

– Prune
Aug 22 '18 at 22:57













Thanks for the help. I figured out how to do collect all the links from the page. However there are some links that are unnecessary. Is it possible to exclude them?

– Kunwar Sodhi
Aug 23 '18 at 2:39





Thanks for the help. I figured out how to do collect all the links from the page. However there are some links that are unnecessary. Is it possible to exclude them?

– Kunwar Sodhi
Aug 23 '18 at 2:39













If you got unnecessary links then your selector is not perfect. Use more specific selector to get rid of them

– Andersson
Aug 23 '18 at 4:58





If you got unnecessary links then your selector is not perfect. Use more specific selector to get rid of them

– Andersson
Aug 23 '18 at 4:58













Hi @KunwarSodhi, what exactly do you mean by unnecessary links? Do you want to edit your post and give us an example of what exactly you need - which links exactly, and we could provide a detailed solution.

– Shlomi Bazel
Aug 23 '18 at 8:21





Hi @KunwarSodhi, what exactly do you mean by unnecessary links? Do you want to edit your post and give us an example of what exactly you need - which links exactly, and we could provide a detailed solution.

– Shlomi Bazel
Aug 23 '18 at 8:21












1 Answer
1






active

oldest

votes


















1














As per your question to click() on a random link from google search results, as per your code trial if you invoke window.scrollTo(0, 873) and then invoke click() as in:



find_element(By.XPATH, '(//h3)[3]/a').click()`


Selenium will still try to attempt click() on the first match, which may not be your desired usecase.



Solution



Inorder to click() on a random link from google search results, you can create a List out of the search results and then generate a Random Number and invoke click() through an index as follows:





  • Code Block:



    from selenium import webdriver
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.support import expected_conditions as EC
    from selenium.webdriver.common.keys import Keys
    from random import randint

    options = webdriver.ChromeOptions()
    options.add_argument("start-maximized")
    options.add_argument('disable-infobars')
    browser=webdriver.Chrome(chrome_options=options, executable_path=r'C:UtilityBrowserDriverschromedriver.exe')
    browser.get('http://www.google.com')
    search = browser.find_element_by_name('q')
    search.send_keys("selenium")
    search.send_keys(Keys.RETURN)
    my_search_list = WebDriverWait(browser, 10).until(EC.visibility_of_all_elements_located((By.XPATH, "//h3[@class='r']/a[not(ancestor::div[@class='xIleA'])]")))
    myRandomNumber = randint(0, len(my_search_list))
    print(myRandomNumber)
    my_search_list[myRandomNumber].click()



  • Console Output:



    4







share|improve this answer
























  • Gives list index error

    – Kunwar Sodhi
    Aug 23 '18 at 14:27











  • Can you ensure you are accessing http://www.google.com as your program might be accessing the local google server instance which is probhably not ending with .com. In that case you may require to tweak the xpath but the logic remains the same. In this run I got an output as 6.

    – DebanjanB
    Aug 23 '18 at 14:30













  • It works!. In this case I got a output of 2. However the first time I ran I got a output of 10 and it didn't work then. Maybe the 10th link isn't on the webpage at the time?

    – Kunwar Sodhi
    Aug 23 '18 at 14:36











  • Yup I just got 10 as a output and it didn't work and said List Index out of range

    – Kunwar Sodhi
    Aug 23 '18 at 14:37






  • 1





    Sorry I just got enough rep to upvote.

    – Kunwar Sodhi
    Aug 24 '18 at 12:42











Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f51976238%2fhow-to-click-a-random-link-from-google-search-results-through-selenium-and-pytho%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes









1














As per your question to click() on a random link from google search results, as per your code trial if you invoke window.scrollTo(0, 873) and then invoke click() as in:



find_element(By.XPATH, '(//h3)[3]/a').click()`


Selenium will still try to attempt click() on the first match, which may not be your desired usecase.



Solution



Inorder to click() on a random link from google search results, you can create a List out of the search results and then generate a Random Number and invoke click() through an index as follows:





  • Code Block:



    from selenium import webdriver
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.support import expected_conditions as EC
    from selenium.webdriver.common.keys import Keys
    from random import randint

    options = webdriver.ChromeOptions()
    options.add_argument("start-maximized")
    options.add_argument('disable-infobars')
    browser=webdriver.Chrome(chrome_options=options, executable_path=r'C:UtilityBrowserDriverschromedriver.exe')
    browser.get('http://www.google.com')
    search = browser.find_element_by_name('q')
    search.send_keys("selenium")
    search.send_keys(Keys.RETURN)
    my_search_list = WebDriverWait(browser, 10).until(EC.visibility_of_all_elements_located((By.XPATH, "//h3[@class='r']/a[not(ancestor::div[@class='xIleA'])]")))
    myRandomNumber = randint(0, len(my_search_list))
    print(myRandomNumber)
    my_search_list[myRandomNumber].click()



  • Console Output:



    4







share|improve this answer
























  • Gives list index error

    – Kunwar Sodhi
    Aug 23 '18 at 14:27











  • Can you ensure you are accessing http://www.google.com as your program might be accessing the local google server instance which is probhably not ending with .com. In that case you may require to tweak the xpath but the logic remains the same. In this run I got an output as 6.

    – DebanjanB
    Aug 23 '18 at 14:30













  • It works!. In this case I got a output of 2. However the first time I ran I got a output of 10 and it didn't work then. Maybe the 10th link isn't on the webpage at the time?

    – Kunwar Sodhi
    Aug 23 '18 at 14:36











  • Yup I just got 10 as a output and it didn't work and said List Index out of range

    – Kunwar Sodhi
    Aug 23 '18 at 14:37






  • 1





    Sorry I just got enough rep to upvote.

    – Kunwar Sodhi
    Aug 24 '18 at 12:42
















1














As per your question to click() on a random link from google search results, as per your code trial if you invoke window.scrollTo(0, 873) and then invoke click() as in:



find_element(By.XPATH, '(//h3)[3]/a').click()`


Selenium will still try to attempt click() on the first match, which may not be your desired usecase.



Solution



Inorder to click() on a random link from google search results, you can create a List out of the search results and then generate a Random Number and invoke click() through an index as follows:





  • Code Block:



    from selenium import webdriver
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.support import expected_conditions as EC
    from selenium.webdriver.common.keys import Keys
    from random import randint

    options = webdriver.ChromeOptions()
    options.add_argument("start-maximized")
    options.add_argument('disable-infobars')
    browser=webdriver.Chrome(chrome_options=options, executable_path=r'C:UtilityBrowserDriverschromedriver.exe')
    browser.get('http://www.google.com')
    search = browser.find_element_by_name('q')
    search.send_keys("selenium")
    search.send_keys(Keys.RETURN)
    my_search_list = WebDriverWait(browser, 10).until(EC.visibility_of_all_elements_located((By.XPATH, "//h3[@class='r']/a[not(ancestor::div[@class='xIleA'])]")))
    myRandomNumber = randint(0, len(my_search_list))
    print(myRandomNumber)
    my_search_list[myRandomNumber].click()



  • Console Output:



    4







share|improve this answer
























  • Gives list index error

    – Kunwar Sodhi
    Aug 23 '18 at 14:27











  • Can you ensure you are accessing http://www.google.com as your program might be accessing the local google server instance which is probhably not ending with .com. In that case you may require to tweak the xpath but the logic remains the same. In this run I got an output as 6.

    – DebanjanB
    Aug 23 '18 at 14:30













  • It works!. In this case I got a output of 2. However the first time I ran I got a output of 10 and it didn't work then. Maybe the 10th link isn't on the webpage at the time?

    – Kunwar Sodhi
    Aug 23 '18 at 14:36











  • Yup I just got 10 as a output and it didn't work and said List Index out of range

    – Kunwar Sodhi
    Aug 23 '18 at 14:37






  • 1





    Sorry I just got enough rep to upvote.

    – Kunwar Sodhi
    Aug 24 '18 at 12:42














1












1








1







As per your question to click() on a random link from google search results, as per your code trial if you invoke window.scrollTo(0, 873) and then invoke click() as in:



find_element(By.XPATH, '(//h3)[3]/a').click()`


Selenium will still try to attempt click() on the first match, which may not be your desired usecase.



Solution



Inorder to click() on a random link from google search results, you can create a List out of the search results and then generate a Random Number and invoke click() through an index as follows:





  • Code Block:



    from selenium import webdriver
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.support import expected_conditions as EC
    from selenium.webdriver.common.keys import Keys
    from random import randint

    options = webdriver.ChromeOptions()
    options.add_argument("start-maximized")
    options.add_argument('disable-infobars')
    browser=webdriver.Chrome(chrome_options=options, executable_path=r'C:UtilityBrowserDriverschromedriver.exe')
    browser.get('http://www.google.com')
    search = browser.find_element_by_name('q')
    search.send_keys("selenium")
    search.send_keys(Keys.RETURN)
    my_search_list = WebDriverWait(browser, 10).until(EC.visibility_of_all_elements_located((By.XPATH, "//h3[@class='r']/a[not(ancestor::div[@class='xIleA'])]")))
    myRandomNumber = randint(0, len(my_search_list))
    print(myRandomNumber)
    my_search_list[myRandomNumber].click()



  • Console Output:



    4







share|improve this answer













As per your question to click() on a random link from google search results, as per your code trial if you invoke window.scrollTo(0, 873) and then invoke click() as in:



find_element(By.XPATH, '(//h3)[3]/a').click()`


Selenium will still try to attempt click() on the first match, which may not be your desired usecase.



Solution



Inorder to click() on a random link from google search results, you can create a List out of the search results and then generate a Random Number and invoke click() through an index as follows:





  • Code Block:



    from selenium import webdriver
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.support import expected_conditions as EC
    from selenium.webdriver.common.keys import Keys
    from random import randint

    options = webdriver.ChromeOptions()
    options.add_argument("start-maximized")
    options.add_argument('disable-infobars')
    browser=webdriver.Chrome(chrome_options=options, executable_path=r'C:UtilityBrowserDriverschromedriver.exe')
    browser.get('http://www.google.com')
    search = browser.find_element_by_name('q')
    search.send_keys("selenium")
    search.send_keys(Keys.RETURN)
    my_search_list = WebDriverWait(browser, 10).until(EC.visibility_of_all_elements_located((By.XPATH, "//h3[@class='r']/a[not(ancestor::div[@class='xIleA'])]")))
    myRandomNumber = randint(0, len(my_search_list))
    print(myRandomNumber)
    my_search_list[myRandomNumber].click()



  • Console Output:



    4








share|improve this answer












share|improve this answer



share|improve this answer










answered Aug 23 '18 at 14:06









DebanjanBDebanjanB

43.3k104384




43.3k104384













  • Gives list index error

    – Kunwar Sodhi
    Aug 23 '18 at 14:27











  • Can you ensure you are accessing http://www.google.com as your program might be accessing the local google server instance which is probhably not ending with .com. In that case you may require to tweak the xpath but the logic remains the same. In this run I got an output as 6.

    – DebanjanB
    Aug 23 '18 at 14:30













  • It works!. In this case I got a output of 2. However the first time I ran I got a output of 10 and it didn't work then. Maybe the 10th link isn't on the webpage at the time?

    – Kunwar Sodhi
    Aug 23 '18 at 14:36











  • Yup I just got 10 as a output and it didn't work and said List Index out of range

    – Kunwar Sodhi
    Aug 23 '18 at 14:37






  • 1





    Sorry I just got enough rep to upvote.

    – Kunwar Sodhi
    Aug 24 '18 at 12:42



















  • Gives list index error

    – Kunwar Sodhi
    Aug 23 '18 at 14:27











  • Can you ensure you are accessing http://www.google.com as your program might be accessing the local google server instance which is probhably not ending with .com. In that case you may require to tweak the xpath but the logic remains the same. In this run I got an output as 6.

    – DebanjanB
    Aug 23 '18 at 14:30













  • It works!. In this case I got a output of 2. However the first time I ran I got a output of 10 and it didn't work then. Maybe the 10th link isn't on the webpage at the time?

    – Kunwar Sodhi
    Aug 23 '18 at 14:36











  • Yup I just got 10 as a output and it didn't work and said List Index out of range

    – Kunwar Sodhi
    Aug 23 '18 at 14:37






  • 1





    Sorry I just got enough rep to upvote.

    – Kunwar Sodhi
    Aug 24 '18 at 12:42

















Gives list index error

– Kunwar Sodhi
Aug 23 '18 at 14:27





Gives list index error

– Kunwar Sodhi
Aug 23 '18 at 14:27













Can you ensure you are accessing http://www.google.com as your program might be accessing the local google server instance which is probhably not ending with .com. In that case you may require to tweak the xpath but the logic remains the same. In this run I got an output as 6.

– DebanjanB
Aug 23 '18 at 14:30







Can you ensure you are accessing http://www.google.com as your program might be accessing the local google server instance which is probhably not ending with .com. In that case you may require to tweak the xpath but the logic remains the same. In this run I got an output as 6.

– DebanjanB
Aug 23 '18 at 14:30















It works!. In this case I got a output of 2. However the first time I ran I got a output of 10 and it didn't work then. Maybe the 10th link isn't on the webpage at the time?

– Kunwar Sodhi
Aug 23 '18 at 14:36





It works!. In this case I got a output of 2. However the first time I ran I got a output of 10 and it didn't work then. Maybe the 10th link isn't on the webpage at the time?

– Kunwar Sodhi
Aug 23 '18 at 14:36













Yup I just got 10 as a output and it didn't work and said List Index out of range

– Kunwar Sodhi
Aug 23 '18 at 14:37





Yup I just got 10 as a output and it didn't work and said List Index out of range

– Kunwar Sodhi
Aug 23 '18 at 14:37




1




1





Sorry I just got enough rep to upvote.

– Kunwar Sodhi
Aug 24 '18 at 12:42





Sorry I just got enough rep to upvote.

– Kunwar Sodhi
Aug 24 '18 at 12:42




















draft saved

draft discarded




















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f51976238%2fhow-to-click-a-random-link-from-google-search-results-through-selenium-and-pytho%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

Can a sorcerer learn a 5th-level spell early by creating spell slots using the Font of Magic feature?

Does disintegrating a polymorphed enemy still kill it after the 2018 errata?

A Topological Invariant for $pi_3(U(n))$