How to click a random link from google search results through Selenium and Python

Currently using Selenium+ Python and this code:

browser.get('http://www.google.com')

search = browser.find_element_by_name('q')

search.send_keys("how to search the internet")

search.send_keys(Keys.RETURN) # hit return after you enter search text

time.sleep(5) # sleep for 5 seconds so you can see the results

browser.execute_script("window.scrollTo(0, 873)") 

time.sleep(2)

browser.find_element(By.XPATH, '(//h3)[3]/a').click()`

as my code.

What Happens: Goes to Google and types in the words and hits search. Scrolls down a bit and then it clicks the first link

What I want: I want to be able to click a random link from the search result page

How to do this?

Edit: This is what I meant when I said unnecessary:
Unnecessary 1: https://imgur.com/a/70qz89x

Unnecessary 2: https://imgur.com/a/8WWvcnC

These are the only results I want:
This: https://imgur.com/a/eTatFV9

edited Aug 24 '18 at 15:14

DebanjanB

43.3k104384

asked Aug 22 '18 at 22:54

Kunwar Sodhi

8817

1

You'll need to collect the links from the page and then make a random choice from that list. See the random package documentation for the choice method.

– Prune
Aug 22 '18 at 22:57

Thanks for the help. I figured out how to do collect all the links from the page. However there are some links that are unnecessary. Is it possible to exclude them?

– Kunwar Sodhi
Aug 23 '18 at 2:39

If you got unnecessary links then your selector is not perfect. Use more specific selector to get rid of them

– Andersson
Aug 23 '18 at 4:58

Hi @KunwarSodhi, what exactly do you mean by unnecessary links? Do you want to edit your post and give us an example of what exactly you need - which links exactly, and we could provide a detailed solution.

– Shlomi Bazel
Aug 23 '18 at 8:21

add a comment |

Currently using Selenium+ Python and this code:

browser.get('http://www.google.com')

search = browser.find_element_by_name('q')

search.send_keys("how to search the internet")

search.send_keys(Keys.RETURN) # hit return after you enter search text

time.sleep(5) # sleep for 5 seconds so you can see the results

browser.execute_script("window.scrollTo(0, 873)") 

time.sleep(2)

browser.find_element(By.XPATH, '(//h3)[3]/a').click()`

as my code.

What Happens: Goes to Google and types in the words and hits search. Scrolls down a bit and then it clicks the first link

What I want: I want to be able to click a random link from the search result page

How to do this?

Edit: This is what I meant when I said unnecessary:
Unnecessary 1: https://imgur.com/a/70qz89x

Unnecessary 2: https://imgur.com/a/8WWvcnC

These are the only results I want:
This: https://imgur.com/a/eTatFV9

edited Aug 24 '18 at 15:14

DebanjanB

43.3k104384

asked Aug 22 '18 at 22:54

Kunwar Sodhi

8817

1

You'll need to collect the links from the page and then make a random choice from that list. See the random package documentation for the choice method.

– Prune
Aug 22 '18 at 22:57

Thanks for the help. I figured out how to do collect all the links from the page. However there are some links that are unnecessary. Is it possible to exclude them?

– Kunwar Sodhi
Aug 23 '18 at 2:39

If you got unnecessary links then your selector is not perfect. Use more specific selector to get rid of them

– Andersson
Aug 23 '18 at 4:58

Hi @KunwarSodhi, what exactly do you mean by unnecessary links? Do you want to edit your post and give us an example of what exactly you need - which links exactly, and we could provide a detailed solution.

– Shlomi Bazel
Aug 23 '18 at 8:21

add a comment |

Currently using Selenium+ Python and this code:

browser.get('http://www.google.com')

search = browser.find_element_by_name('q')

search.send_keys("how to search the internet")

search.send_keys(Keys.RETURN) # hit return after you enter search text

time.sleep(5) # sleep for 5 seconds so you can see the results

browser.execute_script("window.scrollTo(0, 873)") 

time.sleep(2)

browser.find_element(By.XPATH, '(//h3)[3]/a').click()`

as my code.

What Happens: Goes to Google and types in the words and hits search. Scrolls down a bit and then it clicks the first link

What I want: I want to be able to click a random link from the search result page

How to do this?

Edit: This is what I meant when I said unnecessary:
Unnecessary 1: https://imgur.com/a/70qz89x

Unnecessary 2: https://imgur.com/a/8WWvcnC

These are the only results I want:
This: https://imgur.com/a/eTatFV9

edited Aug 24 '18 at 15:14

DebanjanB

43.3k104384

asked Aug 22 '18 at 22:54

Kunwar Sodhi

8817

Currently using Selenium+ Python and this code:

browser.get('http://www.google.com')

search = browser.find_element_by_name('q')

search.send_keys("how to search the internet")

search.send_keys(Keys.RETURN) # hit return after you enter search text

time.sleep(5) # sleep for 5 seconds so you can see the results

browser.execute_script("window.scrollTo(0, 873)") 

time.sleep(2)

browser.find_element(By.XPATH, '(//h3)[3]/a').click()`

as my code.

What Happens: Goes to Google and types in the words and hits search. Scrolls down a bit and then it clicks the first link

What I want: I want to be able to click a random link from the search result page

How to do this?

Edit: This is what I meant when I said unnecessary:
Unnecessary 1: https://imgur.com/a/70qz89x

Unnecessary 2: https://imgur.com/a/8WWvcnC

These are the only results I want:
This: https://imgur.com/a/eTatFV9

python selenium selenium-webdriver xpath webdriver

edited Aug 24 '18 at 15:14

DebanjanB

43.3k104384

asked Aug 22 '18 at 22:54

Kunwar Sodhi

8817

edited Aug 24 '18 at 15:14

DebanjanB

43.3k104384

asked Aug 22 '18 at 22:54

Kunwar Sodhi

8817

edited Aug 24 '18 at 15:14

DebanjanB

43.3k104384

edited Aug 24 '18 at 15:14

DebanjanB

43.3k104384

edited Aug 24 '18 at 15:14

DebanjanB

43.3k104384

asked Aug 22 '18 at 22:54

Kunwar Sodhi

8817

asked Aug 22 '18 at 22:54

Kunwar Sodhi

8817

asked Aug 22 '18 at 22:54

Kunwar Sodhi

8817

1

You'll need to collect the links from the page and then make a random choice from that list. See the random package documentation for the choice method.

– Prune
Aug 22 '18 at 22:57

Thanks for the help. I figured out how to do collect all the links from the page. However there are some links that are unnecessary. Is it possible to exclude them?

– Kunwar Sodhi
Aug 23 '18 at 2:39

If you got unnecessary links then your selector is not perfect. Use more specific selector to get rid of them

– Andersson
Aug 23 '18 at 4:58

Hi @KunwarSodhi, what exactly do you mean by unnecessary links? Do you want to edit your post and give us an example of what exactly you need - which links exactly, and we could provide a detailed solution.

– Shlomi Bazel
Aug 23 '18 at 8:21

add a comment |

1

You'll need to collect the links from the page and then make a random choice from that list. See the random package documentation for the choice method.

– Prune
Aug 22 '18 at 22:57

Thanks for the help. I figured out how to do collect all the links from the page. However there are some links that are unnecessary. Is it possible to exclude them?

– Kunwar Sodhi
Aug 23 '18 at 2:39

If you got unnecessary links then your selector is not perfect. Use more specific selector to get rid of them

– Andersson
Aug 23 '18 at 4:58

Hi @KunwarSodhi, what exactly do you mean by unnecessary links? Do you want to edit your post and give us an example of what exactly you need - which links exactly, and we could provide a detailed solution.

– Shlomi Bazel
Aug 23 '18 at 8:21

You'll need to collect the links from the page and then make a random choice from that list. See the random package documentation for the choice method.

– Prune
Aug 22 '18 at 22:57

Thanks for the help. I figured out how to do collect all the links from the page. However there are some links that are unnecessary. Is it possible to exclude them?

– Kunwar Sodhi
Aug 23 '18 at 2:39

If you got unnecessary links then your selector is not perfect. Use more specific selector to get rid of them

– Andersson
Aug 23 '18 at 4:58

Hi @KunwarSodhi, what exactly do you mean by unnecessary links? Do you want to edit your post and give us an example of what exactly you need - which links exactly, and we could provide a detailed solution.

– Shlomi Bazel
Aug 23 '18 at 8:21

add a comment |

1 Answer
1

active

oldest

votes

As per your question to click() on a random link from google search results, as per your code trial if you invoke window.scrollTo(0, 873) and then invoke click() as in:

find_element(By.XPATH, '(//h3)[3]/a').click()`

Selenium will still try to attempt click() on the first match, which may not be your desired usecase.

Solution

Inorder to click() on a random link from google search results, you can create a List out of the search results and then generate a Random Number and invoke click() through an index as follows:

Code Block:

from selenium import webdriver

from selenium.webdriver.common.by import By

from selenium.webdriver.support.ui import WebDriverWait

from selenium.webdriver.support import expected_conditions as EC

from selenium.webdriver.common.keys import Keys 

from random import randint



options = webdriver.ChromeOptions() 

options.add_argument("start-maximized")

options.add_argument('disable-infobars')

browser=webdriver.Chrome(chrome_options=options, executable_path=r'C:UtilityBrowserDriverschromedriver.exe')

browser.get('http://www.google.com')

search = browser.find_element_by_name('q')

search.send_keys("selenium")

search.send_keys(Keys.RETURN)

my_search_list = WebDriverWait(browser, 10).until(EC.visibility_of_all_elements_located((By.XPATH, "//h3[@class='r']/a[not(ancestor::div[@class='xIleA'])]")))

myRandomNumber = randint(0, len(my_search_list))

print(myRandomNumber)

my_search_list[myRandomNumber].click()

Console Output:
```
4
```

answered Aug 23 '18 at 14:06

DebanjanB

43.3k104384

Gives list index error

– Kunwar Sodhi
Aug 23 '18 at 14:27

Can you ensure you are accessing http://www.google.com as your program might be accessing the local google server instance which is probhably not ending with .com. In that case you may require to tweak the xpath but the logic remains the same. In this run I got an output as 6.

– DebanjanB
Aug 23 '18 at 14:30

It works!. In this case I got a output of 2. However the first time I ran I got a output of 10 and it didn't work then. Maybe the 10th link isn't on the webpage at the time?

– Kunwar Sodhi
Aug 23 '18 at 14:36

Yup I just got 10 as a output and it didn't work and said List Index out of range

– Kunwar Sodhi
Aug 23 '18 at 14:37

1

Sorry I just got enough rep to upvote.

– Kunwar Sodhi
Aug 24 '18 at 12:42

|
show 10 more comments

Your Answer

StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});

}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f51976238%2fhow-to-click-a-random-link-from-google-search-results-through-selenium-and-pytho%23new-answer', 'question_page');
}
);

Post as a guest

Name

Required, but never shown

1 Answer
1

active

oldest

votes

1 Answer
1

active

oldest

votes

As per your question to click() on a random link from google search results, as per your code trial if you invoke window.scrollTo(0, 873) and then invoke click() as in:

find_element(By.XPATH, '(//h3)[3]/a').click()`

Selenium will still try to attempt click() on the first match, which may not be your desired usecase.

Solution

Code Block:

from selenium import webdriver

from selenium.webdriver.common.by import By

from selenium.webdriver.support.ui import WebDriverWait

from selenium.webdriver.support import expected_conditions as EC

from selenium.webdriver.common.keys import Keys 

from random import randint



options = webdriver.ChromeOptions() 

options.add_argument("start-maximized")

options.add_argument('disable-infobars')

browser=webdriver.Chrome(chrome_options=options, executable_path=r'C:UtilityBrowserDriverschromedriver.exe')

browser.get('http://www.google.com')

search = browser.find_element_by_name('q')

search.send_keys("selenium")

search.send_keys(Keys.RETURN)

my_search_list = WebDriverWait(browser, 10).until(EC.visibility_of_all_elements_located((By.XPATH, "//h3[@class='r']/a[not(ancestor::div[@class='xIleA'])]")))

myRandomNumber = randint(0, len(my_search_list))

print(myRandomNumber)

my_search_list[myRandomNumber].click()

Console Output:
```
4
```

answered Aug 23 '18 at 14:06

DebanjanB

43.3k104384

Gives list index error

– Kunwar Sodhi
Aug 23 '18 at 14:27

Can you ensure you are accessing http://www.google.com as your program might be accessing the local google server instance which is probhably not ending with .com. In that case you may require to tweak the xpath but the logic remains the same. In this run I got an output as 6.

– DebanjanB
Aug 23 '18 at 14:30

It works!. In this case I got a output of 2. However the first time I ran I got a output of 10 and it didn't work then. Maybe the 10th link isn't on the webpage at the time?

– Kunwar Sodhi
Aug 23 '18 at 14:36

Yup I just got 10 as a output and it didn't work and said List Index out of range

– Kunwar Sodhi
Aug 23 '18 at 14:37

1

Sorry I just got enough rep to upvote.

– Kunwar Sodhi
Aug 24 '18 at 12:42

|
show 10 more comments

As per your question to click() on a random link from google search results, as per your code trial if you invoke window.scrollTo(0, 873) and then invoke click() as in:

find_element(By.XPATH, '(//h3)[3]/a').click()`

Selenium will still try to attempt click() on the first match, which may not be your desired usecase.

Solution

Code Block:

from selenium import webdriver

from selenium.webdriver.common.by import By

from selenium.webdriver.support.ui import WebDriverWait

from selenium.webdriver.support import expected_conditions as EC

from selenium.webdriver.common.keys import Keys 

from random import randint



options = webdriver.ChromeOptions() 

options.add_argument("start-maximized")

options.add_argument('disable-infobars')

browser=webdriver.Chrome(chrome_options=options, executable_path=r'C:UtilityBrowserDriverschromedriver.exe')

browser.get('http://www.google.com')

search = browser.find_element_by_name('q')

search.send_keys("selenium")

search.send_keys(Keys.RETURN)

my_search_list = WebDriverWait(browser, 10).until(EC.visibility_of_all_elements_located((By.XPATH, "//h3[@class='r']/a[not(ancestor::div[@class='xIleA'])]")))

myRandomNumber = randint(0, len(my_search_list))

print(myRandomNumber)

my_search_list[myRandomNumber].click()

Console Output:
```
4
```

answered Aug 23 '18 at 14:06

DebanjanB

43.3k104384

Gives list index error

– Kunwar Sodhi
Aug 23 '18 at 14:27

Can you ensure you are accessing http://www.google.com as your program might be accessing the local google server instance which is probhably not ending with .com. In that case you may require to tweak the xpath but the logic remains the same. In this run I got an output as 6.

– DebanjanB
Aug 23 '18 at 14:30

It works!. In this case I got a output of 2. However the first time I ran I got a output of 10 and it didn't work then. Maybe the 10th link isn't on the webpage at the time?

– Kunwar Sodhi
Aug 23 '18 at 14:36

Yup I just got 10 as a output and it didn't work and said List Index out of range

– Kunwar Sodhi
Aug 23 '18 at 14:37

1

Sorry I just got enough rep to upvote.

– Kunwar Sodhi
Aug 24 '18 at 12:42

|
show 10 more comments

As per your question to click() on a random link from google search results, as per your code trial if you invoke window.scrollTo(0, 873) and then invoke click() as in:

find_element(By.XPATH, '(//h3)[3]/a').click()`

Selenium will still try to attempt click() on the first match, which may not be your desired usecase.

Solution

Code Block:

from selenium import webdriver

from selenium.webdriver.common.by import By

from selenium.webdriver.support.ui import WebDriverWait

from selenium.webdriver.support import expected_conditions as EC

from selenium.webdriver.common.keys import Keys 

from random import randint



options = webdriver.ChromeOptions() 

options.add_argument("start-maximized")

options.add_argument('disable-infobars')

browser=webdriver.Chrome(chrome_options=options, executable_path=r'C:UtilityBrowserDriverschromedriver.exe')

browser.get('http://www.google.com')

search = browser.find_element_by_name('q')

search.send_keys("selenium")

search.send_keys(Keys.RETURN)

my_search_list = WebDriverWait(browser, 10).until(EC.visibility_of_all_elements_located((By.XPATH, "//h3[@class='r']/a[not(ancestor::div[@class='xIleA'])]")))

myRandomNumber = randint(0, len(my_search_list))

print(myRandomNumber)

my_search_list[myRandomNumber].click()

Console Output:
```
4
```

answered Aug 23 '18 at 14:06

DebanjanB

43.3k104384

As per your question to click() on a random link from google search results, as per your code trial if you invoke window.scrollTo(0, 873) and then invoke click() as in:

find_element(By.XPATH, '(//h3)[3]/a').click()`

Selenium will still try to attempt click() on the first match, which may not be your desired usecase.

Solution

Code Block:

from selenium import webdriver

from selenium.webdriver.common.by import By

from selenium.webdriver.support.ui import WebDriverWait

from selenium.webdriver.support import expected_conditions as EC

from selenium.webdriver.common.keys import Keys 

from random import randint



options = webdriver.ChromeOptions() 

options.add_argument("start-maximized")

options.add_argument('disable-infobars')

browser=webdriver.Chrome(chrome_options=options, executable_path=r'C:UtilityBrowserDriverschromedriver.exe')

browser.get('http://www.google.com')

search = browser.find_element_by_name('q')

search.send_keys("selenium")

search.send_keys(Keys.RETURN)

my_search_list = WebDriverWait(browser, 10).until(EC.visibility_of_all_elements_located((By.XPATH, "//h3[@class='r']/a[not(ancestor::div[@class='xIleA'])]")))

myRandomNumber = randint(0, len(my_search_list))

print(myRandomNumber)

my_search_list[myRandomNumber].click()

Console Output:
```
4
```

answered Aug 23 '18 at 14:06

DebanjanB

43.3k104384

answered Aug 23 '18 at 14:06

DebanjanB

43.3k104384

answered Aug 23 '18 at 14:06

DebanjanB

43.3k104384

answered Aug 23 '18 at 14:06

DebanjanB

43.3k104384

Gives list index error

– Kunwar Sodhi
Aug 23 '18 at 14:27

Can you ensure you are accessing http://www.google.com as your program might be accessing the local google server instance which is probhably not ending with .com. In that case you may require to tweak the xpath but the logic remains the same. In this run I got an output as 6.

– DebanjanB
Aug 23 '18 at 14:30

It works!. In this case I got a output of 2. However the first time I ran I got a output of 10 and it didn't work then. Maybe the 10th link isn't on the webpage at the time?

– Kunwar Sodhi
Aug 23 '18 at 14:36

Yup I just got 10 as a output and it didn't work and said List Index out of range

– Kunwar Sodhi
Aug 23 '18 at 14:37

1

Sorry I just got enough rep to upvote.

– Kunwar Sodhi
Aug 24 '18 at 12:42

|
show 10 more comments

Gives list index error

– Kunwar Sodhi
Aug 23 '18 at 14:27

Can you ensure you are accessing http://www.google.com as your program might be accessing the local google server instance which is probhably not ending with .com. In that case you may require to tweak the xpath but the logic remains the same. In this run I got an output as 6.

– DebanjanB
Aug 23 '18 at 14:30

It works!. In this case I got a output of 2. However the first time I ran I got a output of 10 and it didn't work then. Maybe the 10th link isn't on the webpage at the time?

– Kunwar Sodhi
Aug 23 '18 at 14:36

Yup I just got 10 as a output and it didn't work and said List Index out of range

– Kunwar Sodhi
Aug 23 '18 at 14:37

1

Sorry I just got enough rep to upvote.

– Kunwar Sodhi
Aug 24 '18 at 12:42

Gives list index error

– Kunwar Sodhi
Aug 23 '18 at 14:27

Can you ensure you are accessing http://www.google.com as your program might be accessing the local google server instance which is probhably not ending with .com. In that case you may require to tweak the xpath but the logic remains the same. In this run I got an output as 6.

– DebanjanB
Aug 23 '18 at 14:30

It works!. In this case I got a output of 2. However the first time I ran I got a output of 10 and it didn't work then. Maybe the 10th link isn't on the webpage at the time?

– Kunwar Sodhi
Aug 23 '18 at 14:36

Yup I just got 10 as a output and it didn't work and said List Index out of range

– Kunwar Sodhi
Aug 23 '18 at 14:37

Sorry I just got enough rep to upvote.

– Kunwar Sodhi
Aug 24 '18 at 12:42

|
show 10 more comments

draft saved

draft discarded

Thanks for contributing an answer to Stack Overflow!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

Search This Blog

Ufyukyu