Python 3 BS4 - Extract Data from tags (Continued)

So I have HTML code that looks like this.

<li data-ng-repeat="sector in data.sectors"> <a target="_self" data-ng-href="/stocks/quotes/-382G/components/A" href="/stocks/quotes/-382G/components/A"><span>SIC-3826 Laboratory Analytical Instruments</span></a> </li>

And I want to extract the information w/in the span tag. Unfortunately when I use the following code:

tags = soup.findAll("li",attrs={"data-ng-repeat":"sector in data.sectors"})

# tags = soup.find_all("a",attrs= {"target=","data-ng-href="})

# tags = soup.find_all("a")

for tag in tags:

print(tag.text)

the result is [[sector.description]].
What I want to extract is the information including "SIC-3826 Laboratory Analytical Instruments"

Any help would be greatly appreciated. I have tried all sorts of alternatives yet I cannot get the information that I want.

Thank you in advance!

asked Nov 20 '18 at 20:46

user3814004

What is [[sector.description]]?

– Andersson
Nov 20 '18 at 20:57

1

This looks like a classic case of dynamic contents being scraped. If I had to guess, [[sector.description]] is a placeholder for the scripts to render the page with actual information. You'll need a module that supports dynamic contents, try selenium or requests-html. Unfortunately bs4 just doesn't have the capability to read dynamically generated contents.

– Idlehands
Nov 20 '18 at 21:03

[[sector.description]] is what print(tag.text) responds rather than the text that appears w/in the tag

– user3814004
Nov 20 '18 at 23:11

add a comment |

So I have HTML code that looks like this.

<li data-ng-repeat="sector in data.sectors"> <a target="_self" data-ng-href="/stocks/quotes/-382G/components/A" href="/stocks/quotes/-382G/components/A"><span>SIC-3826 Laboratory Analytical Instruments</span></a> </li>

And I want to extract the information w/in the span tag. Unfortunately when I use the following code:

tags = soup.findAll("li",attrs={"data-ng-repeat":"sector in data.sectors"})

# tags = soup.find_all("a",attrs= {"target=","data-ng-href="})

# tags = soup.find_all("a")

for tag in tags:

print(tag.text)

the result is [[sector.description]].
What I want to extract is the information including "SIC-3826 Laboratory Analytical Instruments"

Any help would be greatly appreciated. I have tried all sorts of alternatives yet I cannot get the information that I want.

Thank you in advance!

asked Nov 20 '18 at 20:46

user3814004

What is [[sector.description]]?

– Andersson
Nov 20 '18 at 20:57

1

This looks like a classic case of dynamic contents being scraped. If I had to guess, [[sector.description]] is a placeholder for the scripts to render the page with actual information. You'll need a module that supports dynamic contents, try selenium or requests-html. Unfortunately bs4 just doesn't have the capability to read dynamically generated contents.

– Idlehands
Nov 20 '18 at 21:03

[[sector.description]] is what print(tag.text) responds rather than the text that appears w/in the tag

– user3814004
Nov 20 '18 at 23:11

add a comment |

So I have HTML code that looks like this.

<li data-ng-repeat="sector in data.sectors"> <a target="_self" data-ng-href="/stocks/quotes/-382G/components/A" href="/stocks/quotes/-382G/components/A"><span>SIC-3826 Laboratory Analytical Instruments</span></a> </li>

And I want to extract the information w/in the span tag. Unfortunately when I use the following code:

tags = soup.findAll("li",attrs={"data-ng-repeat":"sector in data.sectors"})

# tags = soup.find_all("a",attrs= {"target=","data-ng-href="})

# tags = soup.find_all("a")

for tag in tags:

print(tag.text)

the result is [[sector.description]].
What I want to extract is the information including "SIC-3826 Laboratory Analytical Instruments"

Any help would be greatly appreciated. I have tried all sorts of alternatives yet I cannot get the information that I want.

Thank you in advance!

asked Nov 20 '18 at 20:46

user3814004

So I have HTML code that looks like this.

<li data-ng-repeat="sector in data.sectors"> <a target="_self" data-ng-href="/stocks/quotes/-382G/components/A" href="/stocks/quotes/-382G/components/A"><span>SIC-3826 Laboratory Analytical Instruments</span></a> </li>

And I want to extract the information w/in the span tag. Unfortunately when I use the following code:

tags = soup.findAll("li",attrs={"data-ng-repeat":"sector in data.sectors"})

# tags = soup.find_all("a",attrs= {"target=","data-ng-href="})

# tags = soup.find_all("a")

for tag in tags:

print(tag.text)

the result is [[sector.description]].
What I want to extract is the information including "SIC-3826 Laboratory Analytical Instruments"

Any help would be greatly appreciated. I have tried all sorts of alternatives yet I cannot get the information that I want.

Thank you in advance!

<li data-ng-repeat="sector in data.sectors"> <a target="_self" data-ng-href="/stocks/quotes/-382G/components/A" href="/stocks/quotes/-382G/components/A"><span>SIC-3826 Laboratory Analytical Instruments</span></a> </li>

<li data-ng-repeat="sector in data.sectors"> <a target="_self" data-ng-href="/stocks/quotes/-382G/components/A" href="/stocks/quotes/-382G/components/A"><span>SIC-3826 Laboratory Analytical Instruments</span></a> </li>

python beautifulsoup tags

asked Nov 20 '18 at 20:46

user3814004

asked Nov 20 '18 at 20:46

user3814004

asked Nov 20 '18 at 20:46

user3814004

asked Nov 20 '18 at 20:46

user3814004

asked Nov 20 '18 at 20:46

user3814004

What is [[sector.description]]?

– Andersson
Nov 20 '18 at 20:57

1

This looks like a classic case of dynamic contents being scraped. If I had to guess, [[sector.description]] is a placeholder for the scripts to render the page with actual information. You'll need a module that supports dynamic contents, try selenium or requests-html. Unfortunately bs4 just doesn't have the capability to read dynamically generated contents.

– Idlehands
Nov 20 '18 at 21:03

[[sector.description]] is what print(tag.text) responds rather than the text that appears w/in the tag

– user3814004
Nov 20 '18 at 23:11

add a comment |

What is [[sector.description]]?

– Andersson
Nov 20 '18 at 20:57

1

This looks like a classic case of dynamic contents being scraped. If I had to guess, [[sector.description]] is a placeholder for the scripts to render the page with actual information. You'll need a module that supports dynamic contents, try selenium or requests-html. Unfortunately bs4 just doesn't have the capability to read dynamically generated contents.

– Idlehands
Nov 20 '18 at 21:03

[[sector.description]] is what print(tag.text) responds rather than the text that appears w/in the tag

– user3814004
Nov 20 '18 at 23:11

What is [[sector.description]]?

– Andersson
Nov 20 '18 at 20:57

This looks like a classic case of dynamic contents being scraped. If I had to guess, [[sector.description]] is a placeholder for the scripts to render the page with actual information. You'll need a module that supports dynamic contents, try selenium or requests-html. Unfortunately bs4 just doesn't have the capability to read dynamically generated contents.

– Idlehands
Nov 20 '18 at 21:03

[[sector.description]] is what print(tag.text) responds rather than the text that appears w/in the tag

– user3814004
Nov 20 '18 at 23:11

add a comment |

1 Answer
1

active

oldest

votes

Yes all you need to do is:

x = """<li data-ng-repeat="sector in data.sectors"> <a target="_self" data-ng-href="/stocks/quotes/-382G/components/A" href="/stocks/quotes/-382G/components/A"><span>SIC-3826 Laboratory Analytical Instruments</span></a> </li>"""



from bs4 import BeautifulSoup

print(BeautifulSoup(x, "lxml").text)

answered Nov 20 '18 at 20:58

Yang K

678

What if x is a complete webpage? What happens then?

– Idlehands
Nov 20 '18 at 21:01

The issue is that I don't know what x is going to be before the page gets served. Either that or I may be misunderstanding your answer.

– user3814004
Nov 20 '18 at 23:05

add a comment |

Your Answer

StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});

}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53401239%2fpython-3-bs4-extract-data-from-span-tags-continued%23new-answer', 'question_page');
}
);

Post as a guest

Name

Required, but never shown

1 Answer
1

active

oldest

votes

1 Answer
1

active

oldest

votes

Yes all you need to do is:

x = """<li data-ng-repeat="sector in data.sectors"> <a target="_self" data-ng-href="/stocks/quotes/-382G/components/A" href="/stocks/quotes/-382G/components/A"><span>SIC-3826 Laboratory Analytical Instruments</span></a> </li>"""



from bs4 import BeautifulSoup

print(BeautifulSoup(x, "lxml").text)

answered Nov 20 '18 at 20:58

Yang K

678

What if x is a complete webpage? What happens then?

– Idlehands
Nov 20 '18 at 21:01

The issue is that I don't know what x is going to be before the page gets served. Either that or I may be misunderstanding your answer.

– user3814004
Nov 20 '18 at 23:05

add a comment |

Yes all you need to do is:

x = """<li data-ng-repeat="sector in data.sectors"> <a target="_self" data-ng-href="/stocks/quotes/-382G/components/A" href="/stocks/quotes/-382G/components/A"><span>SIC-3826 Laboratory Analytical Instruments</span></a> </li>"""



from bs4 import BeautifulSoup

print(BeautifulSoup(x, "lxml").text)

answered Nov 20 '18 at 20:58

Yang K

678

What if x is a complete webpage? What happens then?

– Idlehands
Nov 20 '18 at 21:01

The issue is that I don't know what x is going to be before the page gets served. Either that or I may be misunderstanding your answer.

– user3814004
Nov 20 '18 at 23:05

add a comment |

Yes all you need to do is:

x = """<li data-ng-repeat="sector in data.sectors"> <a target="_self" data-ng-href="/stocks/quotes/-382G/components/A" href="/stocks/quotes/-382G/components/A"><span>SIC-3826 Laboratory Analytical Instruments</span></a> </li>"""



from bs4 import BeautifulSoup

print(BeautifulSoup(x, "lxml").text)

answered Nov 20 '18 at 20:58

Yang K

678

Yes all you need to do is:

x = """<li data-ng-repeat="sector in data.sectors"> <a target="_self" data-ng-href="/stocks/quotes/-382G/components/A" href="/stocks/quotes/-382G/components/A"><span>SIC-3826 Laboratory Analytical Instruments</span></a> </li>"""



from bs4 import BeautifulSoup

print(BeautifulSoup(x, "lxml").text)

answered Nov 20 '18 at 20:58

Yang K

678

answered Nov 20 '18 at 20:58

Yang K

678

answered Nov 20 '18 at 20:58

Yang K

678

answered Nov 20 '18 at 20:58

Yang K

678

What if x is a complete webpage? What happens then?

– Idlehands
Nov 20 '18 at 21:01

The issue is that I don't know what x is going to be before the page gets served. Either that or I may be misunderstanding your answer.

– user3814004
Nov 20 '18 at 23:05

add a comment |

What if x is a complete webpage? What happens then?

– Idlehands
Nov 20 '18 at 21:01

The issue is that I don't know what x is going to be before the page gets served. Either that or I may be misunderstanding your answer.

– user3814004
Nov 20 '18 at 23:05

What if x is a complete webpage? What happens then?

– Idlehands
Nov 20 '18 at 21:01

The issue is that I don't know what x is going to be before the page gets served. Either that or I may be misunderstanding your answer.

– user3814004
Nov 20 '18 at 23:05

add a comment |

draft saved

draft discarded

Thanks for contributing an answer to Stack Overflow!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

Search This Blog

Ufyukyu