Mixing
How to get data inside tag using xpath
I'm using Scrapy to scrape a website. I have the following html structure
<dl>
<dt><b>Notice Date</b></dt>
<dd>12/17/2018<br> </dd>
<dt><b>Notice Type</b></dt>
<dd>Synopsis<br> </dd>
<dt><b>NAICS</b></dt>
<dd>238990 — All Other Specialty Trade Contractors<br> </dd>
<dt><b>Contracting Office</b></dt>
<dd>1108 South R Street Fort McCoy WI 54656-5142<br> </dd>
<dt><b>ZIP Code</b></dt>
<dd>54656-5142<br> </dd>
<dt><b>Solicitation Number</b></dt>
<dd>W911SA-18-B-2028<br> </dd>
<dt><b>Point of Contact</b></dt>
<dd>Jamie Gregar, Contract Specialist, Phone 6083881007 <br> </dd>
<dt><b>E-Mail Address</b></dt>
<dd><a href="mailto:jamie.j.gregar.civ@mail.mil">jamie.j.gregar.civ@mail.mil</a<br> </dd>
<dt><b>Small Business Set-Aside</b></dt>
<dd>HUBZone<br> </dd>
<dt><b>Description</b></dt>
<dd>some description here<br> </dd>
<dt><b>Web Link</b></dt>
<dd><a target="outside" href="https://www.fbo.gov/notices/54ba577b59c07b3757739e138b20ef3c">
Link To Document</a> <br>(https://www.fbo.gov/notices/54ba577b59c07b3757739e138b20ef3c)
<br> </dd>
<dt><b>Place of Performance</b></dt>
<dd><i>Address:</i> Laredo, TX </dd>
<dd><i>Zip Code:</i> 78040 </dd>
<dd><i>Country:</i> US <br> </dd>
<dt><b>Record</b> </dt>
<dd>SN05176005-F 20181219/181217230006 (fbodaily.com) <br> </dd>
<dt><b>Source</b></dt>
<dd><a target="fedbizopps" href="ftp://ftp.fbo.gov/FBOFeed20181217"> FedBizOpps Link to This Notice</a><br>(may not be valid after Archive Date) </dd>
</dl>
In the above structure, you can see the <dt> tag refers to the title of data and the <dd> tags follows to it, will contain relevant data and data may consists of different types (i.e. email addresses, text, web links).
So far I can get up to <dl> tag using xpath conventions. So can someone suggest me a way to get these data separately into the given fields using xpath starts from <dl> tag.
html xpath scrapy
asked Jan 2 at 15:29
Vajira PrabuddhakaVajira Prabuddhaka
161317
add a comment |
I'm using Scrapy to scrape a website. I have the following html structure
<dl>
<dt><b>Notice Date</b></dt>
<dd>12/17/2018<br> </dd>
<dt><b>Notice Type</b></dt>
<dd>Synopsis<br> </dd>
<dt><b>NAICS</b></dt>
<dd>238990 — All Other Specialty Trade Contractors<br> </dd>
<dt><b>Contracting Office</b></dt>
<dd>1108 South R Street Fort McCoy WI 54656-5142<br> </dd>
<dt><b>ZIP Code</b></dt>
<dd>54656-5142<br> </dd>
<dt><b>Solicitation Number</b></dt>
<dd>W911SA-18-B-2028<br> </dd>
<dt><b>Point of Contact</b></dt>
<dd>Jamie Gregar, Contract Specialist, Phone 6083881007 <br> </dd>
<dt><b>E-Mail Address</b></dt>
<dd><a href="mailto:jamie.j.gregar.civ@mail.mil">jamie.j.gregar.civ@mail.mil</a<br> </dd>
<dt><b>Small Business Set-Aside</b></dt>
<dd>HUBZone<br> </dd>
<dt><b>Description</b></dt>
<dd>some description here<br> </dd>
<dt><b>Web Link</b></dt>
<dd><a target="outside" href="https://www.fbo.gov/notices/54ba577b59c07b3757739e138b20ef3c">
Link To Document</a> <br>(https://www.fbo.gov/notices/54ba577b59c07b3757739e138b20ef3c)
<br> </dd>
<dt><b>Place of Performance</b></dt>
<dd><i>Address:</i> Laredo, TX </dd>
<dd><i>Zip Code:</i> 78040 </dd>
<dd><i>Country:</i> US <br> </dd>
<dt><b>Record</b> </dt>
<dd>SN05176005-F 20181219/181217230006 (fbodaily.com) <br> </dd>
<dt><b>Source</b></dt>
<dd><a target="fedbizopps" href="ftp://ftp.fbo.gov/FBOFeed20181217"> FedBizOpps Link to This Notice</a><br>(may not be valid after Archive Date) </dd>
</dl>
In the above structure, you can see the <dt> tag refers to the title of data and the <dd> tags follows to it, will contain relevant data and data may consists of different types (i.e. email addresses, text, web links).
So far I can get up to <dl> tag using xpath conventions. So can someone suggest me a way to get these data separately into the given fields using xpath starts from <dl> tag.
html xpath scrapy
asked Jan 2 at 15:29
Vajira PrabuddhakaVajira Prabuddhaka
161317
2
Show us how you tried. Share your current and desired output
– Andersson
Jan 2 at 16:35
add a comment |
I'm using Scrapy to scrape a website. I have the following html structure
<dl>
<dt><b>Notice Date</b></dt>
<dd>12/17/2018<br> </dd>
<dt><b>Notice Type</b></dt>
<dd>Synopsis<br> </dd>
<dt><b>NAICS</b></dt>
<dd>238990 — All Other Specialty Trade Contractors<br> </dd>
<dt><b>Contracting Office</b></dt>
<dd>1108 South R Street Fort McCoy WI 54656-5142<br> </dd>
<dt><b>ZIP Code</b></dt>
<dd>54656-5142<br> </dd>
<dt><b>Solicitation Number</b></dt>
<dd>W911SA-18-B-2028<br> </dd>
<dt><b>Point of Contact</b></dt>
<dd>Jamie Gregar, Contract Specialist, Phone 6083881007 <br> </dd>
<dt><b>E-Mail Address</b></dt>
<dd><a href="mailto:jamie.j.gregar.civ@mail.mil">jamie.j.gregar.civ@mail.mil</a<br> </dd>
<dt><b>Small Business Set-Aside</b></dt>
<dd>HUBZone<br> </dd>
<dt><b>Description</b></dt>
<dd>some description here<br> </dd>
<dt><b>Web Link</b></dt>
<dd><a target="outside" href="https://www.fbo.gov/notices/54ba577b59c07b3757739e138b20ef3c">
Link To Document</a> <br>(https://www.fbo.gov/notices/54ba577b59c07b3757739e138b20ef3c)
<br> </dd>
<dt><b>Place of Performance</b></dt>
<dd><i>Address:</i> Laredo, TX </dd>
<dd><i>Zip Code:</i> 78040 </dd>
<dd><i>Country:</i> US <br> </dd>
<dt><b>Record</b> </dt>
<dd>SN05176005-F 20181219/181217230006 (fbodaily.com) <br> </dd>
<dt><b>Source</b></dt>
<dd><a target="fedbizopps" href="ftp://ftp.fbo.gov/FBOFeed20181217"> FedBizOpps Link to This Notice</a><br>(may not be valid after Archive Date) </dd>
</dl>
In the above structure, you can see the <dt> tag refers to the title of data and the <dd> tags follows to it, will contain relevant data and data may consists of different types (i.e. email addresses, text, web links).
So far I can get up to <dl> tag using xpath conventions. So can someone suggest me a way to get these data separately into the given fields using xpath starts from <dl> tag.
html xpath scrapy
asked Jan 2 at 15:29
Vajira PrabuddhakaVajira Prabuddhaka
161317
I'm using Scrapy to scrape a website. I have the following html structure
<dl>
<dt><b>Notice Date</b></dt>
<dd>12/17/2018<br> </dd>
<dt><b>Notice Type</b></dt>
<dd>Synopsis<br> </dd>
<dt><b>NAICS</b></dt>
<dd>238990 — All Other Specialty Trade Contractors<br> </dd>
<dt><b>Contracting Office</b></dt>
<dd>1108 South R Street Fort McCoy WI 54656-5142<br> </dd>
<dt><b>ZIP Code</b></dt>
<dd>54656-5142<br> </dd>
<dt><b>Solicitation Number</b></dt>
<dd>W911SA-18-B-2028<br> </dd>
<dt><b>Point of Contact</b></dt>
<dd>Jamie Gregar, Contract Specialist, Phone 6083881007 <br> </dd>
<dt><b>E-Mail Address</b></dt>
<dd><a href="mailto:jamie.j.gregar.civ@mail.mil">jamie.j.gregar.civ@mail.mil</a<br> </dd>
<dt><b>Small Business Set-Aside</b></dt>
<dd>HUBZone<br> </dd>
<dt><b>Description</b></dt>
<dd>some description here<br> </dd>
<dt><b>Web Link</b></dt>
<dd><a target="outside" href="https://www.fbo.gov/notices/54ba577b59c07b3757739e138b20ef3c">
Link To Document</a> <br>(https://www.fbo.gov/notices/54ba577b59c07b3757739e138b20ef3c)
<br> </dd>
<dt><b>Place of Performance</b></dt>
<dd><i>Address:</i> Laredo, TX </dd>
<dd><i>Zip Code:</i> 78040 </dd>
<dd><i>Country:</i> US <br> </dd>
<dt><b>Record</b> </dt>
<dd>SN05176005-F 20181219/181217230006 (fbodaily.com) <br> </dd>
<dt><b>Source</b></dt>
<dd><a target="fedbizopps" href="ftp://ftp.fbo.gov/FBOFeed20181217"> FedBizOpps Link to This Notice</a><br>(may not be valid after Archive Date) </dd>
</dl>
In the above structure, you can see the <dt> tag refers to the title of data and the <dd> tags follows to it, will contain relevant data and data may consists of different types (i.e. email addresses, text, web links).
So far I can get up to <dl> tag using xpath conventions. So can someone suggest me a way to get these data separately into the given fields using xpath starts from <dl> tag.
html xpath scrapy
html xpath scrapy
asked Jan 2 at 15:29
Vajira PrabuddhakaVajira Prabuddhaka
161317
asked Jan 2 at 15:29
Vajira PrabuddhakaVajira Prabuddhaka
161317
asked Jan 2 at 15:29
Vajira PrabuddhakaVajira Prabuddhaka
161317
asked Jan 2 at 15:29
Vajira PrabuddhakaVajira Prabuddhaka
161317
asked Jan 2 at 15:29
Vajira PrabuddhakaVajira Prabuddhaka
161317
161317
2
Show us how you tried. Share your current and desired output
– Andersson
Jan 2 at 16:35
add a comment |
2
Show us how you tried. Share your current and desired output
– Andersson
Jan 2 at 16:35
2
2
Show us how you tried. Share your current and desired output
– Andersson
Jan 2 at 16:35
Show us how you tried. Share your current and desired output
– Andersson
Jan 2 at 16:35
add a comment |
1 Answer
1
active
oldest
votes
for dt in response.css('dt'):
key = dt.get()
value = dt.xpath('./following::dd[1]').get()
You might want to clean the data further, extracting only text either by using the .//text() XPath or extract_text().
answered Jan 16 at 11:42
GallaecioGallaecio
96011023
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54008984%2fhow-to-get-data-inside-dl-tag-using-xpath%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
for dt in response.css('dt'):
key = dt.get()
value = dt.xpath('./following::dd[1]').get()
You might want to clean the data further, extracting only text either by using the .//text() XPath or extract_text().
answered Jan 16 at 11:42
GallaecioGallaecio
96011023
add a comment |
for dt in response.css('dt'):
key = dt.get()
value = dt.xpath('./following::dd[1]').get()
You might want to clean the data further, extracting only text either by using the .//text() XPath or extract_text().
answered Jan 16 at 11:42
GallaecioGallaecio
96011023
add a comment |
for dt in response.css('dt'):
key = dt.get()
value = dt.xpath('./following::dd[1]').get()
You might want to clean the data further, extracting only text either by using the .//text() XPath or extract_text().
answered Jan 16 at 11:42
GallaecioGallaecio
96011023
for dt in response.css('dt'):
key = dt.get()
value = dt.xpath('./following::dd[1]').get()
You might want to clean the data further, extracting only text either by using the .//text() XPath or extract_text().
answered Jan 16 at 11:42
GallaecioGallaecio
96011023
answered Jan 16 at 11:42
GallaecioGallaecio
96011023
answered Jan 16 at 11:42
GallaecioGallaecio
96011023
answered Jan 16 at 11:42
GallaecioGallaecio
96011023
96011023
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54008984%2fhow-to-get-data-inside-dl-tag-using-xpath%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
2
Show us how you tried. Share your current and desired output
– Andersson
Jan 2 at 16:35