Mixing
Date formatting and including days
I have a csv file which has column of dates (D/M/Y format) and I want to convert it to days column. I used following approach:
In [1]: import numpy as np
import pandas as pd
from pandas import Series, DataFrame
#f = pd.read_csv(".some_file.csv")
In [2]: f=pd.DataFrame([['1/1/2013', 400, 1000]['2/1/2013', 500, 3000]], columns=['date','flights','distance'])
f['date']=pd.to_datetime(f['date'], format='%d/%m/%y',errors='ignore')
f['day']=f['date'].dt.weekday_name
I was expecting the day column. It appeared for the first time and I used that table too. However after clearing kernel I am getting NaT in date and NaN in day. After that days never appeared.
Am I doing anything wrong? If yes, how did day appeared for the first time?
Thanks for reading. Any help will be appreciated.
PS CSV has more than 330K rows. I have to assign day for each date.
python pandas datetime
asked Nov 22 '18 at 9:56
Nikhil JagtapNikhil Jagtap
16
add a comment |
I have a csv file which has column of dates (D/M/Y format) and I want to convert it to days column. I used following approach:
In [1]: import numpy as np
import pandas as pd
from pandas import Series, DataFrame
#f = pd.read_csv(".some_file.csv")
In [2]: f=pd.DataFrame([['1/1/2013', 400, 1000]['2/1/2013', 500, 3000]], columns=['date','flights','distance'])
f['date']=pd.to_datetime(f['date'], format='%d/%m/%y',errors='ignore')
f['day']=f['date'].dt.weekday_name
I was expecting the day column. It appeared for the first time and I used that table too. However after clearing kernel I am getting NaT in date and NaN in day. After that days never appeared.
Am I doing anything wrong? If yes, how did day appeared for the first time?
Thanks for reading. Any help will be appreciated.
PS CSV has more than 330K rows. I have to assign day for each date.
python pandas datetime
asked Nov 22 '18 at 9:56
Nikhil JagtapNikhil Jagtap
16
add a comment |
I have a csv file which has column of dates (D/M/Y format) and I want to convert it to days column. I used following approach:
In [1]: import numpy as np
import pandas as pd
from pandas import Series, DataFrame
#f = pd.read_csv(".some_file.csv")
In [2]: f=pd.DataFrame([['1/1/2013', 400, 1000]['2/1/2013', 500, 3000]], columns=['date','flights','distance'])
f['date']=pd.to_datetime(f['date'], format='%d/%m/%y',errors='ignore')
f['day']=f['date'].dt.weekday_name
I was expecting the day column. It appeared for the first time and I used that table too. However after clearing kernel I am getting NaT in date and NaN in day. After that days never appeared.
Am I doing anything wrong? If yes, how did day appeared for the first time?
Thanks for reading. Any help will be appreciated.
PS CSV has more than 330K rows. I have to assign day for each date.
python pandas datetime
asked Nov 22 '18 at 9:56
Nikhil JagtapNikhil Jagtap
16
I have a csv file which has column of dates (D/M/Y format) and I want to convert it to days column. I used following approach:
In [1]: import numpy as np
import pandas as pd
from pandas import Series, DataFrame
#f = pd.read_csv(".some_file.csv")
In [2]: f=pd.DataFrame([['1/1/2013', 400, 1000]['2/1/2013', 500, 3000]], columns=['date','flights','distance'])
f['date']=pd.to_datetime(f['date'], format='%d/%m/%y',errors='ignore')
f['day']=f['date'].dt.weekday_name
I was expecting the day column. It appeared for the first time and I used that table too. However after clearing kernel I am getting NaT in date and NaN in day. After that days never appeared.
Am I doing anything wrong? If yes, how did day appeared for the first time?
Thanks for reading. Any help will be appreciated.
PS CSV has more than 330K rows. I have to assign day for each date.
python pandas datetime
python pandas datetime
asked Nov 22 '18 at 9:56
Nikhil JagtapNikhil Jagtap
16
asked Nov 22 '18 at 9:56
Nikhil JagtapNikhil Jagtap
16
edited Nov 22 '18 at 10:27
Nikhil Jagtap
asked Nov 22 '18 at 9:56
Nikhil JagtapNikhil Jagtap
16
asked Nov 22 '18 at 9:56
Nikhil JagtapNikhil Jagtap
16
asked Nov 22 '18 at 9:56
Nikhil JagtapNikhil Jagtap
16
16
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
The problem is you have incorrectly specified the datetime format and hidden the error by setting errors='ignore'. You need '%Y' for the full year (see Python's strftime directives):
f['date'] = pd.to_datetime(f['date'], format='%d/%m/%Y', errors='coerce')
errors='coerce' will give NaN for non-convertible dates. Or, to raise errors, just omit the errors parameter altogether.
For such problems, it's good practice to look at f.dtypes to see whether your type conversion has succeeded. For example, you should see:
print(f['date'].dtype)
# datetime64[ns]
You should not see:
print(f['date'].dtype)
# object
answered Nov 22 '18 at 10:27
jppjpp
101k2163112
Thank you. It worked. :)
– Nikhil Jagtap
Nov 22 '18 at 10:32
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53428258%2fdate-formatting-and-including-days%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
The problem is you have incorrectly specified the datetime format and hidden the error by setting errors='ignore'. You need '%Y' for the full year (see Python's strftime directives):
f['date'] = pd.to_datetime(f['date'], format='%d/%m/%Y', errors='coerce')
errors='coerce' will give NaN for non-convertible dates. Or, to raise errors, just omit the errors parameter altogether.
For such problems, it's good practice to look at f.dtypes to see whether your type conversion has succeeded. For example, you should see:
print(f['date'].dtype)
# datetime64[ns]
You should not see:
print(f['date'].dtype)
# object
answered Nov 22 '18 at 10:27
jppjpp
101k2163112
Thank you. It worked. :)
– Nikhil Jagtap
Nov 22 '18 at 10:32
add a comment |
The problem is you have incorrectly specified the datetime format and hidden the error by setting errors='ignore'. You need '%Y' for the full year (see Python's strftime directives):
f['date'] = pd.to_datetime(f['date'], format='%d/%m/%Y', errors='coerce')
errors='coerce' will give NaN for non-convertible dates. Or, to raise errors, just omit the errors parameter altogether.
For such problems, it's good practice to look at f.dtypes to see whether your type conversion has succeeded. For example, you should see:
print(f['date'].dtype)
# datetime64[ns]
You should not see:
print(f['date'].dtype)
# object
answered Nov 22 '18 at 10:27
jppjpp
101k2163112
Thank you. It worked. :)
– Nikhil Jagtap
Nov 22 '18 at 10:32
add a comment |
The problem is you have incorrectly specified the datetime format and hidden the error by setting errors='ignore'. You need '%Y' for the full year (see Python's strftime directives):
f['date'] = pd.to_datetime(f['date'], format='%d/%m/%Y', errors='coerce')
errors='coerce' will give NaN for non-convertible dates. Or, to raise errors, just omit the errors parameter altogether.
For such problems, it's good practice to look at f.dtypes to see whether your type conversion has succeeded. For example, you should see:
print(f['date'].dtype)
# datetime64[ns]
You should not see:
print(f['date'].dtype)
# object
answered Nov 22 '18 at 10:27
jppjpp
101k2163112
The problem is you have incorrectly specified the datetime format and hidden the error by setting errors='ignore'. You need '%Y' for the full year (see Python's strftime directives):
f['date'] = pd.to_datetime(f['date'], format='%d/%m/%Y', errors='coerce')
errors='coerce' will give NaN for non-convertible dates. Or, to raise errors, just omit the errors parameter altogether.
For such problems, it's good practice to look at f.dtypes to see whether your type conversion has succeeded. For example, you should see:
print(f['date'].dtype)
# datetime64[ns]
You should not see:
print(f['date'].dtype)
# object
answered Nov 22 '18 at 10:27
jppjpp
101k2163112
answered Nov 22 '18 at 10:27
jppjpp
101k2163112
answered Nov 22 '18 at 10:27
jppjpp
101k2163112
answered Nov 22 '18 at 10:27
jppjpp
101k2163112
101k2163112
Thank you. It worked. :)
– Nikhil Jagtap
Nov 22 '18 at 10:32
add a comment |
Thank you. It worked. :)
– Nikhil Jagtap
Nov 22 '18 at 10:32
Thank you. It worked. :)
– Nikhil Jagtap
Nov 22 '18 at 10:32
Thank you. It worked. :)
– Nikhil Jagtap
Nov 22 '18 at 10:32
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53428258%2fdate-formatting-and-including-days%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
