element-wise merge np.array in multiple pandas column

I got a pandas dataframe, in which there are several columns’ value are np.array， I would like to merge these np.arrays into one array elementwise based row.

e.g

 col1          col2        col3 

[2.1, 3]      [4, 4]      [2, 3] 

[4, 5]        [6, 7]      [9, 9] 

[7, 8]        [8, 9]      [5, 4] 

...             ...         ...

expected result:

col_f

[2.1, 3, 4, 4, 2, 3] 

[4, 5, 6, 7, 9, 9] 

[7, 8, 8, 9 5, 4]

........

I use kind of for loop to realize it, but just wondering if there is the more elegant way to do it.

below is my for loop cod:

f_vector = 

for i in range(len(df.index)):

    vector = np.hstack((df['A0_vector'][i], items_df['A1_vector'][i], items_df['A2_vector'][i], items_df['A3_vector'][i], items_df['A4_vector'][i], items_df['A5_vector'][i])) 

    f_vector.append(vector)

X = np.array(f_vector)

edited Nov 21 '18 at 1:30

Michal

9141021

asked Nov 21 '18 at 1:23

Kevin Li

How about a column_stack of the individual Series (or their .values)? What does df.values look like?

– hpaulj
Nov 21 '18 at 2:08

add a comment |

I got a pandas dataframe, in which there are several columns’ value are np.array， I would like to merge these np.arrays into one array elementwise based row.

e.g

 col1          col2        col3 

[2.1, 3]      [4, 4]      [2, 3] 

[4, 5]        [6, 7]      [9, 9] 

[7, 8]        [8, 9]      [5, 4] 

...             ...         ...

expected result:

col_f

[2.1, 3, 4, 4, 2, 3] 

[4, 5, 6, 7, 9, 9] 

[7, 8, 8, 9 5, 4]

........

I use kind of for loop to realize it, but just wondering if there is the more elegant way to do it.

below is my for loop cod:

f_vector = 

for i in range(len(df.index)):

    vector = np.hstack((df['A0_vector'][i], items_df['A1_vector'][i], items_df['A2_vector'][i], items_df['A3_vector'][i], items_df['A4_vector'][i], items_df['A5_vector'][i])) 

    f_vector.append(vector)

X = np.array(f_vector)

edited Nov 21 '18 at 1:30

Michal

9141021

asked Nov 21 '18 at 1:23

Kevin Li

How about a column_stack of the individual Series (or their .values)? What does df.values look like?

– hpaulj
Nov 21 '18 at 2:08

add a comment |

I got a pandas dataframe, in which there are several columns’ value are np.array， I would like to merge these np.arrays into one array elementwise based row.

e.g

 col1          col2        col3 

[2.1, 3]      [4, 4]      [2, 3] 

[4, 5]        [6, 7]      [9, 9] 

[7, 8]        [8, 9]      [5, 4] 

...             ...         ...

expected result:

col_f

[2.1, 3, 4, 4, 2, 3] 

[4, 5, 6, 7, 9, 9] 

[7, 8, 8, 9 5, 4]

........

I use kind of for loop to realize it, but just wondering if there is the more elegant way to do it.

below is my for loop cod:

f_vector = 

for i in range(len(df.index)):

    vector = np.hstack((df['A0_vector'][i], items_df['A1_vector'][i], items_df['A2_vector'][i], items_df['A3_vector'][i], items_df['A4_vector'][i], items_df['A5_vector'][i])) 

    f_vector.append(vector)

X = np.array(f_vector)

edited Nov 21 '18 at 1:30

Michal

9141021

asked Nov 21 '18 at 1:23

Kevin Li

I got a pandas dataframe, in which there are several columns’ value are np.array， I would like to merge these np.arrays into one array elementwise based row.

e.g

 col1          col2        col3 

[2.1, 3]      [4, 4]      [2, 3] 

[4, 5]        [6, 7]      [9, 9] 

[7, 8]        [8, 9]      [5, 4] 

...             ...         ...

expected result:

col_f

[2.1, 3, 4, 4, 2, 3] 

[4, 5, 6, 7, 9, 9] 

[7, 8, 8, 9 5, 4]

........

I use kind of for loop to realize it, but just wondering if there is the more elegant way to do it.

below is my for loop cod:

f_vector = 

for i in range(len(df.index)):

    vector = np.hstack((df['A0_vector'][i], items_df['A1_vector'][i], items_df['A2_vector'][i], items_df['A3_vector'][i], items_df['A4_vector'][i], items_df['A5_vector'][i])) 

    f_vector.append(vector)

X = np.array(f_vector)

python pandas numpy

edited Nov 21 '18 at 1:30

Michal

9141021

asked Nov 21 '18 at 1:23

Kevin Li

edited Nov 21 '18 at 1:30

Michal

9141021

asked Nov 21 '18 at 1:23

Kevin Li

edited Nov 21 '18 at 1:30

Michal

9141021

edited Nov 21 '18 at 1:30

Michal

9141021

edited Nov 21 '18 at 1:30

Michal

9141021

asked Nov 21 '18 at 1:23

Kevin Li

asked Nov 21 '18 at 1:23

Kevin Li

asked Nov 21 '18 at 1:23

Kevin Li

How about a column_stack of the individual Series (or their .values)? What does df.values look like?

– hpaulj
Nov 21 '18 at 2:08

add a comment |

How about a column_stack of the individual Series (or their .values)? What does df.values look like?

– hpaulj
Nov 21 '18 at 2:08

How about a column_stack of the individual Series (or their .values)? What does df.values look like?

– hpaulj
Nov 21 '18 at 2:08

add a comment |

1 Answer
1

active

oldest

votes

You can use numpy.concatenate with apply along axis=1:

import numpy as np

df['col_f'] = df[['col1', 'col2', 'col3']].apply(np.concatenate, axis=1)

If those were lists instead of np.arrays, + operator would have worked:

df['col_f'] = df['col1'] + df['col2'] + + df['col3']

Note: edited after comments thread below.

edited Nov 21 '18 at 2:18

answered Nov 21 '18 at 1:30

Julian Peller

8941511

Hi Julian, if use + operator, these arrays will plus together, I don't want them to be added, just like kind of extend. e.g the array in each column has 5 elements, once merged the 5 columns, then the total element in the new array will be 25 elements. if we use + operator, the final array will have only 5 elements only, right?

– Kevin Li
Nov 21 '18 at 1:34

No, it's concatenating the arrays as you want, not adding the elements. It will have the 25 elements.

– Julian Peller
Nov 21 '18 at 1:41

At least for pandas version 0.22.0. Give it a shot and let me know if it doesn't work. I just reproduced the same example you wanted: [2.1, 3] [4, 4] [2, 3] ==> [2.1, 3, 4, 4, 2, 3] with the line I posted.

– Julian Peller
Nov 21 '18 at 1:43

No, I am using pandas 0.23, it's addition, not the way I want.

– Kevin Li
Nov 21 '18 at 1:50

That's weird. What about df['col_f'] = df[['col1', 'col2', 'col3']].sum(axis=1)?

– Julian Peller
Nov 21 '18 at 2:00

|
show 4 more comments

Your Answer

StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});

}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53404010%2felement-wise-merge-np-array-in-multiple-pandas-column%23new-answer', 'question_page');
}
);

Post as a guest

Name

Required, but never shown

1 Answer
1

active

oldest

votes

1 Answer
1

active

oldest

votes

You can use numpy.concatenate with apply along axis=1:

import numpy as np

df['col_f'] = df[['col1', 'col2', 'col3']].apply(np.concatenate, axis=1)

If those were lists instead of np.arrays, + operator would have worked:

df['col_f'] = df['col1'] + df['col2'] + + df['col3']

Note: edited after comments thread below.

edited Nov 21 '18 at 2:18

answered Nov 21 '18 at 1:30

Julian Peller

8941511

Hi Julian, if use + operator, these arrays will plus together, I don't want them to be added, just like kind of extend. e.g the array in each column has 5 elements, once merged the 5 columns, then the total element in the new array will be 25 elements. if we use + operator, the final array will have only 5 elements only, right?

– Kevin Li
Nov 21 '18 at 1:34

No, it's concatenating the arrays as you want, not adding the elements. It will have the 25 elements.

– Julian Peller
Nov 21 '18 at 1:41

At least for pandas version 0.22.0. Give it a shot and let me know if it doesn't work. I just reproduced the same example you wanted: [2.1, 3] [4, 4] [2, 3] ==> [2.1, 3, 4, 4, 2, 3] with the line I posted.

– Julian Peller
Nov 21 '18 at 1:43

No, I am using pandas 0.23, it's addition, not the way I want.

– Kevin Li
Nov 21 '18 at 1:50

That's weird. What about df['col_f'] = df[['col1', 'col2', 'col3']].sum(axis=1)?

– Julian Peller
Nov 21 '18 at 2:00

|
show 4 more comments

You can use numpy.concatenate with apply along axis=1:

import numpy as np

df['col_f'] = df[['col1', 'col2', 'col3']].apply(np.concatenate, axis=1)

If those were lists instead of np.arrays, + operator would have worked:

df['col_f'] = df['col1'] + df['col2'] + + df['col3']

Note: edited after comments thread below.

edited Nov 21 '18 at 2:18

answered Nov 21 '18 at 1:30

Julian Peller

8941511

Hi Julian, if use + operator, these arrays will plus together, I don't want them to be added, just like kind of extend. e.g the array in each column has 5 elements, once merged the 5 columns, then the total element in the new array will be 25 elements. if we use + operator, the final array will have only 5 elements only, right?

– Kevin Li
Nov 21 '18 at 1:34

No, it's concatenating the arrays as you want, not adding the elements. It will have the 25 elements.

– Julian Peller
Nov 21 '18 at 1:41

At least for pandas version 0.22.0. Give it a shot and let me know if it doesn't work. I just reproduced the same example you wanted: [2.1, 3] [4, 4] [2, 3] ==> [2.1, 3, 4, 4, 2, 3] with the line I posted.

– Julian Peller
Nov 21 '18 at 1:43

No, I am using pandas 0.23, it's addition, not the way I want.

– Kevin Li
Nov 21 '18 at 1:50

That's weird. What about df['col_f'] = df[['col1', 'col2', 'col3']].sum(axis=1)?

– Julian Peller
Nov 21 '18 at 2:00

|
show 4 more comments

You can use numpy.concatenate with apply along axis=1:

import numpy as np

df['col_f'] = df[['col1', 'col2', 'col3']].apply(np.concatenate, axis=1)

If those were lists instead of np.arrays, + operator would have worked:

df['col_f'] = df['col1'] + df['col2'] + + df['col3']

Note: edited after comments thread below.

edited Nov 21 '18 at 2:18

answered Nov 21 '18 at 1:30

Julian Peller

8941511

You can use numpy.concatenate with apply along axis=1:

import numpy as np

df['col_f'] = df[['col1', 'col2', 'col3']].apply(np.concatenate, axis=1)

If those were lists instead of np.arrays, + operator would have worked:

df['col_f'] = df['col1'] + df['col2'] + + df['col3']

Note: edited after comments thread below.

edited Nov 21 '18 at 2:18

answered Nov 21 '18 at 1:30

Julian Peller

8941511

edited Nov 21 '18 at 2:18

answered Nov 21 '18 at 1:30

Julian Peller

8941511

answered Nov 21 '18 at 1:30

Julian Peller

8941511

answered Nov 21 '18 at 1:30

Julian Peller

8941511

Hi Julian, if use + operator, these arrays will plus together, I don't want them to be added, just like kind of extend. e.g the array in each column has 5 elements, once merged the 5 columns, then the total element in the new array will be 25 elements. if we use + operator, the final array will have only 5 elements only, right?

– Kevin Li
Nov 21 '18 at 1:34

No, it's concatenating the arrays as you want, not adding the elements. It will have the 25 elements.

– Julian Peller
Nov 21 '18 at 1:41

At least for pandas version 0.22.0. Give it a shot and let me know if it doesn't work. I just reproduced the same example you wanted: [2.1, 3] [4, 4] [2, 3] ==> [2.1, 3, 4, 4, 2, 3] with the line I posted.

– Julian Peller
Nov 21 '18 at 1:43

No, I am using pandas 0.23, it's addition, not the way I want.

– Kevin Li
Nov 21 '18 at 1:50

That's weird. What about df['col_f'] = df[['col1', 'col2', 'col3']].sum(axis=1)?

– Julian Peller
Nov 21 '18 at 2:00

|
show 4 more comments

Hi Julian, if use + operator, these arrays will plus together, I don't want them to be added, just like kind of extend. e.g the array in each column has 5 elements, once merged the 5 columns, then the total element in the new array will be 25 elements. if we use + operator, the final array will have only 5 elements only, right?

– Kevin Li
Nov 21 '18 at 1:34

No, it's concatenating the arrays as you want, not adding the elements. It will have the 25 elements.

– Julian Peller
Nov 21 '18 at 1:41

At least for pandas version 0.22.0. Give it a shot and let me know if it doesn't work. I just reproduced the same example you wanted: [2.1, 3] [4, 4] [2, 3] ==> [2.1, 3, 4, 4, 2, 3] with the line I posted.

– Julian Peller
Nov 21 '18 at 1:43

No, I am using pandas 0.23, it's addition, not the way I want.

– Kevin Li
Nov 21 '18 at 1:50

That's weird. What about df['col_f'] = df[['col1', 'col2', 'col3']].sum(axis=1)?

– Julian Peller
Nov 21 '18 at 2:00

Hi Julian, if use + operator, these arrays will plus together, I don't want them to be added, just like kind of extend. e.g the array in each column has 5 elements, once merged the 5 columns, then the total element in the new array will be 25 elements. if we use + operator, the final array will have only 5 elements only, right?

– Kevin Li
Nov 21 '18 at 1:34

No, it's concatenating the arrays as you want, not adding the elements. It will have the 25 elements.

– Julian Peller
Nov 21 '18 at 1:41

At least for pandas version 0.22.0. Give it a shot and let me know if it doesn't work. I just reproduced the same example you wanted: [2.1, 3] [4, 4] [2, 3] ==> [2.1, 3, 4, 4, 2, 3] with the line I posted.

– Julian Peller
Nov 21 '18 at 1:43

No, I am using pandas 0.23, it's addition, not the way I want.

– Kevin Li
Nov 21 '18 at 1:50

That's weird. What about df['col_f'] = df[['col1', 'col2', 'col3']].sum(axis=1)?

– Julian Peller
Nov 21 '18 at 2:00

|
show 4 more comments

draft saved

draft discarded

Thanks for contributing an answer to Stack Overflow!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

Search This Blog

Ufyukyu