W2VTransformer: Only works with one word as input?

-1

Following reproducible script is used to compute the accuracy of a Word2Vec classifier with the W2VTransformer wrapper in gensim:

import numpy as np

import pandas as pd

from sklearn.linear_model import LogisticRegression

from sklearn.pipeline import Pipeline

from gensim.sklearn_api import W2VTransformer

from gensim.utils import simple_preprocess



# Load synthetic data

data = pd.read_csv('https://pastebin.com/raw/EPCmabvN')

data = data.head(10)



# Set random seed

np.random.seed(0)



# Tokenize text

X_train = data.apply(lambda r: simple_preprocess(r['text'], min_len=2), axis=1)

# Get labels

y_train = data.label



train_input = [x[0] for x in X_train]



# Train W2V Model

model = W2VTransformer(size=10, min_count=1)

model.fit(X_train)



clf = LogisticRegression(penalty='l2', C=0.1)

clf.fit(model.transform(train_input), y_train)



text_w2v = Pipeline(

    [('features', model),

     ('classifier', clf)])



score = text_w2v.score(train_input, y_train)

score

0.80000000000000004

The problem with this script is that it only works when train_input = [x[0] for x in X_train], which essentially is always the first word only.
Once change to train_input = X_train (or train_input simply substituted by X_train), the script returns:

ValueError: cannot reshape array of size 10 into shape (10,10)

How can I solve this issue, i.e. how can the classifier work with more than one word of input?

Edit:

Apparently, the W2V wrapper can't work with the variable-length train input, as compared to D2V. Here is a working D2V version:

import numpy as np

import pandas as pd

from sklearn.linear_model import LogisticRegression

from sklearn.model_selection import cross_val_score

from sklearn.metrics import accuracy_score, classification_report

from sklearn.pipeline import Pipeline

from gensim.utils import simple_preprocess, lemmatize

from gensim.sklearn_api import D2VTransformer



data = pd.read_csv('https://pastebin.com/raw/bSGWiBfs')



np.random.seed(0)



X_train = data.apply(lambda r: simple_preprocess(r['text'], min_len=2), axis=1)

y_train = data.label



model = D2VTransformer(dm=1, size=50, min_count=2, iter=10, seed=0)

model.fit(X_train)



clf = LogisticRegression(penalty='l2', C=0.1, random_state=0)

clf.fit(model.transform(X_train), y_train)



pipeline = Pipeline([

        ('vec', model),

        ('clf', clf)

    ])



y_pred = pipeline.predict(X_train)

score = accuracy_score(y_train,y_pred)

print(score)

edited Jan 2 at 9:13

asked Jan 1 at 19:47

Christopher

4021823

1

Where does the script return that ValueError? (It's much easier to see what's going wrong if you can show the full error stack, so you should edit the question to include that extra level of detail.)

– gojomo
Jan 1 at 23:20

re: your update Yes, the W2VTransformer does not collapse a variable-length list-of-words to a single vector, as that's not an automatically-desirable capability of the wrapped Word2Vec model. Instead, it converts a variable-length list-of-words to a same-length list-of-vectors. If you need to collapse those to a single vector for later steps, you could implement that as a following transformer, perhaps one that averages all the vectors together. (That often works as a simple baseline approach, though other weightings or algorithms might work better depending on your data & goals.)

– gojomo
Jan 2 at 18:24

add a comment |

-1

Following reproducible script is used to compute the accuracy of a Word2Vec classifier with the W2VTransformer wrapper in gensim:

import numpy as np

import pandas as pd

from sklearn.linear_model import LogisticRegression

from sklearn.pipeline import Pipeline

from gensim.sklearn_api import W2VTransformer

from gensim.utils import simple_preprocess



# Load synthetic data

data = pd.read_csv('https://pastebin.com/raw/EPCmabvN')

data = data.head(10)



# Set random seed

np.random.seed(0)



# Tokenize text

X_train = data.apply(lambda r: simple_preprocess(r['text'], min_len=2), axis=1)

# Get labels

y_train = data.label



train_input = [x[0] for x in X_train]



# Train W2V Model

model = W2VTransformer(size=10, min_count=1)

model.fit(X_train)



clf = LogisticRegression(penalty='l2', C=0.1)

clf.fit(model.transform(train_input), y_train)



text_w2v = Pipeline(

    [('features', model),

     ('classifier', clf)])



score = text_w2v.score(train_input, y_train)

score

0.80000000000000004

ValueError: cannot reshape array of size 10 into shape (10,10)

How can I solve this issue, i.e. how can the classifier work with more than one word of input?

Edit:

Apparently, the W2V wrapper can't work with the variable-length train input, as compared to D2V. Here is a working D2V version:

import numpy as np

import pandas as pd

from sklearn.linear_model import LogisticRegression

from sklearn.model_selection import cross_val_score

from sklearn.metrics import accuracy_score, classification_report

from sklearn.pipeline import Pipeline

from gensim.utils import simple_preprocess, lemmatize

from gensim.sklearn_api import D2VTransformer



data = pd.read_csv('https://pastebin.com/raw/bSGWiBfs')



np.random.seed(0)



X_train = data.apply(lambda r: simple_preprocess(r['text'], min_len=2), axis=1)

y_train = data.label



model = D2VTransformer(dm=1, size=50, min_count=2, iter=10, seed=0)

model.fit(X_train)



clf = LogisticRegression(penalty='l2', C=0.1, random_state=0)

clf.fit(model.transform(X_train), y_train)



pipeline = Pipeline([

        ('vec', model),

        ('clf', clf)

    ])



y_pred = pipeline.predict(X_train)

score = accuracy_score(y_train,y_pred)

print(score)

edited Jan 2 at 9:13

asked Jan 1 at 19:47

Christopher

4021823

1

Where does the script return that ValueError? (It's much easier to see what's going wrong if you can show the full error stack, so you should edit the question to include that extra level of detail.)

– gojomo
Jan 1 at 23:20

re: your update Yes, the W2VTransformer does not collapse a variable-length list-of-words to a single vector, as that's not an automatically-desirable capability of the wrapped Word2Vec model. Instead, it converts a variable-length list-of-words to a same-length list-of-vectors. If you need to collapse those to a single vector for later steps, you could implement that as a following transformer, perhaps one that averages all the vectors together. (That often works as a simple baseline approach, though other weightings or algorithms might work better depending on your data & goals.)

– gojomo
Jan 2 at 18:24

add a comment |

-1

Following reproducible script is used to compute the accuracy of a Word2Vec classifier with the W2VTransformer wrapper in gensim:

import numpy as np

import pandas as pd

from sklearn.linear_model import LogisticRegression

from sklearn.pipeline import Pipeline

from gensim.sklearn_api import W2VTransformer

from gensim.utils import simple_preprocess



# Load synthetic data

data = pd.read_csv('https://pastebin.com/raw/EPCmabvN')

data = data.head(10)



# Set random seed

np.random.seed(0)



# Tokenize text

X_train = data.apply(lambda r: simple_preprocess(r['text'], min_len=2), axis=1)

# Get labels

y_train = data.label



train_input = [x[0] for x in X_train]



# Train W2V Model

model = W2VTransformer(size=10, min_count=1)

model.fit(X_train)



clf = LogisticRegression(penalty='l2', C=0.1)

clf.fit(model.transform(train_input), y_train)



text_w2v = Pipeline(

    [('features', model),

     ('classifier', clf)])



score = text_w2v.score(train_input, y_train)

score

0.80000000000000004

ValueError: cannot reshape array of size 10 into shape (10,10)

How can I solve this issue, i.e. how can the classifier work with more than one word of input?

Edit:

Apparently, the W2V wrapper can't work with the variable-length train input, as compared to D2V. Here is a working D2V version:

import numpy as np

import pandas as pd

from sklearn.linear_model import LogisticRegression

from sklearn.model_selection import cross_val_score

from sklearn.metrics import accuracy_score, classification_report

from sklearn.pipeline import Pipeline

from gensim.utils import simple_preprocess, lemmatize

from gensim.sklearn_api import D2VTransformer



data = pd.read_csv('https://pastebin.com/raw/bSGWiBfs')



np.random.seed(0)



X_train = data.apply(lambda r: simple_preprocess(r['text'], min_len=2), axis=1)

y_train = data.label



model = D2VTransformer(dm=1, size=50, min_count=2, iter=10, seed=0)

model.fit(X_train)



clf = LogisticRegression(penalty='l2', C=0.1, random_state=0)

clf.fit(model.transform(X_train), y_train)



pipeline = Pipeline([

        ('vec', model),

        ('clf', clf)

    ])



y_pred = pipeline.predict(X_train)

score = accuracy_score(y_train,y_pred)

print(score)

edited Jan 2 at 9:13

asked Jan 1 at 19:47

Christopher

4021823

Following reproducible script is used to compute the accuracy of a Word2Vec classifier with the W2VTransformer wrapper in gensim:

import numpy as np

import pandas as pd

from sklearn.linear_model import LogisticRegression

from sklearn.pipeline import Pipeline

from gensim.sklearn_api import W2VTransformer

from gensim.utils import simple_preprocess



# Load synthetic data

data = pd.read_csv('https://pastebin.com/raw/EPCmabvN')

data = data.head(10)



# Set random seed

np.random.seed(0)



# Tokenize text

X_train = data.apply(lambda r: simple_preprocess(r['text'], min_len=2), axis=1)

# Get labels

y_train = data.label



train_input = [x[0] for x in X_train]



# Train W2V Model

model = W2VTransformer(size=10, min_count=1)

model.fit(X_train)



clf = LogisticRegression(penalty='l2', C=0.1)

clf.fit(model.transform(train_input), y_train)



text_w2v = Pipeline(

    [('features', model),

     ('classifier', clf)])



score = text_w2v.score(train_input, y_train)

score

0.80000000000000004

ValueError: cannot reshape array of size 10 into shape (10,10)

How can I solve this issue, i.e. how can the classifier work with more than one word of input?

Edit:

Apparently, the W2V wrapper can't work with the variable-length train input, as compared to D2V. Here is a working D2V version:

import numpy as np

import pandas as pd

from sklearn.linear_model import LogisticRegression

from sklearn.model_selection import cross_val_score

from sklearn.metrics import accuracy_score, classification_report

from sklearn.pipeline import Pipeline

from gensim.utils import simple_preprocess, lemmatize

from gensim.sklearn_api import D2VTransformer



data = pd.read_csv('https://pastebin.com/raw/bSGWiBfs')



np.random.seed(0)



X_train = data.apply(lambda r: simple_preprocess(r['text'], min_len=2), axis=1)

y_train = data.label



model = D2VTransformer(dm=1, size=50, min_count=2, iter=10, seed=0)

model.fit(X_train)



clf = LogisticRegression(penalty='l2', C=0.1, random_state=0)

clf.fit(model.transform(X_train), y_train)



pipeline = Pipeline([

        ('vec', model),

        ('clf', clf)

    ])



y_pred = pipeline.predict(X_train)

score = accuracy_score(y_train,y_pred)

print(score)

scikit-learn gensim word2vec

edited Jan 2 at 9:13

asked Jan 1 at 19:47

Christopher

4021823

edited Jan 2 at 9:13

asked Jan 1 at 19:47

Christopher

4021823

edited Jan 2 at 9:13

asked Jan 1 at 19:47

Christopher

4021823

asked Jan 1 at 19:47

Christopher

4021823

asked Jan 1 at 19:47

Christopher

4021823

1

Where does the script return that ValueError? (It's much easier to see what's going wrong if you can show the full error stack, so you should edit the question to include that extra level of detail.)

– gojomo
Jan 1 at 23:20

re: your update Yes, the W2VTransformer does not collapse a variable-length list-of-words to a single vector, as that's not an automatically-desirable capability of the wrapped Word2Vec model. Instead, it converts a variable-length list-of-words to a same-length list-of-vectors. If you need to collapse those to a single vector for later steps, you could implement that as a following transformer, perhaps one that averages all the vectors together. (That often works as a simple baseline approach, though other weightings or algorithms might work better depending on your data & goals.)

– gojomo
Jan 2 at 18:24

add a comment |

1

Where does the script return that ValueError? (It's much easier to see what's going wrong if you can show the full error stack, so you should edit the question to include that extra level of detail.)

– gojomo
Jan 1 at 23:20

re: your update Yes, the W2VTransformer does not collapse a variable-length list-of-words to a single vector, as that's not an automatically-desirable capability of the wrapped Word2Vec model. Instead, it converts a variable-length list-of-words to a same-length list-of-vectors. If you need to collapse those to a single vector for later steps, you could implement that as a following transformer, perhaps one that averages all the vectors together. (That often works as a simple baseline approach, though other weightings or algorithms might work better depending on your data & goals.)

– gojomo
Jan 2 at 18:24

Where does the script return that ValueError? (It's much easier to see what's going wrong if you can show the full error stack, so you should edit the question to include that extra level of detail.)

– gojomo
Jan 1 at 23:20

re: your update Yes, the W2VTransformer does not collapse a variable-length list-of-words to a single vector, as that's not an automatically-desirable capability of the wrapped Word2Vec model. Instead, it converts a variable-length list-of-words to a same-length list-of-vectors. If you need to collapse those to a single vector for later steps, you could implement that as a following transformer, perhaps one that averages all the vectors together. (That often works as a simple baseline approach, though other weightings or algorithms might work better depending on your data & goals.)

– gojomo
Jan 2 at 18:24

add a comment |

1 Answer
1

active

oldest

votes

This is technically not an answer, but cannot be written in comments so here it is. There are multiple issues here:

LogisticRegression class (and most other scikit-learn models) work with 2-d data (n_samples, n_features).

That means that it needs a collection of 1-d arrays (one for each row (sample), in which the elements of array contains the feature values).

In your data, a single word will be a 1-d array, which means that the single sentence (sample) will be a 2-d array. Which means that the complete data (collection of sentences here) will be a collection of 2-d arrays. Even in that, since each sentence can have different number of words, it cannot be combined into a single 3-d array.

Secondly, the W2VTransformer in gensim looks like a scikit-learn compatible class, but its not. It tries to follows "scikit-learn API conventions" for defining the methods fit(), fit_transform() and transform(). They are not compatible with scikit-learn Pipeline.

You can see that the input param requirements of fit() and fit_transform() are different.
- fit():
  
  X (iterable of iterables of str) – The input corpus.
  
  X can be simply a list of lists of tokens, but for larger corpora, consider an iterable that streams the sentences directly from
  disk/network. See BrownCorpus, Text8Corpus or LineSentence in word2vec
  module for such examples.
- fit_transform():
  
  X (numpy array of shape [n_samples, n_features]) – Training set.

If you want to use scikit-learn, then you will need to have the 2-d shape. You will need to "somehow merge" word-vectors for a single sentence to form a 1-d array for that sentence. That means that you need to form a kind of sentence-vector, by doing:

sum of individual words

average of individual words

weighted averaging of individual words based on frequency, tf-idf etc.

using other techniques like sent2vec, paragraph2vec, doc2vec etc.

Note:- I noticed now that you were doing this thing based on D2VTransformer. That should be the correct approach here if you want to use sklearn.

The issue in that question was this line (since that question is now deleted):

X_train = vectorizer.fit_transform(X_train)

Here, you overwrite your original X_train (list of list of words) with already calculated word vectors and hence that error.

Or else, you can use other tools / libraries (keras, tensorflow) which allow sequential input of variable size. For example, LSTMs can be configured here to take a variable input and an ending token to mark the end of sentence (a sample).

Update:

In the above given solution, you can replace the lines:

model = D2VTransformer(dm=1, size=50, min_count=2, iter=10, seed=0)

model.fit(X_train)



clf = LogisticRegression(penalty='l2', C=0.1, random_state=0)

clf.fit(model.transform(X_train), y_train)



pipeline = Pipeline([

        ('vec', model),

        ('clf', clf)

    ])



y_pred = pipeline.predict(X_train)

with

pipeline = Pipeline([

        ('vec', model),

        ('clf', clf)

    ])



pipeline.fit(X_train, y_train)

y_pred = pipeline.predict(X_train)

No need to fit and transform separately, since pipeline.fit() will automatically do that.

edited Jan 2 at 9:36

answered Jan 2 at 7:58

Vivek Kumar

16.5k42156

Yes, apparently the D2V version seems to work.. I also added a working version above. Thanks for the answer!

– Christopher
Jan 2 at 9:10

@Christopher I have updated the answer with the error in that question.

– Vivek Kumar
Jan 2 at 9:18

Where exact? It seems to work. I think model.transform(X_train)solved it?

– Christopher
Jan 2 at 9:26

@Christopher The error about "ufunc 'add' did not contain a loop with ...". It will happen when you combine both the codes active (D2V Example - Without Pipeline and D2V Example - With Pipeline). In that case, the X_train that gets passed to pipeline is already doc vectors. If you remove the without pipeline example and its code, then X_train is Series of words.

– Vivek Kumar
Jan 2 at 9:30

1

@Christopher Yes its correct. I have updated it slightly for better code.

– Vivek Kumar
Jan 2 at 9:37

|
show 5 more comments

Your Answer

StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});

}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53998446%2fw2vtransformer-only-works-with-one-word-as-input%23new-answer', 'question_page');
}
);

Post as a guest

Name

Required, but never shown

1 Answer
1

active

oldest

votes

1 Answer
1

active

oldest

votes

This is technically not an answer, but cannot be written in comments so here it is. There are multiple issues here:

LogisticRegression class (and most other scikit-learn models) work with 2-d data (n_samples, n_features).

That means that it needs a collection of 1-d arrays (one for each row (sample), in which the elements of array contains the feature values).

In your data, a single word will be a 1-d array, which means that the single sentence (sample) will be a 2-d array. Which means that the complete data (collection of sentences here) will be a collection of 2-d arrays. Even in that, since each sentence can have different number of words, it cannot be combined into a single 3-d array.

Secondly, the W2VTransformer in gensim looks like a scikit-learn compatible class, but its not. It tries to follows "scikit-learn API conventions" for defining the methods fit(), fit_transform() and transform(). They are not compatible with scikit-learn Pipeline.

You can see that the input param requirements of fit() and fit_transform() are different.
- fit():
  
  X (iterable of iterables of str) – The input corpus.
  
  X can be simply a list of lists of tokens, but for larger corpora, consider an iterable that streams the sentences directly from
  disk/network. See BrownCorpus, Text8Corpus or LineSentence in word2vec
  module for such examples.
- fit_transform():
  
  X (numpy array of shape [n_samples, n_features]) – Training set.

sum of individual words

average of individual words

weighted averaging of individual words based on frequency, tf-idf etc.

using other techniques like sent2vec, paragraph2vec, doc2vec etc.

Note:- I noticed now that you were doing this thing based on D2VTransformer. That should be the correct approach here if you want to use sklearn.

The issue in that question was this line (since that question is now deleted):

X_train = vectorizer.fit_transform(X_train)

Here, you overwrite your original X_train (list of list of words) with already calculated word vectors and hence that error.

Update:

In the above given solution, you can replace the lines:

model = D2VTransformer(dm=1, size=50, min_count=2, iter=10, seed=0)

model.fit(X_train)



clf = LogisticRegression(penalty='l2', C=0.1, random_state=0)

clf.fit(model.transform(X_train), y_train)



pipeline = Pipeline([

        ('vec', model),

        ('clf', clf)

    ])



y_pred = pipeline.predict(X_train)

with

pipeline = Pipeline([

        ('vec', model),

        ('clf', clf)

    ])



pipeline.fit(X_train, y_train)

y_pred = pipeline.predict(X_train)

No need to fit and transform separately, since pipeline.fit() will automatically do that.

edited Jan 2 at 9:36

answered Jan 2 at 7:58

Vivek Kumar

16.5k42156

Yes, apparently the D2V version seems to work.. I also added a working version above. Thanks for the answer!

– Christopher
Jan 2 at 9:10

@Christopher I have updated the answer with the error in that question.

– Vivek Kumar
Jan 2 at 9:18

Where exact? It seems to work. I think model.transform(X_train)solved it?

– Christopher
Jan 2 at 9:26

@Christopher The error about "ufunc 'add' did not contain a loop with ...". It will happen when you combine both the codes active (D2V Example - Without Pipeline and D2V Example - With Pipeline). In that case, the X_train that gets passed to pipeline is already doc vectors. If you remove the without pipeline example and its code, then X_train is Series of words.

– Vivek Kumar
Jan 2 at 9:30

1

@Christopher Yes its correct. I have updated it slightly for better code.

– Vivek Kumar
Jan 2 at 9:37

|
show 5 more comments

This is technically not an answer, but cannot be written in comments so here it is. There are multiple issues here:

LogisticRegression class (and most other scikit-learn models) work with 2-d data (n_samples, n_features).

That means that it needs a collection of 1-d arrays (one for each row (sample), in which the elements of array contains the feature values).

In your data, a single word will be a 1-d array, which means that the single sentence (sample) will be a 2-d array. Which means that the complete data (collection of sentences here) will be a collection of 2-d arrays. Even in that, since each sentence can have different number of words, it cannot be combined into a single 3-d array.

Secondly, the W2VTransformer in gensim looks like a scikit-learn compatible class, but its not. It tries to follows "scikit-learn API conventions" for defining the methods fit(), fit_transform() and transform(). They are not compatible with scikit-learn Pipeline.

You can see that the input param requirements of fit() and fit_transform() are different.
- fit():
  
  X (iterable of iterables of str) – The input corpus.
  
  X can be simply a list of lists of tokens, but for larger corpora, consider an iterable that streams the sentences directly from
  disk/network. See BrownCorpus, Text8Corpus or LineSentence in word2vec
  module for such examples.
- fit_transform():
  
  X (numpy array of shape [n_samples, n_features]) – Training set.

sum of individual words

average of individual words

weighted averaging of individual words based on frequency, tf-idf etc.

using other techniques like sent2vec, paragraph2vec, doc2vec etc.

Note:- I noticed now that you were doing this thing based on D2VTransformer. That should be the correct approach here if you want to use sklearn.

The issue in that question was this line (since that question is now deleted):

X_train = vectorizer.fit_transform(X_train)

Here, you overwrite your original X_train (list of list of words) with already calculated word vectors and hence that error.

Update:

In the above given solution, you can replace the lines:

model = D2VTransformer(dm=1, size=50, min_count=2, iter=10, seed=0)

model.fit(X_train)



clf = LogisticRegression(penalty='l2', C=0.1, random_state=0)

clf.fit(model.transform(X_train), y_train)



pipeline = Pipeline([

        ('vec', model),

        ('clf', clf)

    ])



y_pred = pipeline.predict(X_train)

with

pipeline = Pipeline([

        ('vec', model),

        ('clf', clf)

    ])



pipeline.fit(X_train, y_train)

y_pred = pipeline.predict(X_train)

No need to fit and transform separately, since pipeline.fit() will automatically do that.

edited Jan 2 at 9:36

answered Jan 2 at 7:58

Vivek Kumar

16.5k42156

Yes, apparently the D2V version seems to work.. I also added a working version above. Thanks for the answer!

– Christopher
Jan 2 at 9:10

@Christopher I have updated the answer with the error in that question.

– Vivek Kumar
Jan 2 at 9:18

Where exact? It seems to work. I think model.transform(X_train)solved it?

– Christopher
Jan 2 at 9:26

@Christopher The error about "ufunc 'add' did not contain a loop with ...". It will happen when you combine both the codes active (D2V Example - Without Pipeline and D2V Example - With Pipeline). In that case, the X_train that gets passed to pipeline is already doc vectors. If you remove the without pipeline example and its code, then X_train is Series of words.

– Vivek Kumar
Jan 2 at 9:30

1

@Christopher Yes its correct. I have updated it slightly for better code.

– Vivek Kumar
Jan 2 at 9:37

|
show 5 more comments

This is technically not an answer, but cannot be written in comments so here it is. There are multiple issues here:

LogisticRegression class (and most other scikit-learn models) work with 2-d data (n_samples, n_features).

That means that it needs a collection of 1-d arrays (one for each row (sample), in which the elements of array contains the feature values).

In your data, a single word will be a 1-d array, which means that the single sentence (sample) will be a 2-d array. Which means that the complete data (collection of sentences here) will be a collection of 2-d arrays. Even in that, since each sentence can have different number of words, it cannot be combined into a single 3-d array.

Secondly, the W2VTransformer in gensim looks like a scikit-learn compatible class, but its not. It tries to follows "scikit-learn API conventions" for defining the methods fit(), fit_transform() and transform(). They are not compatible with scikit-learn Pipeline.

You can see that the input param requirements of fit() and fit_transform() are different.
- fit():
  
  X (iterable of iterables of str) – The input corpus.
  
  X can be simply a list of lists of tokens, but for larger corpora, consider an iterable that streams the sentences directly from
  disk/network. See BrownCorpus, Text8Corpus or LineSentence in word2vec
  module for such examples.
- fit_transform():
  
  X (numpy array of shape [n_samples, n_features]) – Training set.

sum of individual words

average of individual words

weighted averaging of individual words based on frequency, tf-idf etc.

using other techniques like sent2vec, paragraph2vec, doc2vec etc.

Note:- I noticed now that you were doing this thing based on D2VTransformer. That should be the correct approach here if you want to use sklearn.

The issue in that question was this line (since that question is now deleted):

X_train = vectorizer.fit_transform(X_train)

Here, you overwrite your original X_train (list of list of words) with already calculated word vectors and hence that error.

Update:

In the above given solution, you can replace the lines:

model = D2VTransformer(dm=1, size=50, min_count=2, iter=10, seed=0)

model.fit(X_train)



clf = LogisticRegression(penalty='l2', C=0.1, random_state=0)

clf.fit(model.transform(X_train), y_train)



pipeline = Pipeline([

        ('vec', model),

        ('clf', clf)

    ])



y_pred = pipeline.predict(X_train)

with

pipeline = Pipeline([

        ('vec', model),

        ('clf', clf)

    ])



pipeline.fit(X_train, y_train)

y_pred = pipeline.predict(X_train)

No need to fit and transform separately, since pipeline.fit() will automatically do that.

edited Jan 2 at 9:36

answered Jan 2 at 7:58

Vivek Kumar

16.5k42156

This is technically not an answer, but cannot be written in comments so here it is. There are multiple issues here:

LogisticRegression class (and most other scikit-learn models) work with 2-d data (n_samples, n_features).

That means that it needs a collection of 1-d arrays (one for each row (sample), in which the elements of array contains the feature values).

In your data, a single word will be a 1-d array, which means that the single sentence (sample) will be a 2-d array. Which means that the complete data (collection of sentences here) will be a collection of 2-d arrays. Even in that, since each sentence can have different number of words, it cannot be combined into a single 3-d array.

Secondly, the W2VTransformer in gensim looks like a scikit-learn compatible class, but its not. It tries to follows "scikit-learn API conventions" for defining the methods fit(), fit_transform() and transform(). They are not compatible with scikit-learn Pipeline.

You can see that the input param requirements of fit() and fit_transform() are different.
- fit():
  
  X (iterable of iterables of str) – The input corpus.
  
  X can be simply a list of lists of tokens, but for larger corpora, consider an iterable that streams the sentences directly from
  disk/network. See BrownCorpus, Text8Corpus or LineSentence in word2vec
  module for such examples.
- fit_transform():
  
  X (numpy array of shape [n_samples, n_features]) – Training set.

sum of individual words

average of individual words

weighted averaging of individual words based on frequency, tf-idf etc.

using other techniques like sent2vec, paragraph2vec, doc2vec etc.

Note:- I noticed now that you were doing this thing based on D2VTransformer. That should be the correct approach here if you want to use sklearn.

The issue in that question was this line (since that question is now deleted):

X_train = vectorizer.fit_transform(X_train)

Here, you overwrite your original X_train (list of list of words) with already calculated word vectors and hence that error.

Update:

In the above given solution, you can replace the lines:

model = D2VTransformer(dm=1, size=50, min_count=2, iter=10, seed=0)

model.fit(X_train)



clf = LogisticRegression(penalty='l2', C=0.1, random_state=0)

clf.fit(model.transform(X_train), y_train)



pipeline = Pipeline([

        ('vec', model),

        ('clf', clf)

    ])



y_pred = pipeline.predict(X_train)

with

pipeline = Pipeline([

        ('vec', model),

        ('clf', clf)

    ])



pipeline.fit(X_train, y_train)

y_pred = pipeline.predict(X_train)

No need to fit and transform separately, since pipeline.fit() will automatically do that.

edited Jan 2 at 9:36

answered Jan 2 at 7:58

Vivek Kumar

16.5k42156

edited Jan 2 at 9:36

answered Jan 2 at 7:58

Vivek Kumar

16.5k42156

answered Jan 2 at 7:58

Vivek Kumar

16.5k42156

answered Jan 2 at 7:58

Vivek Kumar

16.5k42156

Yes, apparently the D2V version seems to work.. I also added a working version above. Thanks for the answer!

– Christopher
Jan 2 at 9:10

@Christopher I have updated the answer with the error in that question.

– Vivek Kumar
Jan 2 at 9:18

Where exact? It seems to work. I think model.transform(X_train)solved it?

– Christopher
Jan 2 at 9:26

@Christopher The error about "ufunc 'add' did not contain a loop with ...". It will happen when you combine both the codes active (D2V Example - Without Pipeline and D2V Example - With Pipeline). In that case, the X_train that gets passed to pipeline is already doc vectors. If you remove the without pipeline example and its code, then X_train is Series of words.

– Vivek Kumar
Jan 2 at 9:30

1

@Christopher Yes its correct. I have updated it slightly for better code.

– Vivek Kumar
Jan 2 at 9:37

|
show 5 more comments

Yes, apparently the D2V version seems to work.. I also added a working version above. Thanks for the answer!

– Christopher
Jan 2 at 9:10

@Christopher I have updated the answer with the error in that question.

– Vivek Kumar
Jan 2 at 9:18

Where exact? It seems to work. I think model.transform(X_train)solved it?

– Christopher
Jan 2 at 9:26

@Christopher The error about "ufunc 'add' did not contain a loop with ...". It will happen when you combine both the codes active (D2V Example - Without Pipeline and D2V Example - With Pipeline). In that case, the X_train that gets passed to pipeline is already doc vectors. If you remove the without pipeline example and its code, then X_train is Series of words.

– Vivek Kumar
Jan 2 at 9:30

1

@Christopher Yes its correct. I have updated it slightly for better code.

– Vivek Kumar
Jan 2 at 9:37

Yes, apparently the D2V version seems to work.. I also added a working version above. Thanks for the answer!

– Christopher
Jan 2 at 9:10

@Christopher I have updated the answer with the error in that question.

– Vivek Kumar
Jan 2 at 9:18

Where exact? It seems to work. I think model.transform(X_train)solved it?

– Christopher
Jan 2 at 9:26

@Christopher The error about "ufunc 'add' did not contain a loop with ...". It will happen when you combine both the codes active (D2V Example - Without Pipeline and D2V Example - With Pipeline). In that case, the X_train that gets passed to pipeline is already doc vectors. If you remove the without pipeline example and its code, then X_train is Series of words.

– Vivek Kumar
Jan 2 at 9:30

@Christopher Yes its correct. I have updated it slightly for better code.

– Vivek Kumar
Jan 2 at 9:37

|
show 5 more comments

draft saved

draft discarded

Thanks for contributing an answer to Stack Overflow!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

Search This Blog

Ufyukyu