Mixing
Size of the vocabulary in Laplace smoothing for a trigram language model
$begingroup$
Let's say we have a text document with $N$ unique words making up a vocabulary $V$, $|V| = N$. For a bigram language model with add-one smoothing, we define a conditional probability of any word $w_{i}$ given the preceeding word $w_{i-1}$ as: $$P(w_{i}|w_{i-1}) = frac{count(w_{i-1}w_{i}) + 1}{count(w_{i-1}) + |V|}$$ As far as I understand (or not) the conditional probability, and basing on a 3rd point of this Wikipedia article, $w_{i-1}$ might be assumed to be "constant" here, so by summing this expression for all possible $w_{i}$ we should obtain 1, and so it is, which is obvious.
However, I do not understand the answers given for this question saying that for n-gram model the size of the vocabulary should be the count of the unique (n-1)-grams occuring in a document, for example, given a 3-gram model (let $V_{2}$ be the dictionary of bigrams): $$P(w_{i}|w_{i-2}w_{i-1}) = frac{count(w_{i-2}w_{i-1}w_{i}) + 1}{count(w_{i-2}w_{i-1}) + |V_{2}|}$$ It just doesn't add up to 1 when we try to sum it for every possible $w_{i}$. Therefore - should the $|V|$ really be equal to the count of unique (n-1)-grams given an n-gram language model or should it be the count of unique unigrams?
probability machine-learning
asked May 30 '16 at 22:51
cafe_cafe_
263
$endgroup$
add a comment |
$begingroup$
Let's say we have a text document with $N$ unique words making up a vocabulary $V$, $|V| = N$. For a bigram language model with add-one smoothing, we define a conditional probability of any word $w_{i}$ given the preceeding word $w_{i-1}$ as: $$P(w_{i}|w_{i-1}) = frac{count(w_{i-1}w_{i}) + 1}{count(w_{i-1}) + |V|}$$ As far as I understand (or not) the conditional probability, and basing on a 3rd point of this Wikipedia article, $w_{i-1}$ might be assumed to be "constant" here, so by summing this expression for all possible $w_{i}$ we should obtain 1, and so it is, which is obvious.
However, I do not understand the answers given for this question saying that for n-gram model the size of the vocabulary should be the count of the unique (n-1)-grams occuring in a document, for example, given a 3-gram model (let $V_{2}$ be the dictionary of bigrams): $$P(w_{i}|w_{i-2}w_{i-1}) = frac{count(w_{i-2}w_{i-1}w_{i}) + 1}{count(w_{i-2}w_{i-1}) + |V_{2}|}$$ It just doesn't add up to 1 when we try to sum it for every possible $w_{i}$. Therefore - should the $|V|$ really be equal to the count of unique (n-1)-grams given an n-gram language model or should it be the count of unique unigrams?
probability machine-learning
asked May 30 '16 at 22:51
cafe_cafe_
263
$endgroup$
add a comment |
$begingroup$
Let's say we have a text document with $N$ unique words making up a vocabulary $V$, $|V| = N$. For a bigram language model with add-one smoothing, we define a conditional probability of any word $w_{i}$ given the preceeding word $w_{i-1}$ as: $$P(w_{i}|w_{i-1}) = frac{count(w_{i-1}w_{i}) + 1}{count(w_{i-1}) + |V|}$$ As far as I understand (or not) the conditional probability, and basing on a 3rd point of this Wikipedia article, $w_{i-1}$ might be assumed to be "constant" here, so by summing this expression for all possible $w_{i}$ we should obtain 1, and so it is, which is obvious.
However, I do not understand the answers given for this question saying that for n-gram model the size of the vocabulary should be the count of the unique (n-1)-grams occuring in a document, for example, given a 3-gram model (let $V_{2}$ be the dictionary of bigrams): $$P(w_{i}|w_{i-2}w_{i-1}) = frac{count(w_{i-2}w_{i-1}w_{i}) + 1}{count(w_{i-2}w_{i-1}) + |V_{2}|}$$ It just doesn't add up to 1 when we try to sum it for every possible $w_{i}$. Therefore - should the $|V|$ really be equal to the count of unique (n-1)-grams given an n-gram language model or should it be the count of unique unigrams?
probability machine-learning
asked May 30 '16 at 22:51
cafe_cafe_
263
$endgroup$
Let's say we have a text document with $N$ unique words making up a vocabulary $V$, $|V| = N$. For a bigram language model with add-one smoothing, we define a conditional probability of any word $w_{i}$ given the preceeding word $w_{i-1}$ as: $$P(w_{i}|w_{i-1}) = frac{count(w_{i-1}w_{i}) + 1}{count(w_{i-1}) + |V|}$$ As far as I understand (or not) the conditional probability, and basing on a 3rd point of this Wikipedia article, $w_{i-1}$ might be assumed to be "constant" here, so by summing this expression for all possible $w_{i}$ we should obtain 1, and so it is, which is obvious.
However, I do not understand the answers given for this question saying that for n-gram model the size of the vocabulary should be the count of the unique (n-1)-grams occuring in a document, for example, given a 3-gram model (let $V_{2}$ be the dictionary of bigrams): $$P(w_{i}|w_{i-2}w_{i-1}) = frac{count(w_{i-2}w_{i-1}w_{i}) + 1}{count(w_{i-2}w_{i-1}) + |V_{2}|}$$ It just doesn't add up to 1 when we try to sum it for every possible $w_{i}$. Therefore - should the $|V|$ really be equal to the count of unique (n-1)-grams given an n-gram language model or should it be the count of unique unigrams?
probability machine-learning
probability machine-learning
asked May 30 '16 at 22:51
cafe_cafe_
263
asked May 30 '16 at 22:51
cafe_cafe_
263
edited May 30 '16 at 23:21
cafe_
asked May 30 '16 at 22:51
cafe_cafe_
263
asked May 30 '16 at 22:51
cafe_cafe_
263
asked May 30 '16 at 22:51
cafe_cafe_
263
263
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
The slides here mention that V should be the size of the vocabulary which makes sense for me.
answered Feb 4 '18 at 5:09
Avinash KumarAvinash Kumar
1286
$endgroup$
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "69"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
noCode: true, onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f1806317%2fsize-of-the-vocabulary-in-laplace-smoothing-for-a-trigram-language-model%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
The slides here mention that V should be the size of the vocabulary which makes sense for me.
answered Feb 4 '18 at 5:09
Avinash KumarAvinash Kumar
1286
$endgroup$
add a comment |
$begingroup$
The slides here mention that V should be the size of the vocabulary which makes sense for me.
answered Feb 4 '18 at 5:09
Avinash KumarAvinash Kumar
1286
$endgroup$
add a comment |
$begingroup$
The slides here mention that V should be the size of the vocabulary which makes sense for me.
answered Feb 4 '18 at 5:09
Avinash KumarAvinash Kumar
1286
$endgroup$
The slides here mention that V should be the size of the vocabulary which makes sense for me.
answered Feb 4 '18 at 5:09
Avinash KumarAvinash Kumar
1286
answered Feb 4 '18 at 5:09
Avinash KumarAvinash Kumar
1286
answered Feb 4 '18 at 5:09
Avinash KumarAvinash Kumar
1286
answered Feb 4 '18 at 5:09
Avinash KumarAvinash Kumar
1286
1286
add a comment |
add a comment |
Thanks for contributing an answer to Mathematics Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f1806317%2fsize-of-the-vocabulary-in-laplace-smoothing-for-a-trigram-language-model%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
