Large Language Model (LLM) Notes

Definition for \(n\)-gram:

An \(n\)-gram language model assumes each word depends only on the last \(n−1\) words (markov assumptions with order \(n-1\)​):

\[\begin{align} P_{ngram}(w_1,\ldots,w_N)&=P(w_1)P(w_2|w_1)\ldots P(w_i|\underbrace{w_{i-1},\ldots,w_{i-(n+1)})}_{n-1 \text{ items}}\ldots P(w_N|w_{N-1},\ldots,w_{N-(n+1)}))\\ &=\prod_{i=1}^N P(w_i|w_{i-1},\ldots,w_{i-(n+1)}). \end{align} \]

• Use EOS (end-of-sentence) </s> token to limit sentence length.
• Add \(n-1\) beginning-of-sentence (BOS) (<s>) to each sentence for an \(n\)-gram mode to ensure consistency for \(P(w_i|w_{i-1},\ldots,w_{i-(n+1))}\) for the first \(n-1\) items.

• Unigram model (\(1\)-gram​): \(P(w_1,\ldots,w_i)=\prod_{k=1}^iP(w_k)\).
• Bigram model (\(2\)-gram): \(P(w_1,\ldots,w_i)=\prod_{k=1}^iP(w_k|w_{k-1})\).
• Trigram model (\(3\)-gram): \(P(w_1,\ldots,w_i)=\prod_{k=1}^iP(w_k|w_{k-1},w_{k-2})\)​.

(Intrinsic Evaluation). Perplexity:

Inverse (\(1\over P(\ldots)\)) of the probability of the test set, normalized (\(\sqrt[N]{\ldots}\)) by the # of tokens (\(N\)) in the test set.

If a LM assigns probability \(P(w_1,\ldots,w_N)\) to a test corpus \(w_1,\ldots,w_N\), the perplexity for \(n\)-gram language model could be written as:

\[PP(w_1,\ldots,w_N)=\sqrt[N]{1 \over P(w_1,\ldots,w_N)}=\sqrt[N]{1 \over \prod_{i=1}^N P(w_i|w_{i-1},\ldots,w_{i-(n+1)})}. \]

Rewrite it into log-form (exponent of mean of log likelihood of all the words in an input sequence):

\[PPL(w_1,\ldots,w_N)=\exp\left(-\frac{1}{N}\sum_{i=1}^N\log\left(P(w_i|w_{i-1},\ldots,w_{i-(n+1)})\right)\right). \]

• Lower perplexity \(\rightarrow\) a higher probability to the unseen test corpus.

(Extrinsic Evaluation). Word error rate (WER):

\[\text{WER} = \frac{\text{Insertions}+\text{Deletions}+\text{Substitutions}}{\text{Actual words in transcript}}. \]