Nlp Tokenization And Word Sense Disambiguation

Tokenization in natural language processing (NLP)involves breaking down text into smaller units called tokens. These tokens can be words, characters, or subwords. This process is for preparing text for further processing.

Word Tokenization

Word tokenization is the process of splitting text into individual words.

Example: Input: “I like doing NLP”

Tokens: [“I”, “like”, “doing”, “NLP”]

Byte-Pair Encoding Tokenization

It’s used by a lot of Transformer models, including GPT, GPT-2, RoBERTa, BART, and DeBERTa. BPE training starts by computing the unique set of words used in the corpus, then iteratively merging the most frequently occurring character or byte pairs in a given dataset until a desired vocabulary size is reached.

Example: word “low” appear 5 times, “lowest” appear 2 times, “newer” appear 6 times, “wider” appear 3 times, “new” appear 2 time in the corpus.

Step 1: Initial Vocabulary

corpus                                              vocabulary

5   l o w _                                           _, d, e, i, l, n, o, r, s, t, w

2    l o w e s t _

6    n e w e r _

3    w i d e r _

2    n e w _

Step 2: Count and Merge Frequent Pairs (er appear 9 times)

corpus                                              vocabulary

5   l o w _                                          _, d, e, i, l, n, o, r, s, t, w, er

2    l o w e s t _

6    n e w er _

3    w i d er _

2    n e w _

step 3: continues to count and merge Frequent Pairs (er_ appear 9 times)

corpus                                              vocabulary

5   l o w _                                         _ , d, e, i, l, n, o, r, s, t, w, er, er_

2    l o w e s t _

6    n e w er_

3    w i d er_

2    n e w _

step n: continues until the max vocabulary set is reached.

Algorithm:

The articles I found which are very helpful for understanding Byte-Pair Encoding Tokenization:

• Byte-Pair Encoding: Subword-based tokenization algorithm written by Chetna Khanna
• BPE Tokenization Demystified: Implementation and Examples written by MartinLwx
• Vinija’s notes written by Jain, Vinija, and Chadha, Aman.

Advantage

• Handling Rare Words: By tokenizing words into subwords, BPE mitigates the issue of out-of-vocabulary(OOV) by allowing the model to understand and generate words that it might not have seen during training.
• Reduced Model Size: By tokenizing words into subwords, BPE allows the model to work with a more manageable vocabulary size. This can reduce the model’s complexity and the amount of memory required.

Implementation

I’ve implemented both word tokenizer and BPE tokenizer in Python from scratch:

dataset

python code

output

WordPiece Tokenization

Word Piece is also a subword tokenization algorithm, it’s similar to BPE but differs slightly in its approach to creating new tokens. Instead of just merging the most frequent pairs, WordPiece looks at the likelihood of the entire vocabulary and adds the token that increases the likelihood of the data the most. It’s first proposed in the paper “Japanese and Korean Voice Search”. The core idea of Word Piece method:

• does putting “a” and “b” together increase the ability to model the corpus? can be quantified by: p(‘a’,’b’) / (p(‘a’)p(‘b’))

The articles I found which are very helpful for understanding WordPiece:

• WordPiece: Subword-based tokenization algorithm written by Chetna Khanna
• Vinija’s notes written by Jain, Vinija, and Chadha, Aman.

Logistic Regression for Word Sense Disambiguation (WSD)

WSD is the process of identifying which sense of a word (i.e., its meaning) is used in a sentence when the word has multiple meanings. For example, the word “return” has different meanings such as:

1. come back to a place or situation.
2. To yield a profit or result.

This task is crucial for many natural language processing (NLP) applications, such as machine translation, information retrieval, and text understanding. Logistic Regression is a simple (easy to interpret) and useful model to identify the sense of a word in a sentence, but the pitfall of logistic regression is it might not come with the most accurate result.

More details about logistic regression or generalized linear regression can be seen in this post

Implementation

I’ve implemented logistic regression using PyTorch for Word Sense Disambiguation (WSD) from scratch: (One also can use CNN, RNN, or transformer to do this task, as those may yield better accuracy)

dataset

python code

output