Self-Attention

The self-attention mechanism uses three matrices - query (Q), key (K), and value (V) - to help the system understand and process the relationships between words in a sentence.

Let’s say we have an input sequence of length N, represented as a matrix X of shape (N, d), where d is the dimension of the input embedding vector. We can calculate the query, key, and value vectors as follows:

• Query: We multiply the input embedding matrix X with a learned weight matrix Wq of shape (d, h), where h is the number of attention heads. This produces a matrix Q of shape (N, h), where each row corresponds to the query vector for the corresponding token in the input sequence.Q = XWq
• Key: We multiply the input embedding matrix X with a learned weight matrix Wk of shape (d, h). This produces a matrix K of shape (N, h), where each row corresponds to the key vector for the corresponding token in the input sequence.K = XWk
• Value: We multiply the input embedding matrix X with a learned weight matrix Wv of shape (d, h). This produces a matrix V of shape (N, h), where each row corresponds to the value vector for the corresponding token in the input sequence.V = XWv Once we have calculated the query, key, and value vectors, we can use them to compute the attention scores between all pairs of tokens in the input sequence. The attention scores are then used to weight the corresponding value vectors, which are summed to produce the final output of the self-attention mechanism.

Word and Positional Embeddings

A typical word embedding is a dense vector that represents the meaning and semantic information of a word in a continuous space. The dimension of the embeddings depends on the specific model and configuration. Let’s assume we have a 3-dimensional embedding space for simplicity. An example of embeddings for the tokens in the input sentence “The cat sat on the mat” could look like this:

• The: [0.21, -0.15, 0.32]
• cat: [0.85, 0.29, -0.61]
• sat: [-0.37, 0.72, 0.45]
• on: [0.12, -0.64, 0.27]
• the: [0.21, -0.15, 0.32] (Note: Same as ‘The’ embedding, as word embeddings are usually case-insensitive)
• mat: [-0.53, 0.31, 0.81] These are just example values and not real embeddings. In practice, the embeddings are learned by the model during training, and the dimensionality is usually much larger, such as 128, 256, 512, or even larger. In addition to the word embeddings, we also need to add positional encoding to these vectors to preserve the order of the words in the input sentence. Positional encodings is sinusoidal functions
• the output of sine and cosine is in [-1, 1], which is normalized. It won’t grow to an unmanageable size like integers would.
• no additional training has to be done since unique representations are generated for each position. where k is position, i is dimensional index

Assuming we have a simple learned positional encoding, the final input embeddings might look like this:

• The (word + position): [0.21, -0.15, 0.32] + [0.01, 0.02, 0.03] = [0.22, -0.13, 0.35]
• cat (word + position): [0.85, 0.29, -0.61] + [0.04, 0.05, 0.06] = [0.89, 0.34, -0.55]
• sat (word + position): [-0.37, 0.72, 0.45] + [0.07, 0.08, 0.09] = [-0.30, 0.80, 0.54]
• on (word + position): [0.12, -0.64, 0.27] + [0.10, 0.11, 0.12] = [0.22, -0.53, 0.39]
• the (word + position): [0.21, -0.15, 0.32] + [0.13, 0.14, 0.15] = [0.34, -0.01, 0.47]
• mat (word + position): [-0.53, 0.31, 0.81] + [0.16, 0.17, 0.18] = [-0.37, 0.48, 0.99]

Multi-headed attention

In the multi-head attention layer, the self-attention mechanism is applied multiple times in parallel, each with different sets of learned weights. This allows the model to capture various aspects of the context and focus on different relationships among words simultaneously. Here’s a step-by-step explanation of the multi-head attention layer:

• Determine the number of attention heads. Let’s say we have H heads.
• For each attention head h (where h = 1, 2, …, H), initialize separate sets of weight matrices WQ_h, WK_h, and WV_h.
• Perform self-attention H times in parallel, using the different sets of weight matrices for each head. This involves calculating the Q, K, and V matrices, computing the attention scores and weights, and obtaining the context-aware representations.
• Multiply the concatenated matrix by another learned weight matrix, WO (output matrix), to produce the final output of the multi-head attention layer. The multi-head attention layer allows the model to capture different contextual aspects and relationships among words simultaneously. For example, one attention head might focus on syntactic relationships, while another could focus on semantic relationships. Combining the results from multiple attention heads enables the transformer to build a richer understanding of the input text. The output of the multi-head attention layer is then passed through the remaining layers of the transformer, such as position-wise feedforward layers, layer normalization, and residual connections, before producing the final output.

FFNS

In a transformer architecture, after the multi-head attention layer, the output goes through a position-wise feedforward layer. This layer applies a feedforward neural network (FFNN) independently to each position in the input sequence. The purpose of this layer is to process and combine the features learned by the multi-head attention layer for each token while maintaining their positions. Here’s an overview of what happens in the position-wise feedforward layer:

• For each position (i.e., each token) in the input sequence, apply a feedforward neural network. This FFNN has the same architecture and weights across all positions, but it operates independently on each token.
• The FFNN typically consists of two linear layers (i.e., fully connected layers) with a non-linear activation function, such as ReLU (Rectified Linear Unit), applied in between. This structure allows the model to learn more complex relationships and features within the input data.
• The first linear layer expands the dimension of the input, which helps the model capture more complex patterns. The dimension is then reduced back to the original size by the second linear layer.
• The output of the position-wise feedforward layer for each token is then combined with its corresponding input through a residual connection (addition) and passed through a layer normalization step.

In summary, the position-wise feedforward layer applies a feedforward neural network independently to each token in the input sequence. This helps process and combine the features learned by the multi-head attention layer while maintaining the positional information of the tokens. The layer enhances the expressive power of the transformer model, allowing it to capture more complex patterns and relationships within the input data.

Layer Normalization

Normalization is a process that standardizes the input layer by adjusting and scaling the activations. Specifically, Transformers use Layer Normalization (LN). Unlike other normalization methods that operate over the batch dimension, LN operates over the feature dimension (across all channels). The mathematical representation of Layer Normalization is:

Reference： https://towardsdatascience.com/into-the-transformer-5ad892e0cee https://www.linkedin.com/pulse/gpt-4-explaining-self-attention-mechanism-fatos-ismali/ https://www.linkedin.com/pulse/clear-explanation-transformer-neural-networks-ebin-babu-thomas/

Bert Implementation

https://readmedium.com/en/https:/towardsdatascience.com/masked-language-modelling-with-bert-7d49793e5d2c

Generative Language Modeling (Greedy, Beam Search, Random Walk)

Greedy Decoding

• It is one of the simplest and fastest methods for generating text and involves selecting the most likely next word at each step in the sequence.

  Beam Search

• The fundamental idea of beam search is exploring multiple paths instead of just single one.: Unlike greedy decoding, which only keeps the single best path (the most probable next word) at each step, beam search keeps track of the k most probable paths at each step. This means at every step in the sequence,
• it doesn’t just consider the single most likely next word, but the k most likely next words.

  Random Walk

  method where the selection of each next word in a sequence is based on some level of randomness rather than deterministic rules or purely greedy choices.

• Stochastic Sampling: In language generation, a random walk can be analogous to sampling methods where each word is chosen based on a probability distribution over the vocabulary. The model computes probabilities for the next word based on the context provided by previous words, and then a word is randomly selected based on this distribution.
• Temperature Adjusted Sampling: This is a refinement where the “temperature” parameter is used to control the randomness of the word selection:
  • A high temperature results in more uniform distributions (higher randomness), making the generation process more exploratory and diverse, akin to a random walk with many possible paths.
  • A low temperature results in sharper distributions (less randomness), making the generation more conservative, focusing on more likely words.
• Top-k and Top-p Sampling: These are methods to constrain the randomness to the most likely next words:
  • Top-k Sampling: Top-k sampling refines the selection process by limiting the pool of potential next tokens to the top-k most likely candidates. Among these top-k tokens, the next token is chosen based on its probability, maintaining a degree of randomness while focusing on more likely options.
  • Top-p (Nucleus) Sampling: In top-p sampling or nucleus sampling, the selection pool for the next token is determined by the cumulative probability of the most probable tokens. When you set a threshold p, the model includes just enough of the most probable tokens so that their combined probability reaches or exceeds this threshold

https://medium.com/@pashashaik/natural-language-generation-from-scratch-in-large-language-models-with-pytorch-4d9379635316

Pre-training

Pre-training is a foundational step in the LLM training process, where the model gains a general understanding of language by exposure to vast amounts of text data.

Think of it as the model’s introduction to the world of words, phrases, and ideas. During this phase, the LLM learns grammar rules, linguistic patterns, factual information, and reasoning abilities.

To achieve this, the model is fed an extensive dataset containing diverse texts from books, articles, websites, and more. Take the example of GPT-3, with its immense parameter count. It ingested around 570 GB of text data during pre-training. This is like reading hundreds of thousands of books in multiple languages, giving the model a rich tapestry of language to draw from.

fine-tuning

After the initial pre-training phase, like primary and secondary education where the LLM gets its general language skills, fine-tuning is like giving it on-the-job training. It’s honing its abilities for particular tasks or domains, transforming it from a language learner into a task-specific expert.

For instance, scientists might fine-tune a model with a dataset of medical texts, making it exceptional at understanding medical jargon and answering health-related questions.

Similarly, you can train a model on legal documents, turning it into a whizz at summarizing contracts and legal discussions.

During fine-tuning, the parameters of the pre-trained model are updated (fine-tuned) using task-specific data, typically with a smaller learning rate than during pre-training.

task fine-tuning

This method allows the model to adapt its parameters to the nuances and requirements of the targeted task, thereby enhancing its performance and relevance to that particular domain. Task-specific fine-tuning is particularly valuable when you want to optimize the model’s performance for a single, well-defined task, ensuring that the model excels in generating task-specific content with precision and accuracy.

Task-specific fine-tuning is closely related to transfer learning, but transfer learning is more about leveraging the general features learned by the model, whereas task-specific fine-tuning is about adapting the model to the specific requirements of the new task.

Instruction fine-tuning

Whereas supervised fine-tuning consists in training models on example inputs and the results derived from them (output), instruction tuning fleshes out input-output examples with another component: instructions.

This is precisely what enables instruction-tuned models to generalize more easily to new tasks. As a result, LLMs tuned in this way become much more versatile and useful.

input examples and their corresponding outputs, instruction tuning augments input-output examples with instructions, which enables instruction-tuned models to generalize more easily to new tasks.

Contextual Embeddings

Both embedding techniques, traditional word embedding (e.g. word2vec, Glove) and contextual embedding (e.g. ELMo, BERT), aim to learn a continuous (vector) representation for each word in the documents. Continuous representations can be used in downstream machine learning tasks.

Traditional word embedding techniques learn a global word embedding. They first build a global vocabulary using unique words in the documents by ignoring the meaning of words in different context. Then, similar representations are learnt for the words appeared more frequently close each other in the documents. The problem is that in such word representations the words’ contextual meaning (the meaning derived from the words’ surroundings), is ignored. For example, only one representation is learnt for “left” in sentence “I left my phone on the left side of the table.” However, “left” has two different meanings in the sentence, and needs to have two different representations in the embedding space.

On the other hand, contextual embedding methods are used to learn sequence-level semantics by considering the sequence of all words in the documents. Thus, such techniques learn different representations for polysemous words, e.g. “left” in example above, based on their context.

RAG

Retrieval-augmented generation is a technique used in natural language processing that combines the power of both retrieval-based models and generative models to enhance the quality and relevance of generated text. https://webcache.googleusercontent.com/search?q=cache:https://colabdoge.medium.com/what-is-rag-retrieval-augmented-generation-b0afc5dd5e79&strip=0&vwsrc=1&referer=medium-parser

Few short, Zero-shot

• Zero-shot learning — a technique whereby we prompt an LLM without any examples, attempting to take advantage of the reasoning patterns it has gleaned (i.e. a generalist LLM)
• Few-shot learning — a technique whereby we prompt an LLM with several concrete examples of task performance
  • Fine tuning - When you already have a model trained to perform the task you want but on a different dataset, you initialise using the pre-trained weights and train it on target (usually smaller) dataset (usually with a smaller learning rate).
  • Few shot learning - When you want to train a model on any task using very few samples. e.g., you have a model trained on different but related task and you (optionally) modify it and train for target task using small number of examples. For example: Fine tuning - Training a model for intent classification and then fine tuning it on a different dataset. Few shot learning - Training a language model on large text dataset and modifying it (usually last (few) layer) to classify intents by training on small labelled dataset.

Date

Current Date and Time

To get the current date and time, most SQL databases provide functions like NOW(), CURRENT_DATE, and CURRENT_TIMESTAMP.

• SELECT CURRENT_DATE
  • Returns the current date
• SELECT CURRENT_TIMESTAMP
  • Returns the current date and time

Extracting Date Parts

You can extract specific parts of a date (like year, month, day, hour, etc.) using the EXTRACT() function or specific functions like YEAR(), MONTH(), and DAY().

• SELECT EXTRACT(YEAR FROM CURRENT_DATE) AS Year
• SELECT MONTH(CURRENT_DATE) AS Month
• SELECT DAY(CURRENT_DATE) AS Day

Date Arithmetic/DATEDIFF

You can add or subtract days, weeks, months, or years to a date using functions like DATE_ADD or DATE_SUB, or by using interval arithmetic.

• SELECT CURRENT_DATE + INTERVAL 1 DAY
  • Adds 1 day to the current date
• SELECT CURRENT_DATE - INTERVAL 7 DAY
  • Subtracts 7 days from the current date
• MySQL
  • SELECT DATEDIFF('2023-04-01', '2023-01-01') AS DaysBetween
    • Calculate the number of days between two dates
• PostgreSQL
  • SELECT ‘2023-04-01’::date - ‘2023-01-01’::date AS DaysBetween
    • Calculate the number of days between two dates
  • SELECT AGE(‘2023-04-01’::date, ‘2023-01-01’::date) AS DateDifference
    • Calculate the precise age between two dates
• SQL Server
  • SELECT DATEDIFF(day, ‘2023-01-01’, ‘2023-04-01’) AS DaysBetween
    • Calculate the number of days between two dates
  • SELECT DATEDIFF(month, ‘2023-01-01’, ‘2023-04-01’) AS MonthsBetween
    • Calculate the number of months between two dates
• Oracle
  • SELECT (TO_DATE(‘2023-04-01’, ‘YYYY-MM-DD’) - TO_DATE(‘2023-01-01’, ‘YYYY-MM-DD’)) AS DaysBetween
    • Calculate the number of days between two dates

Formatting Dates

To format dates into a specific string format, you might use the DATE_FORMAT() function in MySQL or TO_CHAR() in PostgreSQL and Oracle.

• MySQL
  • SELECT DATE_FORMAT(CURRENT_DATE, '%Y-%m-%d')
• PostgreSQL or Oracle
  • SELECT TO_CHAR(CURRENT_DATE, 'YYYY-MM-DD')

Comparing Dates

You can compare dates to filter records in a WHERE clause.

• SELECT * FROM orders WHERE order_date > ‘2023-01-01’

• SELECT * FROM orders WHERE order_date BETWEEN '2023-01-01' AND '2023-03-31'

Date Conversion

Sometimes you might need to convert strings to dates or vice versa. Functions like STR_TO_DATE() in MySQL or TO_DATE() in PostgreSQL and Oracle can be used.

• MySQL
  • SELECT STR_TO_DATE('2023-01-01', '%Y-%m-%d')
• PostgreSQL or Oracle
  • SELECT TO_DATE('2023-01-01', 'YYYY-MM-DD')

Time Zone Conversions

Handling time zones can be crucial for applications spanning multiple regions.

• PostgreSQL
  • SELECT CURRENT_TIMESTAMP AT TIME ZONE ‘UTC’
  • SELECT CURRENT_TIMESTAMP AT TIME ZONE ‘America/New_York’

GroupBy

In SQL, several standard aggregation functions are typically used with the GROUP BY clause to summarize data across different groups. These functions perform calculations on a set of values and return a single value. Here’s a list of the most commonly used aggregation functions that you can use with GROUP BY:

COUNT(), SUM(), AVG(),MAX(), MIN(),STDDEV(),STDDEV_POP(),VARIANCE(),VAR_POP()

• SELECT category, COUNT(*) FROM products GROUP BY category
• SELECT department, SUM(salary) FROM employees GROUP BY department
• SELECT department, AVG(salary) FROM employees GROUP BY department
• SELECT department, MAX(salary) FROM employees GROUP BY department
• SELECT department, MIN(salary) FROM employees GROUP BY department
• SELECT department, STDDEV(salary) FROM employees GROUP BY department
• SELECT department, VARIANCE(salary) FROM employees GROUP BY department

GROUP_CONCAT() (MySQL/SQLite)

Concatenates values from a group into a single string with values separated by a delimiter. Not standard SQL but available in several SQL databases.

• SELECT department, GROUP_CONCAT(employee_name SEPARATOR ', ') FROM employees GROUP BY department;

Additional Aggregation Considerations(Distinct, Over Paritions)

• Distinct: Many aggregation functions allow you to specify DISTINCT to consider only distinct values within each group.
  • SELECT category, COUNT(DISTINCT product_name) FROM products GROUP BY category;
• Over Partitions: While not used directly with GROUP BY, many aggregation functions can also be used with the OVER() clause to perform calculations across different partitions of data without collapsing the rows into a single output row per group.
  • SELECT department, salary, AVG(salary) OVER (PARTITION BY department) FROM employees;

Chapter 1 Introduction

FUNDAMENTAL INSURANCE EQUATION

• Price = Cost + Profit
• Premium= Losses + LAE + UW Expenses + UW Profit

  Exposure

  The basic unit of risk underlying the insurance premium includes written, earned, unearned, and in-force exposures

• Written exposures are the total exposures arising from policies issued (i.e., underwritten or written) during a specified period of time, such as a calendar year or quarter.
• Earned exposures represent the portion of the written exposures for which coverage has already been provided as of a certain point in time.
• Unearned exposures represent the portion of the written exposures for which coverage has not yet been provided as of that point in time.
• In-force exposures are the number of insured units that are exposed to loss at a given point in time.

Premium

The amount the insured pays for insurance coverage includes written, earned, unearned, and in-force premium

• Written premium is the total premium associated with policies that were issued during a specified period.
• Earned premium represents the portion of the written premium for which coverage has already been provided as of a certain point in time.
• Unearned premium is the portion of the written premium for which coverage has yet to be provided as of a certain point in time.
• In-force premium is the full-term premium for policies that are in effect at a given point in time.

Claim

• An insurance policy entails the insured paying a premium to the insurer for coverage against specific events.
• When such an event occurs, the insured (now the claimant) files a claim. The event’s date is known as the date of loss or accident date.
• Claims unknown to the insurer at a given time are termed unreported claims or incurred but not reported (IBNR) claims
• After the report date, a claim becomes known to the company and is classified as a reported claim. It remains an open claim until settled, after which it is considered closed. However, claims can be reopened if further activity occurs post-closure.

Losses

Loss under an insurance policy is the compensation amount payable to the claimant. Loss terminology includes paid loss, case reserve, reported or case incurred loss, IBNR/IBNER reserves, and ultimate loss.

• Paid loss is amounts already disbursed to the claimant.
• Case reserve is an estimate of the amount of money required to ultimately settle the claim, it excludes any payments already made.
• Reported loss is equal to Paid loss + Case reserve
• Ultimate loss is the total required to settle the claim. Estimated Ultimate Loss equal to Reported Loss + IBNR Reserve + IBNER Reserve.

Loss Adjustment Expense (LAE)

In addition to the money paid to the claimant for compensation, loss adjustment expenses (LAE) are costs insurers incur while settling claims, divided into allocated loss adjustment expenses (ALAE) and unallocated loss adjustment expenses (ULAE):

LAE = ALAE + ULAE.

• ALAE are claim-related expenses that are directly attributable to a specific claim; for example, fees associated with outside legal counsel hired to defend a claim can be directly assigned to a specific claim.
• ULAE are claim-related expenses that cannot be directly assigned to a specific claim. For example, salaries of claims department personnel are not readily assignable to a specific claim and are categorized as ULAE.

Underwriting Expenses (UW Expenses)

In addition to loss adjustment expenses, underwriting expenses are costs insurers incur for various expenses in acquiring and servicing policies.

• Commissions and brokerage: are payments made to insurance agents or brokers for generating business.
• Other acquisition costs: are expenses beyond commissions and brokerage, such as those for media advertising and mailings to potential customers.
• General expenses: are the operational and any other miscellaneous costs. For example, upkeep for the home office.
• Taxes, licenses, and fees: are all taxes and miscellaneous fees paid by the insurer excluding federal income taxes. Premium taxes and licensing fees are examples of items included in this category.

Underwriting Profit (UW Profit)

The two primary profit sources for insurance companies are underwriting profit and investment income.

• Underwriting profit, also known as operating income, is the profit from individual policies calculated as income minus expenses, similar to profit definitions in other industries.
• Investment income is the profit from investing the funds that the insurance company holds.

Basic Insurance Ratios

A set of basic ratios to monitor and evaluate the appropriateness of an insurance company’s rates.

• Frequency: \(\text{Frequency} = \frac{ \text{# of Claims}}{\text{# of Exposures}}\).
  • Help measure the effectiveness of specific underwriting actions.
  • For example, if the number of claims is 100,000 and the number of earned exposures is 2,000,000, then the frequency is 5% (= 100,000 / 2,000,000).
• Severity: \(\text{average cost of claims} =\text{Severity} = \frac{\text{Losses}}{\text{# of Claims}}\).
  • \[\text{Paid Severity} = \frac{\text{Paid Losses}}{\text{# of Claims Closed}}\]
  • \[\text{Reported Severity} = \frac{\text{Reported Losses}}{\text{# of Claims Reported}}\]
  • Additionally, ALAE may be included or excluded from the numerator.
  • Help measure the loss trends.
  • For example, if the total loss dollars are $300,000,000 and the number of claims is 100,000, then the severity is $3,000 (= $300,000,000 / 100,000).
• Pure Premium (or Loss Cost): \(\text{average loss per exposure } =\text{Pure Premium} = \frac{\text{Losses}}{\text{# of Exposures}} = \text{Frequency} X \text{times Severity}\)
  • Typically, reported losses (or ultimate losses) and earned exposures are used.
  • The reported losses may or may not include ALAE and/or ULAE.
  • Help measure the overall loss cost trends.
  • For example, if total loss dollars are $300,000,000 and the number of exposures is 2,000,000, then the pure premium is $150 (= $300,000,000 / 2,000,000 = 5.0% x $3,000).
• Average Premium: \(\text{Average Premium} = \frac{\text{Premuim}}{\text{# Of Expesures}}\)
  • It is important that the premium and the exposures be on the same basis (e.g., written, earned, or in-force).
  • Changes in average premium may indicate
    • Rate change
    • Change in the mix of business, e.g. deductible shift, insured limit shift, risk shifts, and etc.
  • For example, if the total premium is $400,000,000 and the total exposures are 2,000,000, then the average
• Loss Ratio: \(\text{Loss Ratio} = \frac{\text{Losses}}{\text{Premium}} = \frac{\text{Pure premium}}{\text{Average Premium}}\).
  • Typically, the ratio uses total reported losses and total earned premiums; however, other variations are common. Companies may include LAE in the calculation of loss ratios (commonly referred to as loss and LAE ratios).
  • Help measure the adequacy of the rates
  • For example, if the total loss dollars are $300,000,000 and the total premium is $400,000,000, then the loss ratio is 75% (= $300,000,000 / $400,000,000).
• Loss Adjustment Expense Ratio (LAE Ratio): \({\text{LAERatio}} = \frac{\text{Loss Adjustment Expenses}}{\text{Losses}}\)
  • The loss adjustment expenses include both allocated and unallocated loss adjustment expenses. Companies may differ as to whether paid or reported (incurred) figures are used.
  • Companies monitor this ratio over time to determine if costs associated with claim settlement procedures are stable or not. A company may compare its ratio to those of other companies as a benchmark for its claims settlement procedures.

• Underwriting Expense Ratio: \(\text{UW Expense Ratio} = \frac {\text{UW Expense}}{\text{Premium}}\)

• Operating Expense Ratio: \(\text{OER} = \text{UW Expense Ratio} + \frac {\text{LAE}} {\text{Earned Premium}}\)
  • The OER is used to monitor operational expenditures and is key to determining overall profitability.
• Combined Ratio: \(\text{Combined Ratio} = \text{Loss ratio} + \frac{\text{LAE}}{\text{Earned Premium}} + \frac{\text{Underwriting Expenses}}{\text{Written Premium}}\)
  • In calculating the combined ratio, the loss ratio should not include LAE or it will be double counted.
  • some companies may compare underwriting expenses incurred throughout the policy to earned premiums rather than to written premiums. In this case, the companies may choose to define combined ratio as \(\text{Combined Ratio} = \text{Loss Ratio} + \text{OER}\)
• Retention Ratio: \(\text{Retention Ratio} = \frac{\text{# of Policies Renewed}}{\text{# of potential Renewal Policies}}\).
  • Retention ratios and changes in the retention ratios are monitored closely by product management and marketing departments
  • Retention ratios are used to gauge the competitiveness of rates and are very closely examined following rate changes or major changes in service
  • If 100,000 policies are invited to renew in a particular month and 85,000 of the insureds choose to renew, then the retention ratio is 85% (= 85,000 / 100,000).
• Close Ratio: \(\text{Closs ratio} = \frac{\text{# of Accepted Quotes}}{\text{# of Quotes}}\).
  • a prospective insured may receive multiple quotes and companies may count that as one quote or may consider each quote separately.
  • Close ratios and changes in the close ratios are monitored closely by product management and marketing departments.
  • Closed ratios are used to determine the competitiveness of rates for new business.
  • If the company provides 300,000 quotes in a particular month and generates 60,000 new policies from those quotes, then the close ratio is 20% (= 60,000 / 300,000).

CHAPTER 2 RATING MANUALS

For most lines of business, the following information is necessary to calculate the premium for a given risk:

• Rules
• Rate pages (i.e., base rates, rating tables, and fees)
• Rating algorithm
• Underwriting guidelines Generally speaking, the first three items are found in a company’s rating manual, and the underwriting guidelines are maintained in a separate proprietary underwriting manual.

  Rules

• Typically contains qualitative information that is needed to understand and apply the quantitative rating algorithms
• To be an aid in calculating premium

  RATEPAGES

• The rate pages generally contain the numerical inputs (e.g., base rates, rating tables, and fees) needed to calculate the premium
• Base Risk
  • A specific risk pre A specific risk pre–defined by the insurer – Represents a set of risk characteristics that are most common or target market.
• Base Rate
  • The rate applied to the base risk
  • Not the average rate
  • The rate for risks other than base risk is determined by modifying the base rate by a series of multipliers or addends or some unique mathematical expression.

  • Rating Algorithms

• describes in detail how to combine the various components in the rules and rate pages to calculate the overall premium charged for any risk that is not specifically pre-printed in a rate table.
• May include:
  • The order in which rating variables should be considered
  • Multiplicative or additive
  • Maximum and minimum premium

Underwriting Guidelines

• Underwriting guidelines may be used to specify – Decisions to accept, decline, or refer (to senior underwriter) risks – Company Placement
  • Tier placement – Schedule rating credits/debits

HOMEOWNERS RATING MANUAL EXAMPLE Check page 17 for homeowners rating manual example.

CHAPTER 3 RATEMAKING DATA

Data is used by actuaries for many purposes including ratemaking.

Internal Data

• Risk information including exposures, premiums, claim counts, and losses
• Accounting Information including Underwriting expense, ULAE

Risk Data

• Policy Database
  • Policy Identifier
  • Risk Identifier (one policy may have multiple risks, e.g. two cars, two locations of insured)
  • Relevant Dates (original effective date, original expiration date, date of amendment)
  • Premium
  • Exposure
  • Characteristic
• Claim Database
  • Policy Identifier
  • Risk Identifier
  • Claim Identifier
  • Claimant Identifier
  • Event Identifier
  • Relevant Loss Dates (Report data, data of loss payment, date of reserve change, data of claim status change)
  • Claim Status (open, closed, reopen, reclosed)
  • Paid Loss
  • Case Reserve
  • ALAE (ULAE is handled elsewhere)
  • Salvage & Subrogation
  • Type of Injury
  • Cause of Loss

Accounting Information

• Underwriting Expenses
  • Expenses incurred in the acquisition and servicing of the policies, including general expenses, other acquisition expenses, commissions and brokerage, taxes, licenses, and fees
• ULAE

Data Aggregation

• Calendar year
  • Consider all premium and loss transactions that occur during the 12-month calendar year
  • All premiums and exposures are fixed at the end of the CY
  • CY data is available quickly once the CY ends
  • Main disadvantage of CY is the mismatch in timing between premiums and losses. For example, premium earned during CY come from policies in force during that year, and losses may include payments and reserve changes on claims from policies issued years ago.
• Accident year
  • Considers losses for accidents that have occurred during a 12-month period, regardless of the policy-issued date
  • Premium and exposure are defined as the same as CY aggregation
  • Reported losses consist of loss payments made plus case reserves only for those claims that occurred during that year
  • Reported losses can and often do change at the end of AY
    • Additional Claims are reported and paid
    • Reserves are changed
  • Future development on those known losses needs to be estimated
  • Better match of premium and losses than CY
    • Losses on accidents occurring during the year are compared to premiums earned on policies during the same year
• Policy Year
  • Considers all premium and loss transactions on policies that were effective during a 12-month period
  • Premiums and exposures are not fixed until after the expiration date
  • Reported losses for PY consist of payments made plus case reserves only for those claims covered by policies effective during the year
  • Reported losses can and often do change at the end of AY
    • Additional claims are reported and paid
    • Reserves are changed
  • Best Match between losses and premium
    • Losses on policies effective during the year are compared with premiums earned on those same policies
  • Data takes longer to develop than both CY and AY
    • For a product with an annual policy term, premiums are not fully earned until 24 months after the start of the policy year.
• Report year
  • Considers losses for accidents that are reported during a 12-month period, regardless of when the claim occurred
  • Premium and exposures are defined as the same as CY aggregation
  • Reported losses consist of loss payments made plus case reserves only for those claims that are reported during that year
  • Reported losses can and often do change at the end of AY
    • Additional claims are reported and paid
    • Reserves are changed

Word Tokenization

Word tokenization is the process of splitting text into individual words.

Example: Input: “I like doing NLP”

Tokens: [“I”, “like”, “doing”, “NLP”]

Byte-Pair Encoding Tokenization

It’s used by a lot of Transformer models, including GPT, GPT-2, RoBERTa, BART, and DeBERTa. BPE training starts by computing the unique set of words used in the corpus, then iteratively merging the most frequently occurring character or byte pairs in a given dataset until a desired vocabulary size is reached.

Example: word “low” appear 5 times, “lowest” appear 2 times, “newer” appear 6 times, “wider” appear 3 times, “new” appear 2 time in the corpus.

Step 1: Initial Vocabulary

corpus                                              vocabulary

5   l o w _                                           _, d, e, i, l, n, o, r, s, t, w

2    l o w e s t _

6    n e w e r _

3    w i d e r _

2    n e w _

Step 2: Count and Merge Frequent Pairs (er appear 9 times)

corpus                                              vocabulary

5   l o w _                                          _, d, e, i, l, n, o, r, s, t, w, er

2    l o w e s t _

6    n e w er _

3    w i d er _

2    n e w _

step 3: continues to count and merge Frequent Pairs (er_ appear 9 times)

corpus                                              vocabulary

5   l o w _                                         _ , d, e, i, l, n, o, r, s, t, w, er, er_

2    l o w e s t _

6    n e w er_

3    w i d er_

2    n e w _

step n: continues until the max vocabulary set is reached.

Algorithm:

The articles I found which are very helpful for understanding Byte-Pair Encoding Tokenization:

• Byte-Pair Encoding: Subword-based tokenization algorithm written by Chetna Khanna
• BPE Tokenization Demystified: Implementation and Examples written by MartinLwx
• Vinija’s notes written by Jain, Vinija, and Chadha, Aman.

Advantage

• Handling Rare Words: By tokenizing words into subwords, BPE mitigates the issue of out-of-vocabulary(OOV) by allowing the model to understand and generate words that it might not have seen during training.
• Reduced Model Size: By tokenizing words into subwords, BPE allows the model to work with a more manageable vocabulary size. This can reduce the model’s complexity and the amount of memory required.

Implementation

I’ve implemented both word tokenizer and BPE tokenizer in Python from scratch:

dataset

python code

output

WordPiece Tokenization

Word Piece is also a subword tokenization algorithm, it’s similar to BPE but differs slightly in its approach to creating new tokens. Instead of just merging the most frequent pairs, WordPiece looks at the likelihood of the entire vocabulary and adds the token that increases the likelihood of the data the most. It’s first proposed in the paper “Japanese and Korean Voice Search”. The core idea of Word Piece method:

• does putting “a” and “b” together increase the ability to model the corpus? can be quantified by: p(‘a’,’b’) / (p(‘a’)p(‘b’))

The articles I found which are very helpful for understanding WordPiece:

• WordPiece: Subword-based tokenization algorithm written by Chetna Khanna
• Vinija’s notes written by Jain, Vinija, and Chadha, Aman.

Logistic Regression for Word Sense Disambiguation (WSD)

WSD is the process of identifying which sense of a word (i.e., its meaning) is used in a sentence when the word has multiple meanings. For example, the word “return” has different meanings such as:

1. come back to a place or situation.
2. To yield a profit or result.

This task is crucial for many natural language processing (NLP) applications, such as machine translation, information retrieval, and text understanding. Logistic Regression is a simple (easy to interpret) and useful model to identify the sense of a word in a sentence, but the pitfall of logistic regression is it might not come with the most accurate result.

More details about logistic regression or generalized linear regression can be seen in this post

Implementation

I’ve implemented logistic regression using PyTorch for Word Sense Disambiguation (WSD) from scratch: (One also can use CNN, RNN, or transformer to do this task, as those may yield better accuracy)

dataset

python code

output

Suppose we have \(X\) as a vector of \(p\) random variables, then the covariance matrix \(V\) is \(p \times p\). We want to find a vector \(\alpha_1 = (\alpha_{11}, \alpha_{12}, ..., \alpha_{1p})\) such that \(\alpha_{1}^{T} X\) gives maximum variance.

In mathematical terms, it is expressed as:

Taking derivatives with respect to \(\lambda\) and \(\alpha_1\):

Therefore, the maximum value for \(||\alpha_1||^2 = 1\) is \(\lambda_1\). Thus, a vector \(\alpha_1\) that gives maximum variance corresponds to the eigenvector for \(\lambda_1\).

For the second vector \(\alpha_2\), it is also required to be uncorrelated with \(\alpha_1\):

Continues this procedure will find all \(\lambda_k\), \(\alpha_k\) for kth P.C.

Given \(X\) is a vector of \(P\) and the covariance matrix is \(p \times p\):

Total variance = \(\lambda_1 + \lambda_2 + \ldots + \lambda_p\)

The \(k\)th principal component explains: \(\frac{\lambda_k}{\lambda_1 + \lambda_2 + \ldots + \lambda_p}\)

The first \(k\) principal components explain: \(\frac{\lambda_1 + \lambda_2 + \ldots + \lambda_k}{\lambda_1 + \lambda_2 + \ldots + \lambda_p}\)

Eigendecomposition

Principal Component Analysis (PCA) aims to reduce the dimensionality of high-dimensional data points by linearly projecting them onto a lower-dimensional space while preserving as much variance as possible or minimizing the reconstruction error made by this projection. This is achieved by performing eigendecomposition on the covariance matrix V of the dataset \(X\), such that \(V=PDP^T\). Since the covariance matrix is symmetric, eigenvalues in matrix D are always positive. An eigenvector that preserves maximum variance corresponds to the largest eigenvalue.

Singular Value Decomposition

Singular Value Decomposition (SVD) is more stable than eigendecomposition, especially when the dataset has fewer data points than dimensions. Directly performing PCA can be inefficient, so singular value decomposition (SVD) is preferred.

some good lecture notes found

lecture notes

Neural Network is inspired by biological phenomena in the human brain - how information is transferred as signals in the human brain. There are many types of Neural Networks that exist. The basic neural network architecture can be classified as Feed-forward Neural Network, CNN, RNN, and etc.

Gradient Descent:

Gradient Descent is a first-order iterative algorithm for finding a local minimum. It plays a crucial role in a neural network architecture. There are three types of gradient descent: \(\theta_{i+1} = \theta_i - \alpha \nabla J(\theta_i)\) where \(\alpha\) is the learning rate.

• Batch GD: Uses all data points for parameter updates.
• Stochastic GD: Uses one random data point for updates.
• Mini Batch GD: Uses a few random data points for updates.

The learning rate \(\alpha\) cannot be too large; otherwise, it will diverge. We generally adjust the learning rate after several epochs.

Activation Function

There are several commonly used activation functions such as ReLU, tanH, and the Sigmoid functions. Each of these functions has a specific usage.

As in linear regression notes, we know optimization requires the loss function to be convex. However, in deep learning, the loss function is generally non-convex due to non-linear activation functions. Because of this, we only guarantee finding a local minimum.

Forward Propagation

Propagating the input signal through layers to get the output.

Backward Propagation

Backpropagation: Adjusting the parameters based on the gradient of the error with respect to the network’s parameters. Chain Rule is involved.

CNN Architecture

Convolutional Neural Network consists of multiple layers like the input layer, Convolutional layer, Pooling layer, and fully connected layers.

It’s particularly useful for finding patterns in images to recognize objects.

Kernel, Stride, and Padding

• Kernel is used to extract features from images; it can be viewed as a parameter in the model.
• Stride controls the movement of the Kernel across the input image. If strides = 2, it will skip the next 2 pixels.
• Padding controls the addition of extra pixels around the borders of the input images or feature map. If padding = 2, it will add two column/row pixels around the borders.

ResNet

ResNet is the neural network architecture proposed by Kaiming He in the paper Deep Residual Learning for Image Recognition. It is widely known for its outstanding performance in image recognition. One of the problems ResNets solves is the famous known vanishing gradient. This is because when the network is too deep, the gradients from where the loss function is calculated easily shrink to zero after several applications of the chain rule. This results in the weights never updating their values, and therefore, no learning is being performed.

Decision trees are a type of supervised machine learning algorithm used for classification and regression tasks with a tree-like model. The basic structure of a tree consists of the root node, decision node, and leaf node. The general procedure is that it starts at the root, splits based on the best attribute, and repeats until meeting stopping criteria (e.g., all instances in a leaf node belong to the same class).

There are various splitting criteria used by the regression tree and the classification tree

• Gini impurity: a measure of variance across the K classes of the qualitative response. It is widely used in classification trees.
  \(Gini\ Index = \sum_{i} p_{i}(1-p_{i})\)
  For a two-class problem, the Gini impurity for a given node can be written as:
  \(Gini\ Impurity = p_{1}(1-p_{1}) + p_{2}(1-p_{2})\)
• Entropy: a measure similar to the Gini index.
  \(Entropy = -\sum_{i} p_{i} \ln(p_{i})\)
  For a two-class problem, the entropy for a given node can be written as:
  \(Entropy = -p \ln(p) - (1 - p) \ln(1 - p)\)
• Sum of Squared Error (SSE): Unlike the previous two metrics, SEE is widely used in a regression task.
  \(SSE = \sum_{i \in S_{1}} (y_{i} - \bar{y}_{1})^{2} + \sum_{i \in S_{2}} (y_{i} - \bar{y}_{2})^{2}\)

Procedure

1. Encode categorical variables, and handle missing values.
2. Build the Tree
   • use an appropriate metric to find the best splits, and split based on the attribute that results in the most significant reduction in heterogeneity.
3. Limit the tree depth to prevent overfitting.
4. Use cross-validation to assess the model’s performance (optional).
5. Use the trained model on test data

Emsemble Models

Ensemble models combine the results of multiple models to achieve a better result. The core idea is that the aggregation of multiple models can often outperform a single model. There are some advantages over individual trees such as improved accuracy, reduced overfitting, increased robustness, insights into feature importance, and flexibility. Random Forest and Gradient Boosting are popular examples of ensemble decision trees that are highly effective in various domains.

Bootstrap

• Bootstrapping is a fundamental technique in ensemble modeling. It involves generating additional samples by repeatedly sampling from the existing observations with replacement, as new samples are available, the accuracy of statistical estimates (bias and variance) can be assessed.
• More specifically, each subset has the same size as the original dataset, and random sampling aids in estimating the mean and standard deviation by resampling from the dataset.

Bagging

• Bagging is based on the bootstrap. It combines the results of multiple models to obtain a more generalized and robust prediction. It creates subsets (bags) from the original dataset using random sampling with replacement, and each subset is used to train a base model or weak model independently. These models run in parallel and are independent of each other.
• Random Forest is a variant of bagging that seeks to further improve the prediction accuracy of a tree-based model. While bagging involves averaging identically distributed and possibly correlated trees, random forests seek to make the bagged trees less correlated via an additional degree of randomization - every split is allowed to use only m predictors instead of the full set of p predictors.

Boosting

• Boosting uses a different approach to ensemble learning than bagging and random forest. Instead of making independent bootstrapped samples, fitting a decision tree to each bootstrapped sample separately, and averaging the predictions of these trees to reduce variance, boosting builds a sequence of inter-dependent tress, each fitted to the residuals of the preceding tree and a scaled-down version of the current tree’s prediction is subtracted from the preceding tree’s residual to from the new residual.
• Through this iterative process, boosting aims to convert a collection of weak learners into a stronger and more accurate model. The final model is a weighted combination of all the models.

well-known boosting models:

1. Gradient Boosting Machines (GBM): A powerful boosting technique that builds trees sequentially, with each tree learning and correcting errors made by the previous one. GBM minimizes a loss function (e.g., sum squared error for regression, cross-entropy for classification) by using gradients.

2. XGBoost (Extreme Gradient Boosting): A highly efficient and scalable implementation of gradient boosting. It introduces several regularization techniques to prevent overfitting and improve performance.

3. LightGBM: Another high-performance gradient boosting framework developed by Microsoft. It’s known for its speed and memory efficiency, making it suitable for large datasets. LightGBM uses a novel technique called Gradient-based One-Side Sampling (GOSS) to handle large datasets more effectively.

4. CatBoost: Developed by Yandex, CatBoost is a gradient-boosting library that works well with categorical features without the need for one-hot encoding or preprocessing. It employs techniques like ordered boosting and oblivious trees to improve performance.

XGBoost, LightGBM, and CatBoost are very popular models, providing high accuracy.

Generalized linear regression is an extension of ordinary linear regression. The response variable can be categorical, and the distribution of residuals can be non-normal. Here’s a summary:

• Response variable in GLR is no longer confined to the class of normal distribution; it only needs to be a member of the linear exponential family of distributions.
• GLR uses a link function where the response mean is linearly related to the predictors.

Linear Exponential Family of Distribution

Mathematically, a distribution is a member of the linear exponential family of distributions if its probability function can be expressed as:

where:

• \(y\) is the argument of the probability function.
• \(\theta\) is the parameter of interest.
• \(\phi\) is the scale parameter.
• \(b(\theta)\) and \(S(y,\phi)\) are functions of \(\theta\) and \(\phi\), respectively.

For further details, check this website.

Link Functions

Link functions connect the response distribution’s mean to the predictors. It’s a monotonic function that links the mean of the response to a linear combination of predictors:

where \(g(\cdot)\) is the link function, \(\mu\) is the response mean, \(X\) is the matrix of predictors, and \(w\) is the coefficients.

The monotonicity property allows us to invert the link function and estimate the response mean \(\mu\) as soon as the parameter vector \(\beta\) has been estimated from the data:

Canonical Link Function

There are many possible link functions connecting response distribution’s mean to the linear predictor. One comman choice of the link function is the canonical link, which sets the sysmatic component \(\eta\) of the model equal to the paramter \(\theta\).

\(g(\mu) = \eta = \theta\)

Logistic Regression

Logistic regression is one of the most famous generalized linear regression techniques.

• For the classification of two classes (e.g., \(t=1\) or \(t=0\)), the link function used is the logistic function (sigmoid curve) defined as \(f(z) = \frac{1}{1+e^{-z}}\).
• For multiclass classification, there’s an extension called the softmax function defined as:

Let’s derive the optimal weights that minimize least squares errors.

First, the logistic regression model is represented as:

where \(h(x) = \Theta(w^Tx)\) and \(\Theta\) is the logistic function defined as \(\Theta(s) = \frac{e^{s}}{1+e^{s}}\).

Thus,

One reason for choosing the mathematical form \(\Theta(s) = \frac{e^s}{1+e^s}\) is that it leads to this simple expression for \(P(y|x)\)

The likelihood for a sample of size \(N\) is given by:

Maximizing the likelihood:

The first-order derivative of the logistic regression objective function \(E(w)\) with respect to the weight vector \(w\) is derived as follows:

This concludes the derivation.

Decision Boundary

The logistic function is used to map the output of a linear combination of features (i.e., the weighted sum of input features plus a bias term) to a probability value between 0 and 1. And since the logistic function(sigmoid function) is a monotonically increasing function - when x increases, y increases, the decision boundary remains linear as in linear regression.

Proof of Decision Boundary for one-variable when p(y=1|x) = p(y=0|x):
\(\frac{1}{1 + e^{-(\beta_0 + \beta_1 x)}} = \frac{1}{2}\)
\(2 = 1 + e^{-(\beta_0 + \beta_1 x)}\)
\(1 = e^{-(\beta_0 + \beta_1 x)}\)
\(\ln(1) = -(\beta_0 + \beta_1 x)\)
\(0 = -(\beta_0 + \beta_1 x)\)
\(0 = \beta_0 + \beta_1 x\)
\(x = -\frac{\beta_0}{\beta_1}\)

This can be extended to multi-variables.

The plot below is an example of a decision boundary for logistic regression:

Support Vector Machine

Different from Generalized Linear regression, SVM is to find a plane that has the maximum margin, i.e. the maximum distance between data points of both classes. This might yield better accuracy as we also maximize the margin distance, which provides some reinforcement. Hence, future data points can be classified with more confidence(higher accuracy).

Hinge Loss: is the loss function used in Support Vector Machines (SVM). The objective is to find the values of \(w\) and \(b\) that minimize:
\(\min_{w, b} \frac{1}{2}||w||^2 + C \sum_{i=1}^{n} \xi_i\)
Subject to the constraints:
\(y_i(w \cdot x_i + b) \geq 1 - \xi_i, \quad \forall i = 1, \ldots, n\)
\(\xi_i \geq 0, \quad \forall i = 1, \ldots, n\)

Kernel Tricks: The kernel function computes the inner product between two vectors

• Linear Kernel: The simplest form, where the kernel is just the dot product of two vectors. It is used when the data is linearly separable.
  \(K(x, x') = x \cdot x'\)

• Polynomial Kernel: Allows learning of non-linear models by mapping inputs into polynomial feature space.
  \(K(x, x') = (\gamma x \cdot x' + r)^d\)

• Radial Basis Function (RBF) or Gaussian Kernel: One of the most common kernels. It can map the inputs into an infinite-dimensional space, providing great flexibility in fitting the data.
  \(K(x, x') = \exp(-\gamma \|x - x'\|^2)\)

Decision Boundary

The decision boundary of an SVM is determined by the choice of the kernel:

• Linear Kernel: The decision boundary is linear.
• Non-linear Kernels (Polynomial and RBF): The decision boundary is non-linear

An example of an RBF decision boundary is:

Multicollinearity occurs when two or more predictors in a regression model are moderately or highly correlated with each other. In the least squares estimation,

Thus, when collinearity is present, coefficient estimates (\(w\)) for the correlated variables become unstable (X is not invertible).

Several methods address the multicollinearity problem:

• Step-wise regression (variable selection using adjusted \(R^2\), AIC, BIC, or Mallow’s \(C_p\)).
• Shrinkage methods (Ridge, Lasso, or Elastic Net).

Step-Wise Regression (Variable Selection)

Stepwise regression selects a subset of predictor variables based on statistical criteria such as adjusted \(R^2\), AIC, BIC, or Mallow’s \(C_p\).

Adjusted \(R^2\)

The coefficient of determination \(R^2\) is defined as \(R^2 = 1- \frac{RSS}{TSS}\). However, as more predictors are added, \(R^2\) always increases. To address this, statisticians developed adjusted \(R^2\), which rescales \(R^2\) to account for the number of predictors.
Adjusted \(R^2\) formula: \(R^2_{\alpha} = 1- \frac{\frac{RSS}{n-p-1}}{\frac{TSS}{n-1}}\)
As can be seen, adjusted \(R^2\) rescales the ordinary \(R^2\) by dividing RSS and TSS by their respective degrees of freedom in the ANOVA table. The goal of it is that the value would not always increase as more predictor variables are added in. It’s sort of acting like a penalty for adding more variables.

AIC, BIC, and Mallow’s \(C_p\)

These model comparison statistics penalize complex models to favor simplicity, following the parsimony principle.

Here, \(p\) is the number of predictors, \(s_{\text{full}}^2\) is the mean squared error of the full model, \(L\) maximum value of the likelihood function for the model.

Shrinkage Methods

Shrinkage methods regulate or shrink coefficient estimates towards zero.

Ridge Regression

In Ridge regression, the objective function is:

where \(\lambda \geq 0\) is called the tuning parameter. The \(\lambda\) value is a hyperparameter that needs to be tuned (using cross-validation) to obtain the value of \(\lambda\) such that the error (RSS) is minimized.

Derive optimal w using matrix annotation: \(E(w) = ||xw-y||^2+\lambda ||w||^2\)

Lasso Regression

In Lasso regression, the objective function is similar to Ridge, but the penalty term involves the absolute value of the coefficients:

In contrast to Ridge regression, the Lasso has the remarkable effect of shrinking the coefficient estimates to be exactly zeros. Therefore, Lasso can also be used for variable selection.

Elastic Net

Elastic Net combines the penalties of both Lasso and Ridge regression techniques:

Overall, these methods provide effective ways to handle multicollinearity in regression models.

Linear regression is a statistical method used to model the linear relationship between a dependent variable y and one or more independent variables \(x_{1}, x_{2}, \ldots, x_{p}\). It is simple, yet it remains one of the most useful analytical tools. While it may not always yield the highest accuracy, the model simplicity and interpibility are very good.

The linear regression model takes the form:

where:

• y is the dependent variable
• \(x_1, x_2, \ldots, x_p\) are the independent variables
• \(\beta_0, \beta_1, \ldots, \beta_p\) are the coefficients (parameters) of the model
• \(\varepsilon\) is the error term, which represents the difference between the observed and predicted values of y.

Often these \(n\) equations are stacked together and written in matrix notation as:

where:

Least Square Estimation

Least square error tries to minimize the sum of square residuals: \(\varepsilon = \sum(y-\hat{y})^2\).

To minimize this \(\varepsilon\):

where \(X^TX\) is invertible.

It’s known that \(x_{1}, x_{2}, \ldots, x_{p}\) need to be independent of each other for the matrix \(X\) to be invertible. In practice, we often assume that variables are independent from each other, even if they exhibit small correlations. However, if variables are strongly correlated (dependent), they shouldn’t be included in the linear model, as they will lead to incorrect results.

There are several methods to handle multiple dependent variables in regression models:

• Variable selection based on criteria such as Bayesian Information Criterion (BIC) or Akaike Information Criterion (AIC).
• Regularization techniques such as Ridge regression, Lasso regression, and Elastic Net regression, which penalize the inclusion of multiple correlated variables in the model.

These methods help address the issue of multicollinearity and improve the accuracy and reliability of the regression model.

Derivation of the Coefficients

Let’s simplify the derivation for a regression with only one predictor:

Then, the error function E can be expressed as:

To minimize the value of E:

take derivative respect to a

we find optimal \(a = \bar{y} - b\bar{x}\)

then take derivative respect to b

Alternatively:

The Coefficient of Determination R^2

• SSE (Error/Residual Sum of Squares): \(\sum_{i=1}^{n} (y_i - \hat{y_i})^2\), the value we are trying to minimize.
• SSR (Regression Sum of Squares): \(\sum_{i=1}^{n} ({\hat{y_i}} - \bar{y})^2\), the value we are trying to maximize.
• SST (Total Sum of Squares): \(\sum_{i=1}^{n} (\hat{y_i} - \bar{y})^2\)

loosely speaking, \(R^2\) is how many precentage of variance being captured by the model. Therefore, \(R^2 = \frac{SSR}{SST} or 1-\frac{SSE}{SST}\)

Maximum Likilihood Estimation

\(\begin{align*} f(y_i | x_i, \beta_0, \beta_1, \ldots, \beta_p) &= \\ &\frac{1}{\sqrt{2 \pi \sigma^2}} \exp\left(-\frac{(y_i - (\beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2} + \cdots + \beta_p x_{ip}))^2}{2\sigma^2}\right) \end{align*}\)

The likelihood function for the entire dataset \(\{(x_i, y_i)\}_{i=1}^n\) is the product of the likelihoods of individual observations:

Solving this system of equations gives us the maximum likelihood estimates \(\hat{\beta}_0, \hat{\beta}_1, \ldots, \hat{\beta}_p\).

This approach provides estimates that maximize the likelihood of observing the given data under the assumed linear regression model. When \(f_θ\) is a normal distribution with zero mean and variance θ, the resulting estimate is identical to the least square estimation.

Other Estimation Technique (Quantile Regression)

Other estimation techniques like quantile regression are also very useful. It focuses on the conditional quantiles of y given X rather than the conditional mean of y given X. Least square estimation is highly affected by outliers, but quantile regression is more robust to outliers. Quantile regression is very useful in some real-world data with outliers.

Roger Koenker Quantile Regression paper

Convexity

A good error function is required to be convex, which has only one minimum point. This property simplifies the optimization process, as there is no risk of getting stuck in a suboptimal solution. Mathematicians/Statisticians put much effort into designing a convex error function. In addition, the least square error is the quadratic objective function, while the quantile error function is the linear objective function. In terms of efficiency, linear objective function like quantile regression runs much faster than least-squared regression in large datasets.