type
Post
Created date
Jun 16, 2022 01:21 PM
category
Data Science
tags
Machine Learning
Machine Learning
status
Published
Language
From
summary
slug
password
Author
Priority
Featured
Featured
Cover
Origin
Type
URL
Youtube
Youtube
icon
Definition
- As the name suggests, LDA is a linear model for classification and dimensionality reduction. It is used to solve the dimension reduction method for categorical data. [vidhya]
- Start finding directions that maximise the separation between classes, then use these directions to predict the class of individuals. These directions, called linear discriminants, are linear combinations of predictor variables. [STHDA]
- LDA MAXIMISES the between-class variance (‘separability’) and MINIMISES the within-class variance. [Here]
- Finds a linear combination of predictors that maximizes the separation between groups. [Brendi]
Theory
Problem:
Solution:
Assumption: Only works when
The predictors are normally distributed. [vidhya] (i.e., All samples come from normal populations [Slides] )
![notion image](https://www.notion.so/image/https%3A%2F%2Ffile.notion.so%2Ff%2Ff%2Fe1c5ba0e-562a-49b0-8505-ce5e38fc061b%2Fe449618d-9b05-4cd8-bf36-c9854db9122f%2FUntitled.png%3Ftable%3Dblock%26id%3D47b0772b-b58c-477c-8d45-8b0fb85c7914%26spaceId%3De1c5ba0e-562a-49b0-8505-ce5e38fc061b%26expirationTimestamp%3D1725818400000%26signature%3DzcW0dqBsKLze_k3alajJ6q3lfW4b3_f25K_5I51plbw?table=block&id=47b0772b-b58c-477c-8d45-8b0fb85c7914&cache=v2)
- Each of the classes has identical variance-covariance matrices. [vidhya] RMB the shape of the data is determined by the variance-covariance matrices.
- The variances and covariances are the same for the y=1y=1 group and y=0y=0 group. (Here)
Example
Brendi
![notion image](https://www.notion.so/image/https%3A%2F%2Ffile.notion.so%2Ff%2Ff%2Fe1c5ba0e-562a-49b0-8505-ce5e38fc061b%2F251a62e7-d3fb-46f0-884c-96673821e3a0%2Flda-overview.jpeg%3Ftable%3Dblock%26id%3D7b503cb0-e043-4130-b279-a342c333b7c1%26spaceId%3De1c5ba0e-562a-49b0-8505-ce5e38fc061b%26expirationTimestamp%3D1725818400000%26signature%3DVMMsAZE9Xyubm35wfEBbL_xcKHMJWFYLrrxOMXHSVsI?table=block&id=7b503cb0-e043-4130-b279-a342c333b7c1&cache=v2)
![notion image](https://www.notion.so/image/https%3A%2F%2Ffile.notion.so%2Ff%2Ff%2Fe1c5ba0e-562a-49b0-8505-ce5e38fc061b%2F16c84d91-abd8-4621-80f6-f129e61f4b14%2Fvar-cov.jpeg%3Ftable%3Dblock%26id%3D4d7475a2-677c-4574-b3d9-7278302d9d04%26spaceId%3De1c5ba0e-562a-49b0-8505-ce5e38fc061b%26expirationTimestamp%3D1725818400000%26signature%3DGbrqWfiHHos3j4utHsqf5GvjKu21JbOXD0ItKYfXUKU?table=block&id=4d7475a2-677c-4574-b3d9-7278302d9d04&cache=v2)
![notion image](https://www.notion.so/image/https%3A%2F%2Ffile.notion.so%2Ff%2Ff%2Fe1c5ba0e-562a-49b0-8505-ce5e38fc061b%2Feebf1911-73e0-449d-ab1a-5d525eb50116%2Flda-fit.jpeg%3Ftable%3Dblock%26id%3Dc4751dcb-8441-48f2-9062-021b0302a96c%26spaceId%3De1c5ba0e-562a-49b0-8505-ce5e38fc061b%26expirationTimestamp%3D1725818400000%26signature%3DI1qRHQ0xGyX3NtKAqJerr9jlIlMlrXDcI78R1-Q9wAQ?table=block&id=c4751dcb-8441-48f2-9062-021b0302a96c&cache=v2)
Bredwin
![notion image](https://www.notion.so/image/https%3A%2F%2Ffile.notion.so%2Ff%2Ff%2Fe1c5ba0e-562a-49b0-8505-ce5e38fc061b%2F4952078d-2f1a-441c-aa0a-f27fb68c6f4b%2FLDA_drawing.jpeg%3Ftable%3Dblock%26id%3D8d299a96-42a6-4e7b-b5c1-0b8e31a28787%26spaceId%3De1c5ba0e-562a-49b0-8505-ce5e38fc061b%26expirationTimestamp%3D1725818400000%26signature%3DbyiOelQ6aPyo1kz5R9XKvIZpio6MczNm185mPEGcL-Q?table=block&id=8d299a96-42a6-4e7b-b5c1-0b8e31a28787&cache=v2)
R code
olive <- read_csv("http://ggobi.org/book/data/olive.csv") %>% dplyr::select(-`...1`, -area) %>% mutate(region = factor(region)) # Standardise variables olive_std <- olive %>% mutate(across(where(is.numeric), ~ (.x - mean(.x)) / sd(.x)))
library(discrim) library(MASS) set.seed(775) olive_split <- initial_split(olive_std, 2/3, strata = region) olive_train <- training(olive_split) olive_test <- testing(olive_split) lda_mod <- discrim_linear() %>% set_engine("MASS", prior = c(1/3, 1/3, 1/3)) %>% translate() olive_lda_fit <- lda_mod %>% fit(region ~ ., data = olive_train) olive_lda_fit
R interpretation
LDA determines group means and computes, for each individual, the probability of belonging to the different groups. The individual is then affected to the group with the highest probability score.
The
lda()
outputs contain the following elements:- Prior probabilities of groups: the proportion of training observations in each group. For example, there are 31% of the training observations in the setosa group
- Group means: group center of gravity. Shows the mean of each variable in each group.
- Coefficients of linear discriminants: Shows the linear combination of predictor variables that are used to form the LDA decision rule. for example,
LD1 = 0.91*Sepal.Length + 0.64*Sepal.Width - 4.08*Petal.Length - 2.3*Petal.Width
. Similarly,LD2 = 0.03*Sepal.Length + 0.89*Sepal.Width - 2.2*Petal.Length - 2.6*Petal.Width
.
- The proportion of trace: the percentage separation achieved by each discriminant function.
- proportion of between-class variance
For example, LDA 1 has 99.05% to separability.
![notion image](https://www.notion.so/image/https%3A%2F%2Ffile.notion.so%2Ff%2Ff%2Fe1c5ba0e-562a-49b0-8505-ce5e38fc061b%2F7fcce85c-6419-408d-ac70-b346e403183f%2FUntitled.png%3Ftable%3Dblock%26id%3Dff133a44-7622-45ff-828b-d37260f198d1%26spaceId%3De1c5ba0e-562a-49b0-8505-ce5e38fc061b%26expirationTimestamp%3D1725818400000%26signature%3DSd4wlu7MkBfaKgEp8y2uOkFcVaftyJUlt5N0oyIry_g?table=block&id=ff133a44-7622-45ff-828b-d37260f198d1&cache=v2)
Using the function
plot()
produces plots of the linear discriminants, obtained by computing LD1 and LD2 for each of the training observations.Further interpretation and prediction can be read via [STHDA].
Math
Bayes theorem is about probabilities so if we want to decide whether this observation belongs to group 1 or group 2 or group 3, we look at the value on its density function in whichever one has the highest value on the density function, that's the most likely class that it belongs to that's what Bayes theorem corresponds to. [Lecture]
![notion image](https://www.notion.so/image/https%3A%2F%2Ffile.notion.so%2Ff%2Ff%2Fe1c5ba0e-562a-49b0-8505-ce5e38fc061b%2Fb6c8e2b7-1b19-4308-a597-56d851650a1c%2FUntitled.png%3Ftable%3Dblock%26id%3Dbada19cb-edc1-4a11-b533-c77e8befc1a5%26spaceId%3De1c5ba0e-562a-49b0-8505-ce5e38fc061b%26expirationTimestamp%3D1725818400000%26signature%3Dx8mjejgZ00g5iLwtP7eiBVAWnNMLIyifVVK90hrOmKs?table=block&id=bada19cb-edc1-4a11-b533-c77e8befc1a5&cache=v2)
![LDA finds the mean vectors of each class, then finds projection direction (rotation) that maximizes separation of means:](https://www.notion.so/image/https%3A%2F%2Fi.stack.imgur.com%2FSLvnn.png?table=block&id=664ef89e-a9f8-4798-b880-c43b299ba6e4&cache=v2)
![It also takes into account within-class variance to find a projection which minimizes overlap of distributions (covariance) while maximizing the separation of means:](https://www.notion.so/image/https%3A%2F%2Fi.stack.imgur.com%2FfQExt.png?table=block&id=813e4eaf-af8b-4511-897e-979ae6ea9abe&cache=v2)
Another source to look into the math : [vidhya]
FAQ
PCA vs LDA
Component axes here mean direction.
In practice, often a PCA is done followed by an LDA for dimensionality reduction.
- Very similar; only differ that LDA does not have class-specific covariance matrices, but one shared covariance matrix among the classes. [TDS]
What is the difference between LDA and logistic regression
(Here)
If the classes are well-separated estimates from logistic regression tend to be unstable.
If there are a small number of observations, estimates from logistic regression tend to be unstable.
![notion image](https://www.notion.so/image/https%3A%2F%2Ffile.notion.so%2Ff%2Ff%2Fe1c5ba0e-562a-49b0-8505-ce5e38fc061b%2Faeffab5d-d651-4867-96ef-3a1f8dc727e6%2FUntitled.png%3Ftable%3Dblock%26id%3De5d80d69-bcd1-4f21-a096-1e01ed1ad1fd%26spaceId%3De1c5ba0e-562a-49b0-8505-ce5e38fc061b%26expirationTimestamp%3D1725818400000%26signature%3DmnisssMv-teqaFfASxEVjhf3MoWBaJko5hursqAoS1w?table=block&id=e5d80d69-bcd1-4f21-a096-1e01ed1ad1fd&cache=v2)
Reference
Brendi notes [Brendi]
Slides [Slides]
Extra resources
[STHDA] explains how to write LDA in python
Standford lecture
Calculation illustrations
(930) Linear discriminant analysis (LDA) - simply explained - YouTube - Calculation illustration
(930) Lecture 20- Linear Discriminant Analysis ( LDA) (with Solved Example) - YouTube (Same example, but this one is more concise)
Lab
3550 exam explanation
- Author:Jason Siu
- URL:https://jason-siu.com/article/83264fa0-eae8-495d-9ca5-f26f29ca95f1
- Copyright:All articles in this blog, except for special statements, adopt BY-NC-SA agreement. Please indicate the source!
Relate Posts