type
Post
Created date
Jun 16, 2022 01:21 PM
category
Data Science
tags
Machine Learning
Machine Learning
status
Published
Language
From
summary
slug
password
Author
Priority
Featured
Featured
Cover
Origin
Type
URL
Youtube
Youtube
icon
Logistic regression
When do we use Logistic regression?
- When the response variable are binary, which takes one of two values, 0 or 1
Math of Logistic regression: Log-odd
- Sum(Square Residuals) + ()
(More details can be read from the week 9 lecture slides.)
What if there are lot of variables? penalty
Problem :
When there are lot of variables, Multicollinearity is introduced. That means independent variables / Predictor variables in a regression model are correlated. Which is not under the assumption of regression.
How do you know if your data is Multicollinearity?
- i.e. coefficient. is very big.
- Least-squares are unbiased, and variances are large, this results in predicted values being far away from the actual values.
- Above of which leads to overfitting and instability.
Solution:
We add a penalty term that discourages large values of i.e. coefficient.
How do you know if add a penalty?
- Essentially, we tweak around the value of λ to bring trades off between bias and variance.
- When the coefficients are (asymptotically unbiased)
- Increasing decreases the s. When the coefficients are all zero (no variance) [ 係反比關係]
In the coding, we change param of mixture
.
- When
mixture = 0
, this is called Ridge-regression. - The s are shrunk.
- When
mixture = 1
, this is called Logistic lasso. - The s are shrunk by making the least important ones exactly zero.
Resource about Logistic regression
![notion image](https://www.notion.so/image/https%3A%2F%2Ffile.notion.so%2Ff%2Ff%2Fe1c5ba0e-562a-49b0-8505-ce5e38fc061b%2F5ca4c95d-7fd2-4da7-b17f-61586b655e82%2FUntitled.png%3Ftable%3Dblock%26id%3D2dab90d3-ac4a-474e-86e1-7eefbb829da3%26spaceId%3De1c5ba0e-562a-49b0-8505-ce5e38fc061b%26expirationTimestamp%3D1725624000000%26signature%3DCUJPmVx5auY8u8VuC8XTpss609hzl5ZoOrcLLfgcEDA?table=block&id=2dab90d3-ac4a-474e-86e1-7eefbb829da3&cache=v2)
Assumption
- It assumes that there is minimal or no multicollinearity among the independent variables.
- It usually requires a large sample size to predict properly.
- It assumes the observations to be independent of each other.
Pros
- Easy to interpret, implement and train. Doesn’t require too much computational power.
- Makes no assumption of Class-Distribution.
- Fast in classifying unknown records.
- Can easily accommodate new data points.
- Is very efficient when features are linearly separable.
Shortcomings
- Tries to predict precise probabilistic outcomes. Which leads to overfitting in high dimensions.
- Since has a linear decision surface. So, can’t solve non-linear problems.
- Tough to obtain complex relations other than linear relations.
- Requires very less or no multicollinearity.
- Need a large dataset and also sufficient training examples for all the categories to make correct predictions.
Example
Brendi
![notion image](https://www.notion.so/image/https%3A%2F%2Ffile.notion.so%2Ff%2Ff%2Fe1c5ba0e-562a-49b0-8505-ce5e38fc061b%2F300dbde3-aa11-4058-8c8d-65aeaf3c7490%2Flinear-vs-logistic.jpeg%3Ftable%3Dblock%26id%3Da396dc8e-f896-41e3-b5a5-6bf13916a08d%26spaceId%3De1c5ba0e-562a-49b0-8505-ce5e38fc061b%26expirationTimestamp%3D1725624000000%26signature%3Ddzhd0hJe4lKt9HurLVGMWiHitwADbuOkwHfN3C3XOFU?table=block&id=a396dc8e-f896-41e3-b5a5-6bf13916a08d&cache=v2)
![notion image](https://www.notion.so/image/https%3A%2F%2Ffile.notion.so%2Ff%2Ff%2Fe1c5ba0e-562a-49b0-8505-ce5e38fc061b%2Fb614d708-7f4f-48e5-b7b0-4337ce594eac%2Finterpret-logistic.jpeg%3Ftable%3Dblock%26id%3D3aca2ed1-05bf-4d08-8f54-879cd7dd5689%26spaceId%3De1c5ba0e-562a-49b0-8505-ce5e38fc061b%26expirationTimestamp%3D1725624000000%26signature%3DxL_JCdKayRqFVIoffnhJToCdJailKHmpnUu1-I_jV3U?table=block&id=3aca2ed1-05bf-4d08-8f54-879cd7dd5689&cache=v2)
![notion image](https://www.notion.so/image/https%3A%2F%2Ffile.notion.so%2Ff%2Ff%2Fe1c5ba0e-562a-49b0-8505-ce5e38fc061b%2Feadcb86d-f002-45bf-b551-2bdeca6c4aac%2Flogit-function.jpeg%3Ftable%3Dblock%26id%3D54acb23b-be6f-4f1c-8801-738a17a9ab95%26spaceId%3De1c5ba0e-562a-49b0-8505-ce5e38fc061b%26expirationTimestamp%3D1725624000000%26signature%3DxAVJnPTqQA_RLV2hvBqLYxPGspyIr6mkRI-oSZHGIeY?table=block&id=54acb23b-be6f-4f1c-8801-738a17a9ab95&cache=v2)
Brenwin
![notion image](https://www.notion.so/image/https%3A%2F%2Ffile.notion.so%2Ff%2Ff%2Fe1c5ba0e-562a-49b0-8505-ce5e38fc061b%2Feb0132b6-c187-4c26-8e78-27d4dc35f1f8%2Flogistic_reg_drawing.jpeg%3Ftable%3Dblock%26id%3D821ff57a-2ae3-4318-aa80-f806c46d4329%26spaceId%3De1c5ba0e-562a-49b0-8505-ce5e38fc061b%26expirationTimestamp%3D1725624000000%26signature%3DKtTIqT79DuVx6boYoCtvMoen0XkrqbzZAUm9HaVfmNg?table=block&id=821ff57a-2ae3-4318-aa80-f806c46d4329&cache=v2)
Math
Example of writing the equation
![notion image](https://www.notion.so/image/https%3A%2F%2Ffile.notion.so%2Ff%2Ff%2Fe1c5ba0e-562a-49b0-8505-ce5e38fc061b%2F68bc68d6-da18-43b3-b972-8faf2fa982b6%2FUntitled.png%3Ftable%3Dblock%26id%3D5e350a1f-a28e-4894-98dc-6d64002b9130%26spaceId%3De1c5ba0e-562a-49b0-8505-ce5e38fc061b%26expirationTimestamp%3D1725624000000%26signature%3DTqk-6Bikh2OrhgnATutKftZX-2wyd4MEQ9v_sSWN2j4?table=block&id=5e350a1f-a28e-4894-98dc-6d64002b9130&cache=v2)
![notion image](https://www.notion.so/image/https%3A%2F%2Ffile.notion.so%2Ff%2Ff%2Fe1c5ba0e-562a-49b0-8505-ce5e38fc061b%2F6359989f-941d-477e-a0b9-ba86691a095c%2FUntitled.png%3Ftable%3Dblock%26id%3D292282b8-1a87-46f6-9de4-1f8b021fbb2d%26spaceId%3De1c5ba0e-562a-49b0-8505-ce5e38fc061b%26expirationTimestamp%3D1725624000000%26signature%3DsfxZJZ_Bem1KjokKr4KYcvc5YcqggbQ6DPkwwsX5txY?table=block&id=292282b8-1a87-46f6-9de4-1f8b021fbb2d&cache=v2)
In logistic regression the response variable is transformed to fit the problem into a linear model framework. Write down the equation for this transformation.
Deviance is what’s not explained by the model.
![notion image](https://www.notion.so/image/https%3A%2F%2Ffile.notion.so%2Ff%2Ff%2Fe1c5ba0e-562a-49b0-8505-ce5e38fc061b%2F629b8927-594c-4a0c-bb65-67b6e4bc4589%2FUntitled.png%3Ftable%3Dblock%26id%3D824b25a5-87e6-411c-a5e4-500120a69c35%26spaceId%3De1c5ba0e-562a-49b0-8505-ce5e38fc061b%26expirationTimestamp%3D1725624000000%26signature%3Dbk4uBiPcXnsCJ5oqVZn7wxAOkCpF4rlN6yQfmnI_5Ow?table=block&id=824b25a5-87e6-411c-a5e4-500120a69c35&cache=v2)
- Author:Jason Siu
- URL:https://jason-siu.com/article%2Ff1db0d75-c02d-47b5-9060-8e5e9851e87f
- Copyright:All articles in this blog, except for special statements, adopt BY-NC-SA agreement. Please indicate the source!
Relate Posts