LightGBM

LightGBM
Original author(s)	Guolin Ke / Microsoft Research
Developer(s)	Microsoft and LightGBM contributors
Initial release	2016; 8 years ago
Stable release	v4.3.0 / January 15, 2024; 11 months ago
Repository	github.com/microsoft/LightGBM
Written in	C++, Python, R, C
Operating system	Windows, macOS, Linux
Type	Machine learning, gradient boosting framework
License	MIT License
Website	lightgbm.readthedocs.io

LightGBM, short for Light Gradient-Boosting Machine, is a free and open-source distributed gradient-boosting framework for machine learning, originally developed by Microsoft.^[4]^[5] It is based on decision tree algorithms and used for ranking, classification and other machine learning tasks. The development focus is on performance and scalability.

Overview

The LightGBM framework supports different algorithms including GBT, GBDT, GBRT, GBM, MART^[6]^[7] and RF.^[8] LightGBM has many of XGBoost's advantages, including sparse optimization, parallel training, multiple loss functions, regularization, bagging, and early stopping. A major difference between the two lies in the construction of trees. LightGBM does not grow a tree level-wise — row by row — as most other implementations do.^[9] Instead it grows trees leaf-wise. It will choose the leaf with max delta loss to grow. ^[10] Besides, LightGBM does not use the widely used sorted-based decision tree learning algorithm, which searches the best split point on sorted feature values,^[11] as XGBoost or other implementations do. Instead, LightGBM implements a highly optimized histogram-based decision tree learning algorithm, which yields great advantages on both efficiency and memory consumption.^[12] The LightGBM algorithm utilizes two novel techniques called Gradient-Based One-Side Sampling (GOSS) and Exclusive Feature Bundling (EFB) which allow the algorithm to run faster while maintaining a high level of accuracy.^[13]

LightGBM works on Linux, Windows, and macOS and supports C++, Python,^[14] R, and C#.^[15] The source code is licensed under MIT License and available on GitHub.^[16]

Gradient-based one-side sampling

Gradient-based one-side sampling (GOSS) is a method that leverages the fact that there is no native weight for data instance in GBDT. Since data instances with different gradients play different roles in the computation of information gain, the instances with larger gradients will contribute more to the information gain. So to retain the accuracy of the information, GOSS keeps the instances with large gradients and randomly drops the instances with small gradients.^[13]

Exclusive feature bundling

Exclusive feature bundling (EFB) is a near-lossless method to reduce the number of effective features. In a sparse feature space many features are nearly exclusive, implying they rarely take nonzero values simultaneously. One-hot encoded features are a perfect example of exclusive features. EFB bundles these features, reducing dimensionality to improve efficiency while maintaining a high level of accuracy. The bundle of exclusive features into a single feature is called an exclusive feature bundle.^[13]

References

^ "Guolin Ke". GitHub.
^ "microsoft/LightGBM". GitHub. 7 July 2022.
^ "Releases · microsoft/LightGBM". GitHub.
^ Brownlee, Jason (March 31, 2020). "Gradient Boosting with Scikit-Learn, XGBoost, LightGBM, and CatBoost".
^ Kopitar, Leon; Kocbek, Primoz; Cilar, Leona; Sheikh, Aziz; Stiglic, Gregor (July 20, 2020). "Early detection of type 2 diabetes mellitus using machine learning-based prediction models". Scientific Reports. 10 (1): 11981. Bibcode:2020NatSR..1011981K. doi:10.1038/s41598-020-68771-z. PMC 7371679. PMID 32686721 – via www.nature.com.
^ "Understanding LightGBM Parameters (and How to Tune Them)". neptune.ai. May 6, 2020.
^ "An Overview of LightGBM". avanwyk. May 16, 2018.
^ "Parameters — LightGBM 3.0.0.99 documentation". lightgbm.readthedocs.io.
^ The Gradient Boosters IV: LightGBM – Deep & Shallow
^ "Features". LightGBM Official Documentation. Nov 3, 2024.
^ Manish, Mehta; Rakesh, Agrawal; Jorma, Rissanen (Nov 24, 2020). "SLIQ: A fast scalable classifier for data mining". International Conference on Extending Database Technology: 18–32. CiteSeerX 10.1.1.89.7734.
^ "Features — LightGBM 3.1.0.99 documentation". lightgbm.readthedocs.io.
^ ^a ^b ^c Ke, Guolin; Meng, Qi; Finley, Thomas; Wang, Taifeng; Chen, Wei; Ma, Weidong; Ye, Qiwei; Liu, Tie-Yan (2017). "LightGBM: A Highly Efficient Gradient Boosting Decision Tree". Advances in Neural Information Processing Systems. 30.
^ "lightgbm: LightGBM Python Package". 7 July 2022 – via PyPI.
^ "Microsoft.ML.Trainers.LightGbm Namespace". docs.microsoft.com.
^ "microsoft/LightGBM". October 6, 2020 – via GitHub.

External links

[1] "Guolin Ke". GitHub.

[2] "microsoft/LightGBM". GitHub. 7 July 2022.

[3] "Releases · microsoft/LightGBM". GitHub.

[4] Brownlee, Jason (March 31, 2020). "Gradient Boosting with Scikit-Learn, XGBoost, LightGBM, and CatBoost".

[5] Kopitar, Leon; Kocbek, Primoz; Cilar, Leona; Sheikh, Aziz; Stiglic, Gregor (July 20, 2020). "Early detection of type 2 diabetes mellitus using machine learning-based prediction models". Scientific Reports. 10 (1): 11981. Bibcode:2020NatSR..1011981K. doi:10.1038/s41598-020-68771-z. PMC 7371679. PMID 32686721 – via www.nature.com.

[6] "Understanding LightGBM Parameters (and How to Tune Them)". neptune.ai. May 6, 2020.

[7] "An Overview of LightGBM". avanwyk. May 16, 2018.

[8] "Parameters — LightGBM 3.0.0.99 documentation". lightgbm.readthedocs.io.

[9] The Gradient Boosters IV: LightGBM – Deep & Shallow

[10] "Features". LightGBM Official Documentation. Nov 3, 2024.

[11] Manish, Mehta; Rakesh, Agrawal; Jorma, Rissanen (Nov 24, 2020). "SLIQ: A fast scalable classifier for data mining". International Conference on Extending Database Technology: 18–32. CiteSeerX 10.1.1.89.7734.

[12] "Features — LightGBM 3.1.0.99 documentation". lightgbm.readthedocs.io.

[:0-13] Ke, Guolin; Meng, Qi; Finley, Thomas; Wang, Taifeng; Chen, Wei; Ma, Weidong; Ye, Qiwei; Liu, Tie-Yan (2017). "LightGBM: A Highly Efficient Gradient Boosting Decision Tree". Advances in Neural Information Processing Systems. 30.

[14] "lightgbm: LightGBM Python Package". 7 July 2022 – via PyPI.

[15] "Microsoft.ML.Trainers.LightGbm Namespace". docs.microsoft.com.

[16] "microsoft/LightGBM". October 6, 2020 – via GitHub.

[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]

[11]

[12]

[13]

[14]

[15]

[16]

LightGBM

Overview

Gradient-based one-side sampling

Exclusive feature bundling

See also

References

Further reading

External links

Read other articles:

Bernard Goumou

Ekonomi neoklasik

قرية أندوفر (نيويورك)

قرية كانيستو (نيويورك)

لغات أوروبا

Keajaiban Cinta

Nasionalisme Jepang

Joko Anwar

المسجد الأقصى

Chittagong-10

Ferdinando III d'Asburgo

Hernán Salgado Pesantes

Mozambique at the 2020 Summer Paralympics

Kualuh Hilir, Labuhanbatu Utara

Ibéromaurusien

Cratonavis

Heighington railway station

Orders, decorations, and medals of South Korea

Battle of Las Babias

新罕布什尔州