The ID3 Algorithm

Length: 1344 words (3.8 double-spaced pages)
Rating: Excellent
Open Document
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Text Preview

More ↓

Continue reading...

Open Document

The ID3 Algorithm


This paper details the ID3 classification algorithm. Very simply, ID3 builds a decision tree from a fixed set of examples. The resulting tree is used to classify future samples. The example has several attributes and belongs to a class (like yes or no). The leaf nodes of the decision tree contain the class name whereas a non-leaf node is a decision node. The decision node is an attribute test with each branch (to another decision tree) being a possible value of the attribute. ID3 uses information gain to help it decide which attribute goes into a decision node. The advantage of learning a decision tree is that a program, rather than a knowledge engineer, elicits knowledge from an expert.

J. Ross Quinlan originally developed ID3 at the University of Sydney. He first presented ID3 in 1975 in a book, Machine Learning, vol. 1, no. 1. ID3 is based off the Concept Learning System (CLS) algorithm. The basic CLS algorithm over a set of training instances C:

Step 1: If all instances in C are positive, then create YES node and halt.

If all instances in C are negative, create a NO node and halt.

Otherwise select a feature, F with values v1, ..., vn and create a decision node.

Step 2: Partition the training instances in C into subsets C1, C2, ..., Cn according to the values of V.

Step 3: apply the algorithm recursively to each of the sets Ci.

Note, the trainer (the expert) decides which feature to select.

ID3 improves on CLS by adding a feature selection heuristic. ID3 searches through the attributes of the training instances and extracts the attribute that best separates the given examples. If the attribute perfectly classifies the training sets then ID3 stops; otherwise it recursively operates on the n (where n = number of possible values of an attribute) partitioned subsets to get their "best" attribute. The algorithm uses a greedy search, that is, it picks the best attribute and never looks back to reconsider earlier choices.

ID3 is a nonincremental algorithm, meaning it derives its classes from a fixed set of training instances. An incremental algorithm revises the current concept definition, if necessary, with a new sample. The classes created by ID3 are inductive, that is, given a small set of training instances, the specific classes created by ID3 are expected to work for all future instances. The distribution of the unknowns must be the same as the test cases.

How to Cite this Page

MLA Citation:
"The ID3 Algorithm." 26 May 2017
Title Length Color Rating  
Algorithm Research - Quicksort Essay - An algorithm, according to the Random House Unabridged Dictionary, is a set of rules for solving a problem in a finite number of steps. One of the fundamental problems of computer science is sorting a set of items. The solutions to these problems are known as sorting algorithms and rather ironically, “the process of applying an algorithm to an input to obtain an output is called a computation” []. The quest to develop the most memory efficient and the fastest sorting algorithm has become one of the great mathematical challenges of the last half century, resulting in many tried and tested algorithms available to the individual who needs to sort a lis...   [tags: Computer Science Algorithm] 1117 words
(3.2 pages)
Strong Essays [preview]
Study on Trust Evaluation Algorithms of Domains in Grid Environment Essay - In order to achieve a reasonable evaluation of direct trust, this paper proposes a trust evaluation algorithm based on the domain, using the technique of constructing a hierarchical tree of trust evaluation subjectively. The algorithm adopts the rules of series and parallel operations in the D-S theory, acquires the results of the recommended trust problem of a single path by quadrature methods and implements the integration of multiple paths by the weighted algorithm which takes the cooperative roles and industry roles as factors....   [tags: algorithm, manufacturing industry, ]
:: 12 Works Cited
1633 words
(4.7 pages)
Powerful Essays [preview]
Essay on A New Algorithm for Age Group Recognition from Frontal Face Image - In this paper, a new algorithm for age group recognition from frontal face image is presented. The algorithm classifies subjects into four different age categories in four key stages: Pre-processing, Facial feature extraction by a new projection method, Face feature analysis, and Age classification. In order to apply the algorithm to the problem, a face image database focusing on people’s age information is required. Because there are no such these databases, we created a database for this purpose, which is called Iranian Face Database (IFDB)....   [tags: algorithms, age group recognition, frontal face im] 2507 words
(7.2 pages)
Powerful Essays [preview]
Research Study- Improved Algorithms for Yield Driven Clock Skew Scheduling in the Presence of Process Variations - Abstract Traditional yield driven clock skew scheduling in the presence of process variations can be formulated as a sequence of minimum ratio cycle problems, and hence can be solved efficiently by algorithms such as Lawler's and Howard's algorithms. However, the assumption of Gaussian distributions of critical path delays has been made in this formulation, which becomes inapplicable for next generation nanometer technologies. Recently, a generalization of the formulation for non-Gaussian distributions was proposed, and a modification of Lawler's algorithm was developed for solving this generalized problem....   [tags: algorithms]
:: 6 Works Cited
1776 words
(5.1 pages)
Powerful Essays [preview]
Proposal of a New Sorting Algorithm Essay examples - Abstract—Computational problems have significance from the early civilizations. These problems and solutions are used for the study of universe. Numbers and symbols have been used for different fields e.g. mathematics, statistics. After the emergence of computers the number and objects needs to be arranged in a particular order i.e. ascending and descending orders. The ordering of these numbers is generally referred to as sorting. Sorting gained a lot of importance in computer sciences and its applications are in file systems etc....   [tags: computers]
:: 14 Works Cited
1532 words
(4.4 pages)
Powerful Essays [preview]
Essay on The Development of Algorithm for Data Stream - The past decade has seen a lot of research on various time series representations. Various researches have been carried out that focused on representations that are processed in batch mode and visualize each value with almost equal dependability. As the tremendous usage of mobile devices and real time sensors has released the necessity and importance for representations that can simultaneously be updated, and can estimate the time oriented data with reliability and proportional to its time period for extended analysis....   [tags: data, computational resources]
:: 30 Works Cited
939 words
(2.7 pages)
Better Essays [preview]
Essay about The AES Encryption Algorithm - The AES encryption algorithm is a block cipher that uses an encryption key and a several rounds of encryption A block cipher is an encryption algorithm that works on a single block of data at a time. In the case of standard AES encryption the block is 128 bits or 16 bytes in length. The term “rounds” refers to the way in which the encryption algorithm mixes the data re-encrypting into fourteen times depending on length of key. ENCRYPTION KEYS AES encryption uses a single key as a part of they encryption process....   [tags: block cipher]
:: 6 Works Cited
858 words
(2.5 pages)
Better Essays [preview]
Decision Tree Analysis and Genetic Algorithm Essay - Decision Tree Analysis In data mining, the decision tree analysis is used to determine the best choice from various possible options. Through this process, researchers and managers get an opportunity to evaluate the risks, benefits and inconsistencies associated with the decisions. The first step is structuring the problems or issues being faced by the organization as a tree. At the end of each branch, all the benefits are listed to help in evaluating the path with the most benefits. After the benefits have been determined, the next step involves assigning subjective probabilities to all the activities on the tree (Qu, Adam, Yasui, Ward, & Cazares, 2002)....   [tags: Data Mining]
:: 2 Works Cited
577 words
(1.6 pages)
Good Essays [preview]
Multicast Algorithm Essay - This paper proposes an efficient and scalable multicast algorithm that accommodates dynamic groups. Our protocol relies on a shared tree architecture to deal with the problems of scalability and group dynamics. Our algorithm is based on the communication model developed by Bhat et al [2] that considers both network and node heterogeneity. Our algorithm uses a modified version Bhat et al. [3] heuristics for multicasting a message to the group. M I. INTRODUCTION any applications such as teleconferencing, distributed games and any collaborative multimedia application require an efficient group communication....   [tags: Group Communication Software Technology] 1702 words
(4.9 pages)
Strong Essays [preview]
Human Centric Computing: Algorithm Animation Essay - Human Centric Computing: Algorithm Animation To many computer science students, algorithms are evil incarnate. They are hard to learn, understand, and teach. So, is there any way to make this difficult (and rather important) aspect of computer science easier. I believe that if algorithm animations are not a step in the right direction then maybe we need to change the direction. Algorithm animation is just a small piece of the software visualization puzzle. Software visualization is "the visualization of computer programs and algorithms" (Price 4)....   [tags: Essays Papers]
:: 5 Works Cited
1147 words
(3.3 pages)
Strong Essays [preview]

Induction classes cannot be proven to work in every case since they may classify an infinite number of instances. Note that ID3 (or any inductive algorithm) may misclassify data.
Data Description

The sample data used by ID3 has certain requirements, which are:

* Attribute-value description - the same attributes must describe each example and have a fixed number of values.

* Predefined classes - an example's attributes must already be defined, that is, they are not learned by ID3.

* Discrete classes - classes must be sharply delineated. Continuous classes broken up into vague categories such as a metal being "hard, quite hard, flexible, soft, quite soft" are suspect.

* Sufficient examples - since inductive generalization is used (i.e. not provable) there must be enough test cases to distinguish valid patterns from chance occurrences.

Attribute Selection

How does ID3 decide which attribute is the best? A statistical property, called information gain, is used. Gain measures how well a given attribute separates training examples into targeted classes. The one with the highest information (information being the most useful for classification) is selected. In order to define gain, we first borrow an idea from information theory called entropy. Entropy measures the amount of information in an attribute.

Given a collection S of c outcomes

Entropy(S) = S -p(I) log2 p(I)

where p(I) is the proportion of S belonging to class I. S is over c. Log2 is log base 2.

Note that S is not an attribute but the entire sample set.
Example 1

If S is a collection of 14 examples with 9 YES and 5 NO examples then

Entropy(S) = - (9/14) Log2 (9/14) - (5/14) Log2 (5/14) = 0.940

Notice entropy is 0 if all members of S belong to the same class (the data is perfectly classified). The range of entropy is 0 ("perfectly classified") to 1 ("totally random").

Gain(S, A) is information gain of example set S on attribute A is defined as

Gain(S, A) = Entropy(S) - S ((|Sv| / |S|) * Entropy(Sv))


S is each value v of all possible values of attribute A

Sv = subset of S for which attribute A has value v

|Sv| = number of elements in Sv

|S| = number of elements in S
Example 2

Suppose S is a set of 14 examples in which one of the attributes is wind speed. The values of Wind can be Weak or Strong. The classification of these 14 examples are 9 YES and 5 NO. For attribute Wind, suppose there are 8 occurrences of Wind = Weak and 6 occurrences of Wind = Strong. For Wind = Weak, 6 of the examples are YES and 2 are NO. For Wind = Strong, 3 are YES and 3 are NO. Therefore


= 0.940 - (8/14)*0.811 - (6/14)*1.00

= 0.048

Entropy(Sweak) = - (6/8)*log2(6/8) - (2/8)*log2(2/8) = 0.811

Entropy(Sstrong) = - (3/6)*log2(3/6) - (3/6)*log2(3/6) = 1.00

For each attribute, the gain is calculated and the highest gain is used in the decision node.
Example of ID3

Suppose we want ID3 to decide whether the weather is amenable to playing baseball. Over the course of 2 weeks, data is collected to help ID3 build a decision tree (see table 1).

The target classification is "should we play baseball?" which can be yes or no.

The weather attributes are outlook, temperature, humidity, and wind speed. They can have the following values:

outlook = { sunny, overcast, rain }

temperature = {hot, mild, cool }

humidity = { high, normal }

wind = {weak, strong }

Examples of set S are:






Play ball
D1 Sunny Hot High Weak No
D2 Sunny Hot High Strong No
D3 Overcast Hot High Weak Yes
D4 Rain Mild High Weak Yes
D5 Rain Cool Normal Weak Yes
D6 Rain Cool Normal Strong No
D7 Overcast Cool Normal Strong Yes
D8 Sunny Mild High Weak No
D9 Sunny Cool Normal Weak Yes
D10 Rain Mild Normal Weak Yes
D11 Sunny Mild Normal Strong Yes
D12 Overcast Mild High Strong Yes
D13 Overcast Hot Normal Weak Yes
D14 Rain Mild High Strong No

Table 1

We need to find which attribute will be the root node in our decision tree. The gain is calculated for all four attributes:

Gain(S, Outlook) = 0.246

Gain(S, Temperature) = 0.029

Gain(S, Humidity) = 0.151

Gain(S, Wind) = 0.048 (calculated in example 2)

Outlook attribute has the highest gain, therefore it is used as the decision attribute in the root node.

Since Outlook has three possible values, the root node has three branches (sunny, overcast, rain). The next question is "what attribute should be tested at the Sunny branch node?" Since we=92ve used Outlook at the root, we only decide on the remaining three attributes: Humidity, Temperature, or Wind.

Ssunny = {D1, D2, D8, D9, D11} = 5 examples from table 1 with outlook = sunny

Gain(Ssunny, Humidity) = 0.970

Gain(Ssunny, Temperature) = 0.570

Gain(Ssunny, Wind) = 0.019

Humidity has the highest gain; therefore, it is used as the decision node. This process goes on until all data is classified perfectly or we run out of attributes.

The final decision = tree

The decision tree can also be expressed in rule format:

IF outlook = sunny AND humidity = high THEN playball = no

IF outlook = rain AND humidity = high THEN playball = no

IF outlook = rain AND wind = strong THEN playball = yes

IF outlook = overcast THEN playball = yes

IF outlook = rain AND wind = weak THEN playball = yes

ID3 has been incorporated in a number of commercial rule-induction packages. Some specific applications include medical diagnosis, credit risk assessment of loan applications, equipment malfunctions by their cause, classification of soybean diseases, and web search classification.

The discussion and examples given show that ID3 is easy to use. Its primary use is replacing the expert who would normally build a classification tree by hand. As industry has shown, ID3 has been effective.

Return to