Decision trees are a method used in statistics, data mining, and machine learning to model the decisions and possible consequences, including chance event outcomes, resource costs, and utility. Here are some concise class notes on decision trees:
1. **Definition**: A decision tree is a flowchart-like tree structure where an internal node represents a feature (or attribute), a branch represents a decision rule, and each leaf node represents the outcome.
2. **Types of Decision Trees**:
– **Classification trees**: Used when the outcome is a discrete value. They classify a dataset.
– **Regression trees**: Used when the outcome is a continuous value, like predicting temperatures.
3. **Components**:
– **Root Node**: Represents the entire population or sample, further gets divided into two or more homogeneous sets.
– **Splitting**: Process of dividing a node into two or more sub-nodes based on certain conditions.
– **Decision Node**: Sub-node that splits into further sub-nodes.
– **Leaf/Terminal Node**: Nodes that do not split, representing a classification or decision.
4. **Algorithm**:
– Common algorithms include ID3, C4.5, CART (Classification and Regression Tree).
– These algorithms use different metrics (like Gini impurity, information gain, etc.) for choosing the split.
5. **Advantages**:
– Easy to understand and interpret.
– Requires little data preprocessing (no need for normalization, dummy variables).
– Can handle both numerical and categorical data.
6. **Disadvantages**:
– Prone to overfitting, especially with many features.
– Can be unstable because small variations in data might result in a completely different tree.
– Biased with imbalanced datasets.
7. **Applications**: Widely used in operations research, specifically in decision analysis, to help identify a strategy most likely to reach a goal. Also used in machine learning for classification and regression tasks.
8. **Important Considerations**:
– **Pruning**: Reducing the size of decision trees by removing parts that have little power to classify instances, to reduce overfitting.
– **Feature Selection**: Important in building an effective and efficient decision tree.