Actin depolymerizing factor (ADF) is an actin binding protein ubiquitous in animal and plant cells, which play an important role in plant growth and development, and resistance to biotic and abiotic stress. The research of plant ADF family has been restricted to Arabidopsis thaliana and some herb crops, but no woody cash crops have been reported. All members of the Cajanus cajan ADF (CcADF) family were identified from the pigeon pea genome and distributed among the four subfamilies by phylogenetic analysis. CcADFs were relatively conservative in gene structure evolution, protein structure and functional expression, and different CcADFs showed specific expression patterns under different treatments. The expression characteristics of several key CcADFs were revealed by analyzing the stress response pattern of CcADFs and the time series RNA-seq of aluminum stress. Among them, CcADF9 in the first subgroup specifically responded to aluminum stress in the roots, CcADF3 in the second subgroup intensively responded to fungal infection in the leaves, and CcADF2 in the fourth subgroup positively responded to various stress treatments in different tissues. This study extended for the first time the relationship between plant ADF family and aluminum tolerance, as well as the understanding of CcADF family in woody crops.