The emerging digital world has recently utilized the massive power of the emerging Internet of Things (IoT) technology that fuels the growth of many intelligent applications. The Internet of Battlefield Things (IoBT) greatly enables critical information dissemination and efficient war strategy planning with situational awareness. The lightweight Routing Protocol for Low-Power and Lossy Networks (RPL) is critical for successful IoT application deployment. RPL has low-security features that are insufficient to protect the IoBT environment due to device heterogeneity and open wireless device-to-device communication. Hence, it is crucial to provide strong security to RPL-IoBT against multiple attacks and enhance its performance. This work proposes IoBTSec-RPL, a hybrid Deep Learning (DL)-based multi-attack detection model, to overcome the attacks. The proposed IoBTSec-RPL learns prominent routing attacks and efficiently classifies the attackers. It includes four steps: data collection and preprocessing, feature selection, data augmentation, and attack detection and classification. Initially, the proposed model employs min-max normalization and missing value imputation to preprocess network packets. Secondly, the enhanced pelican optimization algorithm selects the most suitable features for attack detection through an efficient ranking method. Thirdly, data augmentation utilizes an auxiliary classifier gated adversarial network to alleviate the class imbalance concerns over the multiple attack classes. Finally, the proposed approach successfully detects and classifies the attacks using a hybrid DL model that combines LongShort-Term Memory (LSTM) and Deep Belief Network (DBN). The performance results reveal that the IoBTSec-RPL accurately recognizes the multiple RPL attacks in IoT and accomplished 98.93% recall. It also achieved improved accuracy of 2.16%, 5.73%, and 6.06% than the LGBM, LSTM, and DBN for 200K traffic samples.