Clustering plays a vital role in Wireless Sensor Networks (WSNs) as it enables energy-efficient data collection and transmission. As a result, several Clustering Routing Algorithms (CRAs) were suggested over the years. Amongst, the Sine Cosine Algorithm with Levy Mutation (SCA-Levy)-based CRA achieved a good balance between energy usage and delay in multi-hop WSNs. This algorithm dynamically chose the best Cluster Head (CH) nodes according to the Residual Energy (RE) and intra-cluster distance. In addition, the optimal Relay Nodes (RNs) were decided based on the distance from CHs to the Base Station (BS) and the RE of CHs to circumvent long-range transmission. However, the RN selection process must satisfy constraints on the coverage space, number of nodes and BS position. As the coverage space or the number of nodes increases, the network lifespan decreases. To satisfy these constraints on the RN selection process in multi-hop WSNs, this article proposes a new Relay selection with an optimized SCA-Levy (RSCA-Levy) algorithm. First, the network is clustered, and CH nodes for all clusters are selected by the SCA-Levy scheme. After that, the RN selection challenge is represented as a Markov Decision Process (MDP) and resolved using the Deep Q-Learning (DQL) algorithm. This DQL employs a decentralized scheme with a rectified update function. Using this algorithm, all clusters train their Q-table and elect optimal RNs based on factors like RE, node density, distance to the BS, and coverage space. Furthermore, it transfers data from the Source (S) to the BS via optimal CHs and RNs. The simulation results demonstrate that the RSCA-Levy algorithm achieves 3.76×105 packets transferred to the BS, 59J total relaying energy, 5.4% network energy utilization, 35 dead nodes, and 98ms End-to-End (E2E) delay in 1000 rounds with 100 nodes, compared to the existing algorithms such as SCA-Levy, Shortest Path Selection for RN (SPSRN), Enhanced Energy Proficient Clustering (EEPC), Analytic Hierarchy Process (AHP) with Technique for Order Preference by Similarity to Ideal Solution (TOPSIS), and Matching Learning-based Relay Selection (MLRS).