Accurate electric vehicle (EV) load forecasting is crucial for efficient grid operations and demand-side management, yet it is challenging in data-scarce scenarios. Transfer learning (TL) offers a solution by transferring knowledge from data-rich to data-limited scenarios. However, when the knowledge domain exhibits highly diverse behaviors, applying TL alone could introduce large biases, reducing accuracy and limiting its effectiveness. To address this problem, this study proposes a hybrid deep learning-based framework that integrates TL and K-means clustering. The proposed approach consists of two phases. In the source domain phase, a deep-learning-based model is trained using the full dataset and then fine-tuned using clustered user behaviors. In the target domain phase with limited data, TL is applied to transfer knowledge from the source-domain finetuned cluster models. For validation, the developed prediction method has been tested using real-world datasets and compared with two other cases: one with applying TL from the source-domain base model trained from full dataset, and one without applying TL. Results show the hybrid method improves forecasting accuracy, reducing the normalized root mean squared error by 3.99 % and 8.22 %, respectively. This study establishes a structured approach for targeted knowledge transfer, enhancing prediction accuracy in data-scarce settings. The framework is scalable and adaptable to other energy forecasting applications, supporting sustainable and resilient energy management.