June 27, 2025
Day 24 - Model Training on Complete Data
What I Learned
The most important thing today of the project is that I was able to process the entire 8 million data. The information had been cleaned and checked well and handed over to the corresponding teams who further analyzed and modelled them. The experience reaffirmed the conviction about the importance of good data processing and teamwork in the sense of large projects. After I was done preparing the data, I then swung into another critical stage, and this was bellowing baseline modeling. This was achieved through employing some generic machine learning algorithms to provide a yardstick on performance on future models. Meanwhile, my other developer Brice began work on the task of developing the utilization of the XGBoost model (the more advanced version of the gradient boosting model that has become famous because of high accuracy and good performance with large datasets). The common development will allow us to compare the different modeling approaches and think over what algorithm suits best to our problem. Overall, this step increased my preparation and information on the process of data pipeline working on it, and advancing knowledge on how to make a model and work with teams in a data science environment.
Blockers
Despite the smooth progress with data completion and baseline modeling, I encountered difficulties specifically with Logistic Regression. While this algorithm is often considered straightforward and interpretable, it posed challenges during implementation on this large and complex dataset. The issues I faced included convergence problems during model training and poor predictive performance compared to expectations.
Reflection
By looking back at this project phase, I realize how much patience and flexibility are required to solve actual data science tasks. It was an accomplishment to have the presentations of data after a long number of preparations and created a good background in modeling. Nonetheless, the problems we had using the Logistic Regression made me remember that even a single algorithm is not always optimal as it may have its advantages or disadvantages according to the circumstances and properties of the data. This taught me not to shy away in trying other methods, like gradient boosting, that could be accurate and scalable on more complicated problems. It also showed the importance of teamwork, since my teammate Brice and I could share the workload and ideas, it was possible to parallelize our joint activities and learn more about one another problems. As a future prospect, I want to know more about the advanced machine learning algorithms and streamline the generation of troubleshooting skills. Comprehensively, I have gained a lot of knowledge intrinsically that made me more technically competent and capable of working in a data-driven culture with colleagues.