Data Pipeline Optimization Using Fivetran and Databricks for Shipping and Logistics Analytics
Keywords:
Data Pipeline Optimization, ELT, Fivetran, Databricks, Logistics Analytics, Delta Lake, Apache Spark, Machine LearningAbstract
The exponential growth of data in shipping and logistics—fueled by IoT sensors, ERP systems,and CRM platforms has made traditional ETL pipelines inadequate due to latency, inflexibility,and poor scalability. This study presents an optimized, cloud-native data pipeline architectureintegrating Fivetran and Databricks
References
• Alsolbi, I., Hosseinnia Shavaki, F., Agarwal, R., Bharathy, G. K., Prakash, S., & Prasad, M. (2023). Big data optimisation and management in supply chain management: A systematic literature review. Artificial Intelligence Review, 56(10), 8257–8302. https://doi.org/10.1007/s10462-023-10505-4
• Brintrup, A., Kito, T., & Wakolbinger, T. (2020). A review of machine learning in supply chain management: Applications, challenges and opportunities. Computers & Industrial Engineering, 139, 105774. https://doi.org/10.1016/j.cie.2019.106764
