Member-only story
8 things one needs to know to understand data science. Described in anatomy terms
Skills block. Session 5
In this article I will breakdown the key components of backend infrastructure for a modern scalable data science project
Above image shows first 4 components any data science project will feature starting with
Prototyping Layer (below 4)
1.Data warehouse
Data warehouse is responsible for storing both historical and real-time data, which is used for analysis and decision-making. Similar to how the brain uses neurons to send and process information, a data warehouse uses various technologies such as databases, ETL processes, and data pipelines to collect, store, and process data. Think of it as a brain in human anatomy
Apps: Snowflake, Databricks
2. Compute resources
Provide the computing power and processing capacity for digital operations and include physical servers, virtual machines, and containers, which are responsible for executing code, processing data, and running applications. These resources work together to perform computational tasks and deliver the necessary performance to meet the demands of the application or service. Think of it as muscles & body
