Data engineering in the cloud
Everyday challenges facing data engineers are similar to the ones facing data scientists. Processing data in its various forms is the center of attention for both specializations. In the data engineering context, however, we focus more on production-grade data pipelines, ETL/ELT workflows, and data platforms that need to be robust, reliable, and efficient.
When it comes to the cloud, Python has proven itself to be good enough to incentivize cloud platform providers to use it for implementing and controlling their services. If we look at the biggest players - namely Amazon Web Services (AWS), Google Cloud Platform (GCP), and Microsoft Azure - they all accommodate Python users in their solutions to a number of problems.
Cloud platforms support Python across a much broader range of data engineering workloads than serverless functions alone. Engineers can use it to trigger event-driven pipelines with services such as AWS Lambda, Google Cloud Functions, and Azure Functions, but also to develop and run data transformations, notebooks, distributed processing jobs, and production workflows within managed data platforms.
Python is now deeply integrated into environments such as Databricks, Microsoft Fabric, and Snowflake through Snowpark. These platforms allow data teams to build and execute Python-based workloads closer to the data, while relying on managed compute, collaborative notebooks, and cloud-native orchestration instead of maintaining the underlying infrastructure themselves.
Python SDKs also make it possible to manage cloud resources, retrieve data, and trigger jobs programmatically across AWS, Google Cloud, and Microsoft Azure, allowing cloud operations to become part of repeatable data workflows.
Parallel computing is, in turn, necessary for heavy-lifting ETL jobs on big data problems. Splitting the transformation workflows among many worker nodes is the only feasible way memory-wise (when the data can’t be kept in memory on one physical machine) and time-wise (when sequential processing would take too long) to accomplish the goal.
While Apache Spark is now the go-to technology for data parallelism, a Python wrapper for the Spark engine called PySpark is supported by AWS Elastic MapReduce (EMR), Dataproc for GCP, and HDInsight for Azure.
As far as controlling and managing the resources in the cloud is concerned, appropriate Application Programming Interfaces (APIs) are exposed for each platform.
APIs are especially useful for performing programmatic data retrieval or job triggering. Those developed by AWS, GCP, and Azure are conveniently wrapped in Python SDKs: boto, google_cloud_*, and azure-sdk-for-python, which makes them easily integrable within Python applications.
Python is therefore widely available across all cloud computing platforms. But the language is also a useful tool for performing a data engineer’s job, which is to design and maintain data pipelines that retrieve, transform, and deliver data for analytics, reporting, and AI.
Designing and maintaining these pipelines requires more than choosing the right programming language. Modern data engineering services combine cloud architecture, orchestration, data quality, and scalable pipeline design to ensure reliable data delivery across the organization.
Focus on data ingestion with Python
Business data may come from various sources of different natures, including databases (both SQL and noSQL), flat files (e.g. CSVs), other files used by companies (e.g. spreadsheets), external systems, APIs, and web documents.
The popularity of Python as a programming language results in an abundance of libraries and modules, including those used for accessing the data, for example SQLAlchemy for some SQL databases, Scrapy, Beautiful Soup, or Requests for data with web origins, among others.
One particularly interesting library is Pandas. It enables reading data into “DataFrames” from a variety of different formats, including CSVs, TSVs, JSON, XML, HTML, LaTeX, SQL, Microsoft, open spreadsheets, and several other binary formats that are results of different business systems exports.
The library also supports column-oriented formats including Apache Parquet, which enables optimizing querying that data later on with tools such as AWS Athena.
Pandas is based on other scientific and calculationally optimized packages, offering a rich programming interface with a huge panel of functions necessary to process and transform data in reliably and efficiently.
AWS Labs maintains the aws-data-wrangler library called “Pandas on AWS” to facilitate well-known DataFrame operations on AWS. The package can be, for instance, used as a Layer for Lambda Functions, thus rendering serverless functions deployment much easier.
Parallel computing with PySpark
Apache Spark is an open-source engine for processing huge volumes of data that leverages the parallel computing principle in a highly efficient and fault-tolerant way. While originally implemented in Scala and natively supporting this language, it has a widely used interface in Python: PySpark.
PySpark supports most of Spark’s features, such as Spark SQL, DataFrame, Streaming, MLlib (Machine Learning), and Spark Core. This makes developing large-scale data transformation pipelines extremely straightforward.
All of the cloud computing platforms I mentioned above support PySpark: Elastic MapReduce (EMR), Dataproc, and HDInsight for AWS, GCP, and Azure, respectively. In addition, you can connect a Jupyter Notebook to facilitate the development of the distributed processing Python code, for example with natively supported EMR Notebooks in AWS.
PySpark is therefore a powerful tool for transforming and aggregating huge volumes of data, making it ready for consumption by end users, such as business analysts, or by further components, for instance by involving machine learning.
Job scheduling with Apache Airflow
The existence of popular and well-regarded Python-based tools on on-premise systems motivates platform cloud providers to commercialize them in the form of “managed” services that are, as a result, easier to set up and operate.
This is, among others, true for Amazon’s Managed Workflows for Apache Airflow, which was launched in 2020 and facilitates using Airflow in some of the AWS zones (nine at the time of writing). Cloud Composer is a GCP alternative for a managed Airflow service.
Apache Airflow is written in Python and remains one of the most widely used platforms for authoring, scheduling, and monitoring data workflows. Its Python-based approach allows engineers to define pipelines programmatically, integrate them with other services, and extend orchestration logic with custom classes and libraries.
Managed offerings such as Amazon Managed Workflows for Apache Airflow and Google Cloud Composer can reduce the operational effort required to deploy and maintain Airflow in cloud environments.
Airflow is not the only option, however. Python-based orchestrators such as Prefect and Dagster offer different approaches to pipeline development, observability, asset management, and local testing. The right choice depends on factors such as the existing cloud ecosystem, deployment model, workflow complexity, team experience, and operational requirements rather than on a single universally best tool.
Regardless of the platform, Python remains valuable because it allows orchestration logic, data processing code, integrations, and automation to work within the same broader engineering ecosystem built around technologies such as dbt, Apache Iceberg, lakehouses, and AI-ready data platforms. Beyond writing transformation code, engineers increasingly use Python to automate testing, monitor data quality, orchestrate cloud services, and support machine learning pipelines.
Although the examples in this article focus on ETL, orchestration, and distributed processing, Python's role in data engineering extends across the entire data lifecycle. The summary below highlights where it fits into today's modern data platforms:
Final thoughts on Python for data engineering
Python has become one of the core languages of modern data engineering because it supports every stage of data processing - from ingestion and transformation to distributed computing, workflow orchestration, and cloud resource management.
Its versatility allows data engineers to work across different platforms and technologies without changing languages at every stage of the pipeline. However, Python is ultimately one part of a broader data ecosystem. Reliable data products also depend on the architecture, quality controls, monitoring, governance, and operational practices surrounding the code.
The next step is understanding how these components should work together to create a data platform that remains reliable as data volumes, use cases, and business expectations grow.
