Rafael Arruda

• Designed and developed scalable data pipelines using Azure Data Factory (ADF) for data ingestion and orchestration.

• Implemented data storage solutions using Azure Data Lake Storage Gen2 (ADLS Gen2) to support analytical workloads.

• Developed Azure Functions to automate custom processing, integrations, and event-driven workflows.

• Built and maintained ETL/ELT processes in Snowflake, supporting analytical data models and business reporting.

• Developed parameterized pipelines with incremental loading, monitoring, error handling, and recovery strategies.

• Optimized data processing performance and contributed to the design of scalable and secure Azure-based data architectures.

• Collaborated with cross-functional teams to deliver reliable cloud data solutions following engineering best practices.

Project: Data Engineer - Databricks (02/26 - 05/26)
Client: Berners Ads
I led the migration of a legacy process from Talend + MySQL to a scalable architecture on AWS + Databricks.
Integrating sources such as Meta, Google, Spotify, and TikTok.
To eliminate the risk of timeouts in centralized processing, I designed an event-driven approach using a Lambda Orchestrator + SQS (one client per event/execution), ensuring greater parallelism and resilience.
After ingestion into S3, I implemented the Lakehouse in Databricks using Bronze/Silver/Gold layers.
Although the entire project could have been built solely on Databricks, we split the architecture between AWS (infrastructure layer) and Databricks (analytical layer) as a cost-optimization strategy.
Project: Data Engineer - Snowflake (06/25 - 01/26)
Client: NTT Data
• Processed XML, JSON, and CSV customer data delivered via Amazon S3, leveraging Snowpark for Python and SQL-based transformations across bronze, silver, and gold layers.
• Developed Stored Procedures combining SQL and Python (Snowpark) for parsing, validation, incremental processing, and semi-structured data transformation.
• Designed and implemented event-driven data pipelines on Snowflake integrating AWS services, Snowpipe Streaming, and Dynamic Tables for near real-time data processing.
• Built ingestion workflows using AWS serverless components including EventBridge, Lambda, SNS, and S3 to automate data delivery into Snowflake.
• Designed transformation models using dbt to standardize and curate analytical datasets consumed by BI platforms.
• Automated data orchestration and deployment workflows using Snowflake Tasks and CI/CD pipelines with GitLab CI/CD and GitHub Actions.
• Delivered curated datasets for analytics and reporting through Power BI and Metabase.
Project Migration of BI Project to Azure Databricks (03/25 - 05/25)
Client: NTT Data

Migrated a BI project from Microsoft environment to Azure Databricks, implementing bronze, silver, and gold layers, and leveraging Microsoft Fabric and Power BI.
• Developed Python scripts to extract and load data into Azure Blob Storage
• Processed data with PySpark and SQL to create the Lakehouse layers
• Created Fabric views for Power BI consumption
• Built and validated the data model used for reporting.
• Lakehouse in Databricks, Views in Fabric and Dashboards in Power BI.

Project Automation of R Models using Pentaho Data Integration (02/24 - 02/25)
Client: Bank
Migrated statistical models developed in R to an automated workflow using Pentaho Data Integration (PDI).
• Prepared and transformed data in PDI, orchestrating R model execution
• Stored results in an Oracle Data Warehouse.
• Validated data quality and documented the process
• Improved scalability and efficiency of the data pipeline

Project 1 - POS Migration to Google Cloud (08/2023 - 01/2024)
Client: Alcampo

Migrated POS data from on-premises to Google Cloud. Participated in architecture definition and customer validation, followed by implementation.
• Developed Python Cloud Functions to extract data and load it into Google Cloud Storage
• Built pipelines with Dataform(dbt from Google) and SQL to prepare data for BigQuery
• Used Terraform to provision and manage GCP resources (Cloud Storage, BigQuery datasets, and service accounts)
• Delivered datasets consumed by Looker, enabling a 100% Google Cloud analytics solution

• Responsible for designing and maintaining enterprise data pipelines and data warehouse processes.
• Developed ETL and process automation workflows using Pentaho Data Integration, integrating data from multiple sources including relational databases, IoT platforms, Kafka, and Cloudera environments.
• Built and maintained Data Warehouse solutions on PostgreSQL hosted in Oracle Cloud Infrastructure (OCI).
• Applied dbt for data transformation, modeling, and standardization within analytical layers of the Data Warehouse.
• Used GitLab for version control and Jenkins to orchestrate end-to-end ETL pipelines and automate CI/CD deployment workflows.
• Developed analytical datasets and business dashboards using Metabase for reporting and data analysis.

• Responsible for maintenance and continuous improvements of the
Data.
• Led the migration of the on-premises Data Warehouse to Azure.
• Implemented Azure Data Lake as the central Data Lake.
• Used Azure Data Factory for end-to-end data ingestion.
• Built a Lakehouse architecture using Databricks.
• Connected Power BI to the Lakehouse for analytics and reporting.

• Using Google Storage for create the Data Lake in Google.
• Using Pentaho or python for ingest of data to Big Query with DW.
• Create of Dashboards in Power BI, and Looker Data Studio.
• Create and defined data model.
• Support for new members of team.

Using S3 for storage.
Creation of Crawlers to be able to transform S3 files into tables in Athena.
Data extraction, processing and processing with Glue job using Python and Spark scripts.
Development scrips in Pyspark for data prep and after create the Data Frames.
After Glue Jobs are processed, the data will be in AWS Athena to be able to manipulate the data.
Making data available to Tableau through Connecting to Athena.
Scheduling jobs with AWS Glue.

Using ODI to integrate public hospital data with SPMS.
Integration from STG to Data Wharehouse.
One of those responsible for developing reports and dashboards in Power BI.
Trainning of Power BI for users of business and I.T.
Creation and maintenance of Tabular models in SSAS, so Power BI can connect.
Field and measurement creation using DAX language.

• Participation of meetings with engineers and managers to understand the
demand and be able to create dashboards in Tableau.
• Tableau Server Administrator provided users access permission to create the
refresh schedule for all data sources in the panels.
• Creating Dashboards and Reports in Power BI
• Development and maintenance of ETL packages by
the Pentaho tool Integration .
• Creation of operational reports in Reporting Services.
• Tableau training for users.

• Creation of load monitoring reports, based on data cubes, using Excel and
Microsoft Reporting Services - SSRS.
• M ation of tables and data sources.
• Query Development (T-SQL) for data manipulation in SQL Server 2012 and
Oracle.
• Handling data in Integration Services, and changing the packages that are
generated every day, to suit all areas of the client.
• Creating and updating cubes, where they will be s available the information for
the business area.
• One of those responsible for the MIS area, where all the reports from different
areas are attended . of the customer.
• All reports are made based on the customers ERP.
• Responsible for all BI and transportation reports, understanding the users
demand until the delivery of the same, attending meetings to define a scope and
deadlines for deliveries of the projects.
• Creation ofReports and Dashboards using the Tableau and Power BI tools.