Data scientist

ETL Migration to Spark and Power BI Dashboards

Project Overview

This project focuses on the migration of our current ETL processes to Apache Spark, a powerful big data processing framework. It also includes the development of data dashboards utilizing Power BI for improved data visualization and analysis. The goal is to modernize our data processing and reporting infrastructure, making it more efficient and capable of handling large volumes of data.

Project Objectives

ETL Migration: Transfer existing ETL processes to Apache Spark, ensuring faster and scalable data processing.

Data Transformation: Implement data transformation logic within Spark to enhance data quality and consistency.

Power BI Dashboards: Design and develop interactive dashboards in Power BI for data visualization, reporting, and analysis.

Automation: Implement automated ETL pipelines and scheduled updates for dashboards.

Tools and Frameworks

Apache Spark, Apache Airflow, Power BI, Big Data, Data Storage, Jira, Git, MongoDB, AWS, Athena, S3

Informations

Exigences

Compétences:

Intelligence artificielle Data science TENSORFLOW PYTHON MACHINE LEARNING PYTORCH

Statistiques