Azure Big Data Architect

Job Description

The Azure Big Data Architect will be responsible for Enterprise data warehouse design, development, implementation, migration, maintenance and operation activities. The position is a key resource or various Enterprise data warehouse projects and building critical data marts, and data ingestion to Big Data platforms'

Responsibilities

• Participate in Team activities, Design discussions, Stand up meetings and planning Review with team.
• Perform data analysis, data profiling, data quality and data ingestion in various layers using big data queries, PySpark programs and UNIX shell scripts.
• Follow organizational coding standard documents, create mappings, sessions and workflows outlined within mapping specification documents.
• Perform Gap and impact analysis of ETL and IOP jobs for new requirement and enhancements.
• Create mockup data, perform unit testing and capture the result sets against the jobs developed in lower environment.
• Updating the production support run book.
• Create and update design documents, provide detail description about workflows after every production release.
• Continuously monitor the production data loads, fix the issues, update the tracker document with the issues, Identify the performance issues.
• Performance tuning long running ETL/ELT jobs by creating partitions, enabling full load and other standard approaches.
• Perform Quality assurance check, Reconciliation post data loads and communicate to vendor for receiving fixed data.
• Participate in ETL/ELT code review and design re-usable frameworks.
• Create re-usable framework for audit control to capture reconciliation, mapping parameters and variables, serves as single point of reference for workflows.
• Participate in meetings to upgrade the functional and technical expertise.

Technical Skills and Experience

• Expertise implementing complex ETL/ELT logic.
• Develop and enforce strong reconciliation process.
• Accountable for ETL/ELT design documentation.
• Good knowledge of Big Data, Hadoop, Hive, data security and dimensional model design.
• Basic knowledge of UNIX/LINUX shell scripting.
• Utilize ETL/ELT standards and practices towards establishing and following centralized metadata repository.
• Effective communication, presentation, & organizational skills.
• 6+ years of experience with Big Data, Hadoop on Data Warehousing or Data Integration projects,and familiarity with PHI and PII data.
• Analysis, Design, development, support and Enhancements of ETL/ELT in data warehouse environment with Azure Technologies.
• Strong development experience in creating powershell scripts, PySpark programs, HDFS commands and HDFS file formats
• Writing Hadoop/Hive scripts for gathering stats on table post data loads.
• Writing complex SQL queries and performed tuning based on the Hadoop/Hive explain plan results.
• Proven ability to write high quality code.
• 6+ years of experience building data sets
• Familiar with Project Management methodologies (Agile)
• Ability to establish priorities & follow through on projects, paying close attention to detail with minimal supervision.

Certifications Required

• Microsoft Azure DP-200: Implementing an Azure Data Solution
• Microsoft AzureDP-201: Designing an Azure Data Solution

Education

• Relevant Bachelor of Science or Bachelor of Arts degree or similar experience

‍