Job Description

Job Role: Cloud Integration | Azure Data Factory (ADF)

Location: Houston TX
 

Job Description

Azure Data Engineering focuses on designing, implementing, and maintaining data management systems within the Azure cloud platform. In today's data-driven world, organizations generate vast amounts of information that need efficient storage, processing, and analysis. As an Azure Data Engineer, you will be responsible for creating scalable data pipelines, ensuring data quality, and optimizing data processing performance. You'll work closely with teams such as data analysts, data scientists, and software developers to ensure they have the necessary data for their operations.

Key Responsibilities

• Designing and Implementing Data Storage Solutions: You will design and implement data storage solutions on Azure, selecting the appropriate services like Azure SQL Database, Azure Cosmos DB, or Azure Data Lake Storage. Your role includes designing optimized data storage schemas suited for each use case.
• Building and Maintaining Data Pipelines: Construct and manage data pipelines that handle data integration and processing. This involves collecting data from various sources, transforming it into usable formats, and loading it into the appropriate Azure storage solutions using tools like Azure Data Factory and Azure Databricks.
• Ensuring Data Quality and Accuracy: You'll ensure the accuracy and reliability of data through rigorous testing and validation at each stage of the data pipeline, from extraction to loading.
• Optimizing Data Processing Performance: Tuning and monitoring data pipelines for efficiency and scalability is key. You will identify bottlenecks and optimize data processing algorithms to enhance performance.
• Developing and Maintaining Data Models and Schemas: Design data models and schemas that are tailored to the business use case, ensuring they are scalable and efficient.
• Collaborating Across Teams: Work closely with data analysts, scientists, and software developers to provide them with data in the required formats and locations on Azure.
• Ensuring Data Security and Privacy Compliance: You will ensure that the data stored and processed in Azure adheres to security and privacy standards, including compliance with regulations such as HIPAA and GDPR.
   

Essential Skills

• Proficiency in SQL and Data Query Languages: Strong command of SQL, T-SQL, and PL/SQL is essential for querying and managing data.
• Experience with Azure Data Storage Solutions: Hands-on experience with Azure services such as Azure SQL Database, Azure Cosmos DB, and Azure Data Lake Storage is required to select the right service for the given use case.
• Knowledge of Data Integration Tools: Familiarity with tools like Azure Data Factory and Azure Databricks for building and managing data pipelines is crucial.
• Familiarity with Data Processing Frameworks: Understanding and experience with frameworks like Apache Spark and Hadoop to optimize data processing tasks.
• Understanding of Data Modeling and Schema Design: You should have a solid understanding of data modeling and schema design principles to develop efficient, scalable data structures.
• Experience with Large Datasets and Data Analysis: You should be comfortable working with large datasets and performing data analysis using tools like Azure Machine Learning.
• Problem-solving and Troubleshooting Skills: Strong analytical skills are needed to troubleshoot data pipeline issues and optimize performance.
   

Required Experience

• In-depth technical Expertise: Comprehensive knowledge of cloud computing, particularly Azure environments and services.
• Data Integration Experience: Proficient in data integration tools and techniques, ensuring secure and efficient data flow.
• Proven ETL Experience: Demonstrated experience with Extract, Transform, and Load (ETL) processes for transforming raw data into usable formats.
• Azure Environment Expertise: Hands-on experience integrating with Azure Data Lake, Azure Databricks, and Azure Synapse Analytics.
• Deployment Speed Optimization: Familiarity with Resource Manager Templates to accelerate deployment.
• Programming Skills: Proficiency in programming languages such as Java and Python.
• Database Expertise: Strong knowledge of both NoSQL and SQL databases.
• Leadership and Communication Skills: Strong team leadership, communication, and collaboration skills to work effectively across departments.
• Stakeholder Engagement: Ensure stakeholders receive real-time data analysis to aid decision-making.
• Innovative Problem-Solving: Collaborate with cross-functional teams to develop innovative solutions to data challenges and provide data-driven insights to support business decisions.