Cloud Data Engineer
Company: TechDigital Corporation
Posted on: November 20, 2022
Cloud Data Engineer who will be responsible for standardizing
our data ingestion pipelines and improving performance of data
storage and retrieval. Our projects ingest structured and
unstructured data from multiple sources that eventually ends up in
a unified Data Lake. This data is available for reporting and
analytics by various other teams. This candidate should be
comfortable working with SQL as well as NoSQL databases. AWS
experience is required.
Your primary focus will be the continuous development/improvement
of our data ingestion pipelines, and data parsing. Major
technologies involved include AWS CloudFormation, Step Functions,
Lambda Functions, AWS Glue, Amazon Athena, Redshift and RDS.
Skills and Responsibilities
Design and develop stream and batch processing data pipelines
Ability to write intermediate to advanced Python and SQL for data
ingestion and processing.
Support existing and develop new data flows as needed by developing
processes that verify, standardize, and scale data input,
transformation and storage.
Collaborate with product managers and other engineers to implement
and document complex and evolving requirements
Research and implement innovative solutions to solve challenges
related to ETL, data processing, and analytics.
Support efforts by the Data Science and Data Analytics teams.
Ability to debug complex data issues without frequent guidance from
senior team members.
Occasional Linux server management including the review or
management of log files, crontab, security configuration, etc.
Developing techniques to work with both tabular and hierarchical
Strong attention to detail, good work ethic, ability to work on
multiple projects simultaneously, and good communication skills
3 to 6 years of experience working in Data Engineering or related
1+ years of Cloud experience required!
A degree in Computer Science or an equivalent major.
Must have strong programming & ETL skills in Python & SQL.
Experience working with various database systems: SQL
Experience working with Step Functions, Lambda Functions, AWS Glue,
& Amazon Athena.
Experience with agile methodologies and short release cycles
Distributed System Development for large-scale applications
Enjoys collaborating with other engineers on architecture and
sharing designs with the team
Hands-on experience with writing CloudFormation
Experience working with PHI/Healthcare
AI / Client
Keywords: TechDigital Corporation, Reston , Cloud Data Engineer, Engineering , Reston, Virginia
Didn't find what you're looking for? Search again!