Things to know before applying:
you must have your own work equipment
you are able to work remotely
you can work full-time for this role
you can work night shift
experience working with clients/companies outside PH is an advantage
only candidates who meet the requirements will be contacted
Role: Data Pipeline Engineer
Work Schedule: Night shift (w/ 10% night differential)
Role Summary:
We are seeking a Data Pipeline Engineer to design, implement and maintain data processes using technologies such as SQL Server stored procedures, DTS and SSIS. These pipelines ingest large volumes of vehicle catalogue data from a diverse set of sources, and transform them to a normalised schema for access from our Electronic Parts Catalogue (EPC) system.
This role is ideal for a data pipeline engineer skilled in Microsoft technologies, who understands the intersection of data processing and software development. The position offers the opportunity to deepen experience high-volume data processing, and gain exposure to the broader ecosystem of operational data pipelines, software product development and DevOps.
The successful applicant will work within a larger team of engineers and will collaborate closely with stakeholders through our organisation, ensuring the accurate and efficient flow of data.
Key Responsibilities:
Design and develop: Create and maintain data ETL pipelines on SQL Server using T-SQL, stored procedures, SSIS, etc.
Collaboration: Work closely with data analysts, engineers, and other stakeholders to understand data requirements and develop solutions.
Data Quality: Implement data quality checks and validation processes to ensure data accuracy and consistency.
Monitoring and Troubleshooting: Monitor data pipeline performance and promptly resolve issues.
Documentation: Develop and maintain documentation for data pipeline processes and workflows.
Continuous Improvement: Work with the broader team to improve technology solutions and ways of working.
Data Ingestion, Processing, and Publishing:
Ingest and process order maker data, including data assets from the manufacturer (such as VIN, model, etc).
Perform analysis, mapping, cleaning, and processing of the data.
Publish the processed data set.
Technology Stack:
Utilise a Microsoft-based technology stack.
Ensure the e-commerce platform is fully integrated with Microsoft's ecosystem, driven by SQL Server.
Data Model and Publishing: Publish the processed data into a canonical data model that integrates with the e-commerce platform.
Tool Migration:
Eventually retire existing e-commerce specific data processing tools.
Plan for a future migration to Apache Spark.
Qualifications:
Experience: 5+ years of experience in data engineering, with a focus on pipeline development and maintenance.
Technical Skills: 10+ years of experience Microsoft data technologies, ideally including SQL Server, Stored Procedures, SSIS, T-SQL, DTS.
Knowledge: Strong knowledge of SQL-based database systems and data modelling.
Problem Solving: Excellent problem-solving skills and attention to detail.
Communication: Strong communication and collaboration skills.
Industry Knowledge: Familiarity with the automotive industry and vehicle catalogue data is a plus, but not required.
Growth Mindset: a willingness to learn and adapt to new technologies and methodologies, ideally having an influence on solutions and ways of working.
Hiring Process:
Phone screening
Behavioral interview
Project fit check interview
Client interview
About Adaca:
Adaca is at the forefront of bridging software development solutions by connecting Australian mid-tier companies with top-tier talents in the Philippines. Our unique approach is rooted in creating an employee-centric experience that fosters a simple, low-pressure environment for our developers. This philosophy not only enhances productivity but also delivers unparalleled value to our clients, fostering a cycle of growth and talent attraction.
