Senior Data Engineer
Responsibilities:
DATA ENGINEERING
- Use Airflow for workflow orchestration to create ETL pipelines that ingest application,loan, fraud, payment, model data from disparate data sources and validate end product
- Use data modeling techniques to store above data in an AWS S3 Data Lake
- Collaborate closely with data science to design new models, aggregating and validatingdifferent features and targets
- Leverage Trino and Pandas for comprehensive analysis of application, loan, fraud,payment, and model datasets, delivering critical insights
- Work with data vendors to provide additional info for segmenting/labeling
customers/apps
- Pull customer/application data for data retros or debt sales
SOFTWARE ENGINEERING
- Implement XGBoost models in Python
- Design and implement new underwriting policies in Python
- Design and implement backend features in Django REST framework to supportcustomer segmentation, A/B testing, and selection of underwriting strategies
- Design and implement decision models using Python in a Django framework
- Integrate new data vendors into the Django web server, designing and constructingrobust data models
- Adapt API request and response structures to accommodate evolving requirements
- Collaborate with external partners to facilitate the implementation of models and ensuretheir accuracy through comprehensive validation
- Unit/end to end testing of models and APIs
DEVOPS/SITE RELIABILITY
- Use Opensearch Slack alerts, AWS SES (email), and AWS Cloudwatch alerts to monitorthe following systems: DecisioningEngine, Modelhoster, Airflow, and UOwnCoreLambda
- Design and implement API endpoints utilizing AWS CDK and AWS Lambda
- Build and maintain CDK AWS CodePipelines for CI/CD
- Maintain AWS Opensearch service. Write queries for aggregating underwriting data used for debugging and alerting purposes
- Deploy and maintain following docker swarm clusters: DecisioningEngine and
Modelhoster
- Maintain/deploy MySQL DB servers: DecisioningEngine and Modelhoster
- Maintain/deploy PostgreSQL DB servers: Zoca
- Maintain/deploy various EC2 containers
- Debug/monitor website/backend logs using CloudWatch
- Design and build docker containers for various services
MISCELLANEOUS
- Maintain/extend single page application built with Node.js, React, MobX, and SCSScode
- Maintain/extend landing page built with Python and Django framework
- Maintain/extend web server built with Python Sanic framework
- Answer ad hoc requests for audits by examining various data stores (Opensearch,Postgresql, MySql, Cloudwatch logs, etc), sometimes writing Python scripts to pull bulkdata
- Write python scripts to interface with LMS via SOAP based APIs or chromium scripts
- Automate underwriting tasks using Docker, Bash and Python such as settling loans orpulling delinquent loans within certain time windows, etc
- Reach out to customers via CampaignMonitor and Twilio for marketing or loansettlement purposes
- Work with data vendors to identify new rules/triggers to stop frau #J-18808-Ljbffr
