The Largest Domain Registrar in US needed a serverless ETL component for its Revenue API system, a core piece of Project Silverstone that separates revenue accounting and eCommerce functions. ASCENDING designed and built that pipeline on AWS Glue, delivering a production system that continuously transforms and processes millions of transactional records every day.
Background
The client serves more than 20 million customers and manages over 80 million domains from 14 offices around the world. To support rapid customer growth and acquisition, it launched Project Silverstone to separate revenue accounting from eCommerce functions. Revenue API, one of the most important pieces of that project, is a serverless, integrated, cloud-based system that validates and processes data from different eCommerce entities and sends it to a third-party data warehouse, RevStream, for further calculation and consolidation. AWS Glue serves as the only ETL component in Revenue API, continuously transferring and processing enterprise-level data on a daily basis.
The Challenge
The client needed an ETL component that could integrate tightly with S3, RDS, and Lambda while handling complex data transformations at production scale.
- Revenue API was the first internal project to use AWS Glue heavily, both for complex transformation logic and for close integration with other serverless AWS services.
- The team ran into specific implementation challenges around Glue features like Glue Crawler and Glue Jobs, along with developing the PySpark programs those jobs relied on.
- Those challenges were compounded by a steep learning curve, since the internal development team was exploring these serverless data engineering patterns for the first time.
- The client needed experts with real AWS Glue and ETL experience rather than a team learning the technology on the job.
Why ASCENDING
ASCENDING is an AWS Advanced Consulting Partner and Lambda Service Delivery Partner, with an all AWS-certified engineering team. Compared to other candidates and vendors, ASCENDING demonstrated a stronger understanding of AWS serverless services and deeper AWS data competency, which is what ultimately led the client to select ASCENDING for the assignment.
The Solution
ASCENDING designed and developed the full data transformation process for the client's Revenue API system using AWS Glue.
- Hundreds of thousands of order transaction records in JSON format flow through API Gateway and Lambda functions for primary data validation before arriving at the Glue ETL process.
- The Glue ETL process generates each transaction and its corresponding order item lines from the JSON data in an S3 bucket, then combines and joins that data with reference and mapping tables in RDS.
- The pipeline produces its final output as CSV files written to a separate S3 bucket, ready for downstream consumption by RevStream.
- Beyond standard Glue integration with S3 and RDS, ASCENDING built supporting Lambda functions for specific business needs, including a Secrets Manager integration so every credential rotation automatically updates the Glue Connection secret.
- ASCENDING also solved scheduling and orchestration for Glue Jobs using Glue Trigger, and worked through the logic trade-offs between Lambda functions, Glue PySpark programs, and RDS reference tables to meet the client's corporate standards and compliance requirements.
The Outcome
The ETL solution ASCENDING built for the client's Revenue API project has processed millions of transactional records with consistent performance in production, establishing a reusable pattern for the rest of the organization.
- Successfully transferred and processed millions of transactional data records with consistent production performance.
- Established a working precedent for AWS Glue ETL solutions that other internal projects and teams can build on.
- Introduced Glue and serverless ETL patterns that were the first of their kind internally, including Secrets Manager-integrated connection rotation.
- Set up an ongoing partnership, with ASCENDING and the client continuing to explore new AWS serverless capabilities together.
