OTR Transportation's growing customer base meant its brokers needed intelligent dashboards covering market demand, container shipments, and traffic footage to make better rate decisions, but the underlying data was scattered across multiple AWS accounts with no central way to query it. ASCENDING built a secure, centralized data lake on AWS Lake Formation that unified those sources and gave OTR's team self-service access to the analytics it needed.
Background
OTR Transportation, Inc., headquartered in Chicago, is a player in the freight transportation arrangement industry known for its experienced team of brokers spanning myriad industries. As OTR's customer base grew, the company wanted to develop intelligent dashboards that would let brokers pull in data on market-demanded areas, container shipments, and traffic footage so stakeholders could make better rate decisions based on real market trends.
OTR needed centralized data storage for the large volume of data generated by its multiple ongoing data acquisition processes, with that same data lake serving as the foundation for future analytics, reporting, and machine learning applications.
The Challenge
OTR's data was growing faster than its existing processes could organize or make useful.
- More than 10 GB of user log and API interaction data was being generated every day, creating a data governance and management bottleneck.
- OTR had no way to run ad hoc queries or research existing data, and instead depended on data specialists to manually finalize Excel reports and metrics.
- Data originated from three separate AWS application accounts, with no centralized, secure way to bring it together.
- Any solution needed to scale with OTR's continued growth while remaining secure and properly governed.
Why ASCENDING
ASCENDING is an AWS Certified Advanced Consulting Partner with deep experience in serverless architecture, ETL, data visualization, and AWS infrastructure automation, having delivered similar data lake projects for organizations at various scales. Previous clients had described the ASCENDING team as responsive, friendly, and easy to communicate with, which, combined with that technical track record, made ASCENDING OTR's choice to take on the assignment.
The Solution
Since OTR's data sources were generated across three different AWS application accounts, ASCENDING designed a centralized AWS account to house the data lake and unify everything in one place.
- AWS Glue and Amazon Kinesis moved data from third-party sources, NoSQL databases, and relational databases into Source S3 buckets that served as the data source layer.
- S3 policies enabled secure cross-account access, letting data flow from each application account into the central data lake account.
- Glue Jobs securely transformed and enriched the data as it moved into the data lake, landing in a dedicated "Landing Bucket."
- Once cataloged in AWS Lake Formation, downstream applications like Amazon Athena and Amazon QuickSight were granted access for further analysis and dashboarding.
The Outcome
ASCENDING delivered the data lake infrastructure on OTR's timeline, giving the company a centralized, secure foundation for its growing data needs.
- Resolved OTR's core big data challenges around data governance, scalable infrastructure, and rapid data acquisition.
- Created a centralized, secured data lake for both internal and external data sources, with access governed by AWS Lake Formation's permissions model.
- Gave staff across the organization the ability to access data with proper permissions and constraints, rather than routing every request through data specialists.
- Included ongoing system maintenance, performance monitoring, and access monitoring as part of the engagement.
- Saved OTR significant time and cost on repeated manual work, freeing the team to focus on turning data into business value.
