Epos Nowโs real issue was architectural overload: one operational SQL Server was being asked to handle transaction processing, reporting, and dashboard delivery for a fast-growing merchant base. That coupling created slow dashboard loads, degraded performance, and rising cost and scalability pressure. The strategic move was to split workloads by access pattern, using Aurora PostgreSQL-Compatible Edition for near-real-time metrics and building a separate analytics platform for broader reporting and machine-learning-oriented use cases. That separation is the real modernization, not the cloud branding around it.
The new pipeline is built around three layers: S3 as the raw data lake, Redshift as the analytical warehouse, and a fast access layer for merchant-facing metrics. MSK Connect and Airbyte feed the landing zone, with dead-letter queues improving resilience for bad messages and retention policies limiting storage bloat. Glue moves curated data into Redshift, where dbt handles ELT in SQL after staging. That choice matters because it preserves familiar skills while enforcing a cleaner model for BI and interactive querying.
The operational value is credible but bounded by execution complexity. Epos Now is still implementing the full lakehouse architecture, and the fast access layer depends on a separate consumer service plus API Gateway, which adds moving parts and integration risk. The payoff comes from matching storage to workload rather than forcing one database to do everything, but that also requires disciplined partitioning, file formats, and retention management. Practitioners should read the result as a workload split that improves stability, not a universal replacement for simpler reporting stacks.
- The first was to rearchitect the near-real-time reporting feature by moving it to a dedicated
Amazon Aurora PostgreSQL-Compatible Edition database, with a specific reporting data model to serve to end consumers. This will improve performance, uptime, and cost.
- The second was to build out a new data platform for reporting, dashboards, and advanced analytics. This will enable use cases for internal data analysts and data scientists to experiment and create multiple data products, ultimately exposing these insights to end customers.
AWS Data Lab. Having developed a strong strategic relationship with AWS over the last 3 years, Epos Now opted to take advantage of the AWS Data lab program to speed up the process of building a reliable, performant, and cost-effective data platform. The AWS Data Lab program offers accelerated, joint-engineering engagements between customers and AWS technical resources to create tangible deliverables that accelerate data and analytics modernization initiatives.
Working with an AWS Data Lab Architect, Epos Now commenced weekly cadence calls to come up with a high-level architecture. After the objective, success criteria, and stretch goals were clearly defined, the final step was to draft a detailed task list for the upcoming 3-day build phase.
Overview of solution
As part of the 3-day build exercise, Epos Now built the following solution with the ongoing support of their AWS Data Lab Architect.
The platform consists of an end-to-end data pipeline with three main components:
- Data lake โ As a central source of truth
- Data warehouse โ For analytics and reporting needs
- Fast access layer โ To serve near-real-time reports to merchants
Amazon Redshift to create a federated data warehouse with conformed dimensions and star schemas for consumption by Microsoft Power BI, running on AWS
- Aurora PostgreSQL to store all the data for near-real-time reporting as a fast access layer
Data lake
The first component of the data pipeline involved ingesting the data from an
Amazon MSK Connect to land the data into an S3 bucket (landing zone). The Epos Now team used the Confluent Amazon S3 sink connector to sink the data to Amazon S3. To make the sink process more resilient, Epos Now added the required configuration for dead-letter queues to redirect the bad messages to another topic. The following code is a sample configuration for a dead-letter queue in Amazon MSK Connect:
Because Epos Now was ingesting from multiple data sources, they used Airbyte to transfer the data to a landing zone in batches. A subsequent 
data retention policy on different layers of the data lake to save money as well as keep the dataset relevant.
Data warehouse
After the data lake foundation was established, Epos Now used a subsequent AWS Glue job to load the data from the S3 curated layer to Amazon Redshift. We used Amazon Redshift to make the data queryable in both Amazon Redshift (internal tables) and
Amazon Redshift Spectrum. The team then used dbt as an extract, load, and transform (ELT) engine to create the target data model and store it in target tables and views for internal business intelligence reporting. The Epos Now team wanted to use their SQL knowledge to do all ELT operations in Amazon Redshift, so they chose dbt to perform all the joins, aggregations, and other transformations after the data was loaded into the staging tables in Amazon Redshift. Epos Now is currently using Power BI for reporting, which was migrated to the AWS Cloud and connected to Amazon Redshift clusters running inside Epos Nowโs VPC.
Fast access layer
To build the fast access layer to deliver the metrics to Epos Nowโs retail and hospitality merchants in near-real time, we decided to create a separate pipeline. This required developing a microservice running a Kafka consumer job to subscribe to the same Kafka topic in an
Amazon API Gateway.
Outcome
The Epos Now team is currently building both the fast access layer and a centralized lakehouse architecture-based data platform on Amazon S3 and Amazon Redshift for advanced analytics use cases. The new data platform is best positioned to address scalability issues and support new use cases. The Epos Now team has also started offloading some of the real-time reporting requirements to the new target data model hosted in Aurora. The team has a clear strategy around the choice of different storage solutions for the right access patterns: Amazon S3 stores all the raw data, and Aurora hosts all the metrics to serve real-time and near-real-time reporting requirements. The Epos Now team will also enhance the overall solution by applying data retention policies in different layers of the data platform. This will address the platform cost without losing any historical datasets. The data model and structure (data partitioning, columnar file format) we designed greatly improved query performance and overall platform stability.Conclusion
Epos Now revolutionized their data analytics capabilities, taking advantage of the breadth and depth of the AWS Cloud. Theyโre now able to serve insights to internal business users, and scale their data platform in a reliable, performant, and cost-effective manner. The AWS Data Lab engagement enabled Epos Now to move from idea to proof of concept in 3 days using several previously unfamiliar AWS analytics services, including AWS Glue, Amazon MSK, Amazon Redshift, and Amazon API Gateway. Epos Now is currently in the process of implementing the full data lake architecture, with a rollout to customers planned for late 2022. Once live, they will deliver on their strategic goal to provide real-time transactional data and put insights directly in the hands of their merchants.About the Authors
Jason Downing is VP of Data and Insights at Epos Now. He is responsible for the Epos Now data platform and product direction. He specializes in product management across a range of industries, including POS systems, mobile money, payments, and eWallets. Debadatta Mohapatra is an AWS Data Lab Architect. He has extensive experience across big data, data science, and IoT, across consulting and industrials. He is an advocate of cloud-native data platforms and the value they can drive for customers across industries.Enjoyed this article? Sign up for our newsletter to receive regular insights and stay connected.