Doing more with less: Moving from transactional to stateful batch processing

Our new architecture needed to address the deficiencies while preserving the core goal of our service: update stateful artifacts based on incoming financial events.

We used an Apache Spark application on a long-running Amazon EMR cluster to simultaneously ingest input batch data and perform reduce operations to produce the stateless artifacts and a corresponding index file for the stateful processing to use.

Continue reading

Authentication (AuthN) and Authorization (AuthZ) Tutorial for Kubernetes

This article will define and contrast AuthN and AuthZ, then delve into how they can be used in the context of Kubernetes to secure your cluster and its operations.

System components, such as kubelets (the Kubernetes node agent), pods, and other components of the control plane (scheduler and controllers) also need to authenticate with the API server.

Continue reading

Integrating IT Security with DevSecOps: Best Practices

By integrating DevOps practices with IT security, also known as DevSecOps, enterprises can create a strong chain of defense around their most important applications and assets from the development process through to operations and deployment.

Many organizations are moving toward DevSecOps as an integrated approach to software development and security

Continue reading

Best DevOps Automation Tools 2022

DevOps automation helps software development teams achieve faster time-to-market for new software releases by speeding up certain aspects of the development pipeline—all while ensuring products are high-quality and error-free.

These include tools to automate version control, container management, application performance monitoring (APM), deployment and server monitoring, configuration management, CI/CD automation, artifact management, and codeless testing.

Continue reading

Build a REST API to enable data consumption from Amazon Redshift

In this post, we walk through setting up an application API using the Amazon Redshift Data API, AWS Lambda, and Amazon API Gateway. The API performs asynchronous processing of user requests, sends user notifications, saves processed data in Amazon Simple Storage Service (Amazon S3), and returns a presigned URL for the user or application to download the dataset over HTTPS

Continue reading

1 172 173 174 175 176 180