Amazon Redshift Serverless makes it simple to run and scale analytics without having to manage the instance type, instance size, lifecycle management, pausing, resuming, and so on. It automatically provisions and intelligently scales data warehouse compute capacity to deliver fast performance for even the most demanding and unpredictable workloads, and you pay only for what you use. Just load your data and start querying right away in the Amazon Redshift Query Editor or in your favorite business intelligence (BI) tool and continue to enjoy the best price performance and familiar SQL features in an easy-to-use, zero administration environment.
Redshift Serverless separates compute and storage and introduces two abstractions:
- Workgroup – A workgroup is a collection of compute resources. It groups together compute resources like RPUs, VPC subnet groups, and security groups.
- Namespace – A namespace is a collection of database objects and users. It groups together data objects, such as databases, schemas, tables, users, or AWS Key Management Service (AWS KMS) keys for encrypting data.
Some organizations want to automate the creation of workgroups and namespaces for automated infrastructure management and consistent configuration across environments, and provide end-to-end self-service capabilities. You can automate the workgroup and namespace management operations using the Redshift Serverless API, the AWS Command Line Interface (AWS CLI), or AWS CloudFormation, which we demonstrate in this post.
Solution overview
In the following sections, we discuss the automation approaches for various tasks involved in Redshift Serverless data warehouse management using AWS CloudFormation (for more information, see RedshiftServerless resource type reference) and the AWS CLI (see redshift-serverless).
The following are some of the key use cases and appropriate automation approaches to use with AWS CloudFormation:
- Enable end-to-end self-service from infrastructure setup to querying
- Automate data consumer onboarding for data provisioned through AWS Data Exchange
- Accelerate workload isolation by creating endpoints
- Create a new data warehouse with consistent configuration across environments
The following are some of the main use cases and approaches for the AWS CLI:
- Automate maintenance operations:
- Backup and limits
- Modify RPU configurations
- Manage limits
- Automate migration from provisioned to serverless
Prerequisites
To run the operations described in this post, make sure that this user or role has AWS Identity Access and Management (IAM) arn:aws:iam::aws:policy/AWSCloudFormationFullAccess, and either the administrator permission arn:aws:iam::aws:policy/AdministratorAccess or the full Amazon Redshift permission arn:aws:iam::aws:policy/AmazonRedshiftFullAccess policy attached. Refer to Security and connections in Amazon Redshift Serverless for further details.
You should have at least three subnets, and they must span across three Availability Zones.It is not enough if just 3 subnets created in same availability zone. To create a new VPC and subnets, use the following CloudFormation template to deploy in your AWS account.
Create a Redshift Serverless namespace and workgroup using AWS CloudFormation
AWS CloudFormation helps you model and set up your AWS resources so that you can spend less time on infrastructure setup and more time focusing on your applications that run in AWS. You create a template that describes all the AWS resources that you want, and AWS CloudFormation takes care of provisioning and configuring those resources based on the given input parameters.
To create the namespace and workgroup for a Redshift Serverless data warehouse using AWS CloudFormation, complete the following steps:
- Choose Launch Stack to launch AWS CloudFormation in your AWS account with a template:
- For Stack name, enter a meaningful name for the stack, for example,
rsserverless. - Enter the parameters detailed in the following table.
| Parameters | Default | Allowed Values | Description |
| Namespace | . | N/A | The name of the namespace of your choice to be created. |
| Database Name | dev | N/A | The name of the first database in the Redshift Serverless environment. |
| Admin User Name | admin | N/A | The administrator’s user name for the Redshift Serverless namespace being create. |
| Admin User Password | . | N/A | The password associated with the admin user. |
| Associate IAM Role | . | Comma-delimited list of ARNs of IAM roles | Associate an IAM role to your Redshift Serverless namespace (optional). |
| Log Export List | userlog, connectionlog, useractivitylog | userlog, connectionlog, useractivitylog | Provide comma-separated values from the list. For example, userlog, connectionlog, useractivitylog. If left blank, LogExport is turned off. |
| Workgroup | . | N/A | The workgroup name of your choice to be created. |
| Base RPU | 128 | Minimum value of 32 and maximum value of 512 | The base RPU for the Redshift Serverless workgroup. |
| Publicly accessible | false | true, false | Indicates if the Redshift Serverless instance is publicly accessible. |
| Subnet Ids | . | N/A | You must have at least three subnets, and they must span across three Availability Zones. |
| Security Group Id | . | N/A | The list of security group IDs in your VPC. |
| Enhanced VPC Routing | false | true, false | The value that specifies whether to enable enhanced VPC routing, which forces Redshift Serverless to route traffic through your VPC. |
- Pass the parameters provided to the
AWS::RedshiftServerless::NamespaceandAWS::RedshiftServerless::Workgroupresource types:Resources: RedshiftServerlessNamespace: Type: 'AWS::RedshiftServerless::Namespace' Properties: AdminUsername: Ref: AdminUsername AdminUserPassword: Ref: AdminUserPassword DbName: Ref: DatabaseName NamespaceName: Ref: NamespaceName IamRoles: Ref: IAMRole LogExports: Ref: LogExportsList RedshiftServerlessWorkgroup: Type: 'AWS::RedshiftServerless::Workgroup' Properties: WorkgroupName: Ref: WorkgroupName NamespaceName: Ref: NamespaceName BaseCapacity: Ref: BaseRPU PubliclyAccessible: Ref: PubliclyAccessible SubnetIds: Ref: SubnetId SecurityGroupIds: Ref: SecurityGroupIds EnhancedVpcRouting: Ref: EnhancedVpcRouting DependsOn: - RedshiftServerlessNamespace
Perform namespace and workgroup management operations using the AWS CLI
The AWS CLI is a unified tool to manage your AWS services. With just one tool to download and configure, you can control multiple AWS services from the command line and automate them through scripts.
To run the Redshift Serverless CLI commands, you need to install the latest version of AWS CLI. For instructions, refer to Installing or updating the latest version of the AWS CLI.
Now you’re ready to complete the following steps:
Use the following command to create a new namespace:
aws redshift-serverless create-namespace \
--admin-user-password '<password>' \
--admin-username cfn-blog-admin \
--db-name cfn-blog-db \
--namespace-name 'cfn-blog-ns'The following screenshot shows an example output.
Use the following command to create a new workgroup mapped to the namespace you just created:
aws redshift-serverless create-workgroup \
--base-capacity 128 \
--namespace-name 'cfn-blog-ns' \
--no-publicly-accessible \
--security-group-ids "sg-0269bd680e0911ce7" \
--subnet-ids "subnet-078eedbdd99398568" "subnet-05defe25a59c0e4c2" "subnet-0f378d07e02da3e48"\
--workgroup-name 'cfn-blog-wg'The following is an example output.
To allow instances and devices outside the VPC to connect to the workgroup, use the publicly-accessible option in the create-workgroup CLI command.
To verify the workgroup has been created and is in AVAILABLE status, use the following command:
aws redshift-serverless get-workgroup \
--workgroup-name 'cfn-blog-wg' \
--output text \
--query 'workgroup.status'The following screenshot shows our output.
Regardless of whether your snapshot was made from a provisioned cluster or serverless workgroup, it can be restored into a new serverless workgroup. Restoring a snapshot replaces the namespace and workgroup with the contents of the snapshot.
Use the following command to restore from a snapshot:
aws redshift-serverless restore-from-snapshot \
--namespace-name 'cfn-blog-ns' \
--snapshot-arn arn:aws:redshift:us-east-1:<account-id>:snapshot:<cluster-identifier>/<snapshot-identifier> \
--workgroup-name 'cfn-blog-wg'The following is an example output.
To check the workgroup status, run the following command:
aws redshift-serverless get-workgroup \
--workgroup-name 'cfn-blog-wg' \
--output text \
--query 'workgroup.status'
To create a snapshot from an existing namespace, run the following command:
aws redshift-serverless create-snapshot \
--namespace-name cfn-blog-ns \
--snapshot-name cfn-blog-snapshot-from-ns \
--retention-period 7The following is an example output.
Redshift Serverless creates recovery points of your namespace that are available for 24 hours. To keep your recovery point longer than 24 hours, convert it to a snapshot.
To find the recovery points associated to your namespace, run the following command:
aws redshift-serverless list-recovery-points \
--namespace-name cfn-blog-ns \
--no-paginateThe following an example output with the list of all the recovery points.
- CRISP DETAIL, CLEAR DISPLAY: Enjoy binge-watching...
- PRO SHOTS WITH EASE: Brilliant sunrises, awesome...
- CHARGE UP AND CHARGE ON: Always be ready for an...
- POWERFUL 5G PERFORMANCE: Do what you love most —...
- NEW LOOK, ADDED DURABILITY: Galaxy A54 5G is...
- Free 6 months of Google One and 3 months of...
- Pure Performance: The OnePlus 12 is powered by the...
- Brilliant Display: The OnePlus 12 has a stunning...
- Powered by Trinity Engine: The OnePlus 12's...
- Powerful, Versatile Camera: Explore the new 4th...
Let’s take the latest recoveryPointId from the list and convert to snapshot.
To create a snapshot from a recovery point, run the following command:
aws redshift-serverless convert-recovery-point-to-snapshot \
--recovery-point-id f9eaf9ac-a98d-4809-9eee-869ef03e98b4 \
--retention-period 7 \
--snapshot-name cfn-blog-snapshot-from-rpThe following is an example output.
In addition to restoring a snapshot to a serverless namespace, you can also restore from a recovery point.
- First, you need to find the recovery point identifier using the list-recovery-points command.
- Then use the following command to restore from a recovery point:
aws redshift-serverless restore-from-recovery-point \
--namespace-name cfn-blog-ns \
--recovery-point-id 15c55fb4-d973-4d8a-a8fe-4741e7911137 \
--workgroup-name cfn-blog-wgThe following is an example output.
The base RPU determines the starting capacity for your serverless environment.
Use the following command to modify the base RPU based on your workload requirements:
aws redshift-serverless update-workgroup \
--base-capacity 256 \
--workgroup-name 'cfn-blog-wg'The following is an example output.
Run the following command to verify the workgroup base RPU capacity has been modified to 256:
aws redshift-serverless get-workgroup \
--workgroup-name 'cfn-blog-wg' \
--output text \
--query 'workgroup.baseCapacity'
To keep costs predictable for Redshift Serverless, you can set the maximum RPU hours used per day, per week, or per month. In addition, you can take action when the limit is reached. Actions include: write a log entry to a system table, receive an alert, or turn off user queries.
Use the following command to first get the workgroup ARN:
aws redshift-serverless get-workgroup --workgroup-name 'cfn-blog-wg' \
--output text \
--query 'workgroup.workgroupArn'The following screenshot shows our output.
Use the workgroupArn output from the preceding command with the following command to set the daily RPU usage limit and set the action behavior to log:
aws redshift-serverless create-usage-limit \
--amount 256 \
--breach-action log \
--period daily \
--resource-arn arn:aws:redshift-serverless:us-east-1:<aws-account-id>:workgroup/1dcdd402-8aeb-432e-8833-b1f78a112a93 \
--usage-type serverless-computeThe following is an example output.
Conclusion
- 【Octa-Core CPU + 128GB Expandable TF Card】...
- 【6.8 HD+ Android 13.0】 This is an Android Cell...
- 【Dual SIM and Global Band 5G Phone】The machine...
- 【6800mAh Long lasting battery】With the 6800mAh...
- 【Business Services】The main additional...
- 【Dimensity 9000 CPU + 128GB Expandable TF...
- 【6.8 HD+ Android 13.0】 This is an Android Cell...
- 【Dual SIM and Global Band 5G Phone】Dual SIM &...
- 【6800mAh Long lasting battery】The I15 Pro MAX...
- 【Business Services】The main additional...
- 🥇【6.8" HD Unlocked Android Phones】Please...
- 💗【Octa-Core CPU+ 256GB Storage】U24 Ultra...
- 💗【Support Global Band 5G Dual SIM】U24 Ultra...
- 💗【80MP Professional Photography】The U24...
- 💗【6800mAh Long Lasting Battery】With the...
You have now learned how to automate management operations on Redshift Serverless namespaces and workgroups using AWS CloudFormation and the AWS CLI. To automate creation and management of Amazon Redshift provisioned clusters, refer to Automate Amazon Redshift Cluster management operations using AWS CloudFormation.
About the Authors
Ranjan Burman is a Analytics Specialist Solutions Architect at AWS. He specializes in Amazon Redshift and helps customers build scalable analytical solutions. He has more than 15 years of experience in different database and data warehousing technologies. He is passionate about automating and solving customer problems with the use of cloud solutions.
Satesh Sonti is a Sr. Analytics Specialist Solutions Architect based out of Atlanta, specialized in building enterprise data platforms, data warehousing, and analytics solutions. He has over 16 years of experience in building data assets and leading complex data platform programs for banking and insurance clients across the globe.
Urvish Shah is a Senior Database Engineer at Amazon Redshift. He has more than a decade of experience working on databases, data warehousing and in analytics space. Outside of work, he enjoys cooking, travelling and spending time with his daughter.