ClickPipes supports cross-VPC resource access on AWS!

Today, we’re proud to announce that ClickPipes now supports AWS’s new way to provide private, unidirectional connectivity to individual data sources using AWS PrivateLink and VPC Lattice.

ClickPipes is our continuous data ingestion service designed specifically for ClickHouse Cloud. It connects your external data sources like Amazon MSK, Amazon S3, or Amazon Kinesis, enabling you to set up robust and scalable data pipelines quickly and efficiently to get data into your ClickHouse service. Since its inception, ClickPipes has provided support for AWS PrivateLink to support our customers wanting to avoid their data and network traffic going over the public internet.

AWS PrivateLink architecture with load balancer

Cross-VPC resource access with AWS PrivateLink and VPC Lattice gives the added benefit of reducing the amount of resources needed as you no longer require a load-balancer (ALB/NLB) to take advantage of sharing your data to ClickPipes.

This new capability leverages AWS PrivateLink VPC endpoints and VPC Lattice to share and access resources across VPCs and accounts.

AWS PrivateLink architecture without load balancer

A resource owner can share specific resources from their VPC, without exposing any other resources in that VPC to a resource consumer. A resource consumer can access shared resources in another VPC/account privately.

With this capability, you can connect your individual resources across VPC and account boundaries, or even from on-premise networks* with the privacy and security benefits of AWS PrivateLink.

ClickPipes using AWS PrivateLink and VPC Lattice for cross-VPC resource access

In order to get started using AWS MSK shared through AWS PrivateLink and VPC Lattice, you will need to do the following steps:

1. Create a Resource-Gateway to allow access into your VPC
2. Provide a Resource-Configuration to specify which resource should be shared
3. Share the Resource-Configuration with ClickHouse to use with ClickPipes

In order to share resources that are within your VPC, you will need to create a Resource-Gateway. The following command will create the Resource-Gateway.

aws vpc-lattice create-resource-gateway \ 
--vpc-identifier <VPC_ID> \
--subnet-ids <SUBNET_IDS> \ 
--security-group-ids <SG_IDs> \
--name <NAME> 

Before you can proceed, you will need to wait for the Resource-Gateway to enter into an “Active” state. You can check the state by running the following:

aws vpc-lattice get-resource-gateway \
--resource-gateway-identifier <RESOURCE_GATEWAY_ID> 

To share your resource, you’ll need to create a Resource-Configuration that will be used by the Resource-Gateway.

You can create a Resource-Configuration that either shares an IP-address or a domain-name that is publicly resolvable.

This simple setup will focus on supplying a single Resource-Configuration to share, if you would like to configure Group/Child Resource-Configurations, please refer to our documentation.

aws vpc-lattice get-resource-gateway \
--resource-gateway-identifier <RESOURCE_GATEWAY_ID> 

The output of the command will provide the “Resource-Configuration ARN”, you will use this in the next step and replace the <VPC_RESOURCE_CONFIGURATION_ARN> variable.

Sharing your resource requires the Resource-Configuration to be shared with ClickPipes, this is facilitated through the Resource Access Manager (RAM).

The following command will put the Resource-Configuration into the Resource-Share, enabling ClickHouse to access your resource when you have shared your Resource-Owner Account ID with us.

aws ram create-resource-share \
--principals 072088201116 \
--resource-arns "<VPC_RESOURCE_CONFIGURATION_ARN>" \
--name "<RESOURCE_SHARE_NAME>"

The ClickPipes team will require a resource owner account ID to be shared. You can find this within the AWS console or by running the following CLI command:

aws sts get-caller-identity \
--query Account \
--output text

If data source (e.g. Kafka broker) requires a private DNS setup , please share the host name(s) with ClickPipes team as part of your support request.

We’re excited to get our users set up with cross-VPC resource access. If you would like to start utilizing this feature with ClickPipes, please raise a support ticket through the ClickHouse Console with the Resource Owner Account ID.

* In order to connect PrivateLink Resource Endpoints to an on-premise resource, a Direct Connect or VPN connection is required