Sneller Cloud Onboarding

Prerequisites

To onboard to Sneller Cloud, you will need to have a Sneller Cloud account. We will create that in the next step. You also need to have Terraform installed which is a tool to automate infrastructure.

All examples are written for Linux and should also work on MacOS. All examples have also been tested in WSL2 (Windows Subsystem for Linux).

Register with Sneller Cloud

You have to register on https://console.sneller.ai/ in order to use Sneller Cloud. You can do so either with a Google account or a GitHub handle.

After completing the registration, the onboarding process starts. You will receive an email called “Welcome to Sneller” that contains a (bearer) token that you should provide for the next step:

export TF_VAR_sneller_token=YOUR_TOKEN

Setup Terraform

Installing Terraform is straightforward since it is a single executable. Follow the Install Terraform instructions to download it.

Now it is time to setup Terraform. It uses both the AWS provider and the Sneller provider. The AWS provider uses the current user’s AWS credentials, so make sure you have sufficient rights and are logged in to the proper account. You can verify your AWS user and account ID using¹:

aws sts get-caller-identity

You can download the Terraform scripts as follows:

git clone https://github.com/snellerinc/examples 
cd examples/terraform/cloud-onboarding

In a nutshell these are the actions that Terraform will perform (see here for full details):

• create both a source and an ingestion bucket (in your own AWS account)
• create a Sneller IAM role allowing Sneller to access the S3 source and ingestion buckets
• create a test table called tutorial/table1 with some sample data
• setup S3 event notifications

Apply Terraform

Simply execute the commands below in order to complete the integration between your AWS account and Sneller Cloud:

terraform init  # needed only once
terraform apply

After a successful run, you will see the following output:

...
 ...
  ...
Apply complete! Resources: 16 added, 0 changed, 0 destroyed.

Outputs:

database = "tutorial"
sneller_endpoint = "https://snellerd-production.us-east-1.sneller.ai"
sneller_ingest = "abcd-sneller-ingest"
sneller_source = "abcd-sneller-source"
sneller_token = "SA0M1ABCDEFGHIJKLMNOPQRSTUVWXYZABCDEFGHI"
table = "table1"

Run a test query

Sneller will automatically pick up table definitions and ingest all data for the test table. We can now run some queries. First we’ll setup some environment variables to ensure that we use the correct values:

export SNELLER_DATABASE=$(terraform output -json database | jq -r '.')
export SNELLER_TABLE=$(terraform output -json table | jq -r '.')
export SNELLER_ENDPOINT=$(terraform output -json sneller_endpoint | jq -r '.')
export SNELLER_TOKEN=$(terraform output -json sneller_token | jq -r '.')

Now run the following command to determine the number of items for the test table.

curl -H "Authorization: Bearer $SNELLER_TOKEN" \
     -H "Accept: application/json" \
     -s "$SNELLER_ENDPOINT/query?database=$SNELLER_DATABASE" \
     --data-raw "SELECT COUNT(*) FROM $SNELLER_TABLE"

Further reading

See the tutorial for some further examples such as ingesting more data or creating a new table from scratch. In addition there is a more elaborate example for adding AWS CloudTrail as a new table.

You can also find some more examples here for S3 data sources that are easy to ingest.

For full details on the Terraform scripts used in this onboarding introduction, please read the detailed explanation.