Welcome to Sneller!

Getting started

There are three easy ways to get started with Sneller:

• Getting started
  • Open source
  • Playground
  • Sneller Cloud
  • Curl access
  • Code examples

To send a query to Sneller, you can either use curl or check out these example programs (using just plain HTTP calls).

Open source

If you have an AVX-512-capable CPU, you can download the Sneller source code and run the query engine locally. The Sneller query engine is available under the AGPL-3.0 license. Our README describes how to get started with the sdb tool, which you can install easily via go install:

$ go install github.com/SnellerInc/sneller/cmd/sdb@latest

Note that an AVX-512-capable CPU is required in order to run queries. If you don’t have the appropriate hardware to run sdb locally, you should try the Playground or Sneller Cloud instead.

Playground

The playground hosted on this website is the quickest way to try out Sneller.

Our playground has a pre-loaded table containing one billion records (originally about 5TB worth of JSON) from the GitHub archive data set. Users can query this table for free in order to test out Sneller’s functionality.

In addition, it is easy to experiment with your own data by either:

• uploading one or more files of JSON data
• copy-pasting one or more presigned S3 URLs

Once your data is uploaded, a temporary table is created, and you can query either directly using the playground or via curl or a simple sample program.

Important note: any data that is uploaded will be automatically deleted after 2 days. Do not use the playground to store sensitive or confidential data.

Sneller Cloud

Sneller Cloud is designed for production use at scale and is capable of handling petabyte-scale tables with blistering query performance of many TBs per second.

Sneller Cloud is meant to be used in combination with S3 buckets hosted in your own AWS account. Sneller will not store a single byte of your data outside of your account; all your data will remain under your own control at all times. We achieve this by establishing a trust relationship between your AWS account and an IAM role within Sneller’s AWS account as part of the registration process.

Sneller Cloud provides automatic synchronization between your source data and your SQL tables with low latency. There is no need to run and manage any batch processes yourself in order to keep your tables up-to-date.

You can onboard to Sneller Cloud in less than a minute (using Terraform) and follow a short tutorial to ingest some data and run some test queries.

More extensive documentation of Sneller Cloud is available:

• Sneller Endpoint Reference: REST API description of Sneller Cloud
• Table Examples: examples of table definitions for eg. CloudTrail or VPC Flow Logs
• Table Definition: reference documentation for table definitions and definition.json objects
• Partitioning: reference documentation for partitioning tables along S3 prefixes

Curl access

You can invoke Sneller simply via curl, for example:

curl -H 'accept: application/json' 'https://play.sneller.ai/query?database=demo' \
  --data-raw $'SELECT DISTINCT repo.name FROM gha WHERE repo.name LIKE \'torvalds/%\'' | jq

[
  {
    "name": "torvalds/linux"
  },
  {
    "name": "torvalds/libdc-for-dirk"
  },
  {
    "name": "torvalds/uemacs"
  },
  {
    "name": "torvalds/test-tlb"
  },
  {
    "name": "torvalds/pesconvert"
  },
  {
    "name": "torvalds/subsurface-for-dirk"
  }
]

The playground has a “curl” button that generates the command automatically. Sneller Cloud endpoints require the Authorization HTTP header to contain a valid bearer token, but the play.sneller.ai endpoint does not.

Code examples

Sneller does not require the use of any SDK (Software Development Kit). As illustrated in the examples below, you can execute queries with a simple HTTP POST request from the HTTP client of your choice. Every example program below sends a SQL query to the ‘gha’ table of the ‘demo’ database which is running in Sneller’s playground.

curl -H 'accept: application/json' 'https://play.sneller.ai/query?database=demo' \
  --data-raw $'SELECT DISTINCT repo.name FROM gha WHERE repo.name LIKE \'torvalds/%\''

// Try: go run example.go
package main

import (
	"fmt"
	"io"
	"log"
	"net/http"
	"strings"
)

const (
	endpoint = "https://play.sneller.ai"
	database = "demo"
	table    = "gha"
	url      = endpoint + "/query?database=" + database
	sql      = "SELECT DISTINCT repo.name FROM " + table + " WHERE repo.name LIKE 'torvalds/%'"
)

func main() {
	// We'll be doing a POST with the SQL query as the body
	req, err := http.NewRequest(http.MethodPost, url, strings.NewReader(sql))
	if err != nil {
		log.Fatalln(err)
	}
	req.Header.Set("Authorization", fmt.Sprintf("Bearer %s", "YOUR_TOKEN")) // not needed for playground
	req.Header.Set("Accept", "application/json")                            // get JSON as result

	client := &http.Client{} // create http client
	resp, err := client.Do(req)
	if err != nil {
		log.Fatalln(err)
	}
	defer resp.Body.Close()
	body, _ := io.ReadAll(resp.Body)

	if resp.StatusCode < 200 || resp.StatusCode >= 300 { // any errors?
		log.Fatal(fmt.Errorf("error (%s): %s", resp.Status, string(body)))
	}
	fmt.Println(string(body)) // success
}

using System.Net.Http.Headers;
using System.Web;

const string endpoint = "https://play.sneller.ai/query";
const string database = "demo";
const string table = "gha";
const string sql = "SELECT DISTINCT repo.name FROM " + table + " WHERE repo.name LIKE 'torvalds/%'";

var uri = new UriBuilder(endpoint);
var query = HttpUtility.ParseQueryString(string.Empty);
query.Add("database", database);
uri.Query = query.ToString();

var req = new HttpRequestMessage
{
    RequestUri = uri.Uri,
    Method = HttpMethod.Post,
    Content = new StringContent(sql),
    Headers =
    {
        Accept = { new MediaTypeWithQualityHeaderValue("application/json") },
        // not needed for playground
        Authorization = new AuthenticationHeaderValue("Bearer", "YOUR_TOKEN")
    }
};

using var httpClient = new HttpClient();
var response = await httpClient.SendAsync(req);
response.EnsureSuccessStatusCode();
var body = await response.Content.ReadAsStreamAsync();
await body.CopyToAsync(Console.OpenStandardOutput());

// Try: node example.js
const https = require('node:https');

const table = 'gha';
const query = `SELECT DISTINCT repo.name FROM ${table} WHERE repo.name LIKE 'torvalds/%'`;

const reqOptions = {
    hostname: 'play.sneller.ai',
    method: 'POST',
    path: '/query?database=demo',
    headers: {
        'Accept': 'application/json',
        'Authorization': 'TOKEN_HERE',  // not needed for playground
      },
}

const req = https.request(reqOptions, (res) => {
    if (res.statusCode < 200 || res.statusCode >= 300) {
        process.stderr.write(`Error ${res.statusCode}`);
        return;
    }
    res.on('data', (d) => {
        process.stdout.write(d);
    })
});
req.on('error', (e) => {
    process.stderr.write(`Error: ${e}`);
})
req.write(query);
req.end();

The example program above should output the following JSON data:

[{"name": "torvalds/linux"},{"name": "torvalds/libdc-for-dirk"},{"name": "torvalds/uemacs"},{"name": "torvalds/test-tlb"},{"name": "torvalds/pesconvert"},{"name": "torvalds/subsurface-for-dirk"}]

These examples also work with Sneller Cloud except that you need to provide a valid bearer token and change the endpoint to point to Sneller Cloud.