UPDATE September 28, 2020 – Swarm64 DA no longer enables FPGA-acceleration of PostgreSQL query performance on AWS F1 instances. Why not? Because we developed a new version of Swarm64 DA that accelerates PostgreSQL just as fast, but without requiring specialized (FPGA) hardware. Same fast performance…but much lower hardware cost.
If you would like to avoid the cost of cloud data warehouses by using free, open source Postgres for analytic or reporting-intensive projects, then consider running Postgres on an FPGA-accelerated AWS F1 instance.
FPGA-equipped servers can speed up Postgres queries & ingestion by 10x -50x or more (depending on use case) and are available on nearly all cloud platforms.
In this blog post, we’ll explain how to provision an FPGA-accelerated Postgres instance on Amazon by launching Swarm64 from the AWS Marketplace. It only takes about 5 minutes to get your database up and running.
What is FPGA and how does it help Postgres?
Field programmable gate arrays (FPGAs) are integrated circuits that are loaded with custom firmware dynamically at runtime (an “AFI,” in Amazon parlance, which stands for Amazon FPGA Image). The software on the FPGA runs at firmware speed and adds extra processing power to your server.
FPGA adds parallel processing to Postgres
Swarm64 DA extends Postgres with an AFI that executes over 100 SQL reader and writer processes that work in parallel on the FPGA to speed up Postgres queries and data insertion. It’s a bit like adding 100 new cores to boost parallel processing on your server.
Good fit for analytics and data warehousing
The best Swarm64 FPGA acceleration use cases for Postgres tend to be query intensive; analytics, data science/modeling, time series, data warehousing and reporting are popular targets…especially with more complex queries or many concurrent users.
AWS EC2 F1 instances
Amazon EC2 F1 instances are equipped with FPGA processors to enable hardware acceleration solutions. They are ideal for computing problems that require high bandwidth, enhanced networking, and very high compute capabilities such as database management and analytics.
There are 3 F1 instance types:
- f1.2xlarge – 1 FPGA, 8 vCPUs
- f1.4xlarge – 2 FPGA, 16 vCPUs
- f1.16xlarge – 8 FPGA, 64 vCPUs
Swarm64 only runs on 4xl and 16xl; there’s not enough compute power in the 2xl to make much of a difference to Postgres. The 4xl provides decent acceleration for Postgres at a moderate instance cost. The 16xl is quite a bit more expensive, but allows Swarm64 to accelerate Postgres to a much higher level, and at a cost that is well below that of most data warehouse systems.
If you’d like to try FPGA-accelerated Postgres on an AWS EC2 F1 instance, just follow the steps below. You’ll have a live, accelerated Postgres instance ready to use within 5 minutes.
There’s no charge for the Swarm64 software for the first 7 days, but there is a fee that Amazon charges for the EC2 F1 instance. If you’d like Swarm64 to cover the AWS cost of your F1 instance during your trial period, please contact us.
Step 1 – Find Swarm64 in the AWS Marketplace
On the Swarm64 marketplace page, click the
"Continue to Subscribe" button.
Next, you’ll see a terms and conditions page. Click
"Continue to Configuration" after you finishing reviewing the terms and conditions.
Step 2 – Configure the software
On the software configuration page, you’ll want to keep all the defaults, except maybe for the
Region, depending on where you’re located.
Note: the list of Regions here is limited to only the Amazon regions in which F1 instances are available. Choose the AWS region in which you want the instance deployed.
"Continue to Launch" to launch the instance.
Step3 – Launch the AWS F1 instance
On the next page, Amazon gives you a number of launch options. We recommend that you choose
"Launch through EC2" because that gives you more storage configuration options later in the process.
“Launch through EC2,” click the
Choose an F1 instance type
Next, you’ll see your EC2 Console page. The first thing you have to do is choose an instance type for your new Postgres server. Since Swarm64 runs on FPGA (F1) instances, you can make this step easier by using the drop-down list above the table to show only
As mentioned above, Swarm64 accelerates Postgres on either f1.4xlarge or f1.16xlarge instances, so you must choose one of those.
“Next: Configure Instance Details.”
On the Configure Instance Details page, there are some more instance details to configure. You can accept the defaults here and just click
“Next: Add Storage.”
On the Add Storage page, you can set the amount of storage provisioned for your database (the default is 200GB). Other storage items on this page include the root volume for the operating system (mounted on
/dev/sda1), the ephemeral disks, which are the instance-local SSDs (mounted as NVMe’s) used for caching. Note: do not to change the EBS mount point because Swarm64 relies on the storage device being found at
“Next: Add Tags” after you finish changing your storage settings.
On the Add Tags page, you can enter a meaningful name to identify the instance.
After entering a tag, you click
“Next: Configure Security Group.”
In the final configuration step, you can select a security group which specifies the IP addresses allowed to access your FPGA-accelerated Postgres instance. We recommend using a security group that is already defined for your AWS account (e.g., a policy allowing access from IP addresses at your company) or create a new security policy here.
“Review and Launch.”
On the final page, you’ll see an overview of the instance ready for launch, with all the settings you’ve configured. You can go ahead and just click
Launch, you will be prompted for a private and public key pair, which is used to authenticate SSH access to the instance. Then click
After you launch the instance, Swarm64 takes another one to two minutes to install and start your Postgres database and load the Swarm64 acceleration software onto the FPGA(s). Now you’re ready to experience FPGA-accelerated Postgres performance. Simply SSH into your F1 instance and login to Postgres.
Your Postgres instance is up and running. What next?
The first thing you need to know is that you are using standard Postgres 11.6, and it works like Postgres normally does.
Secondly, you can find Swarm64 product documentation here.
The third important thing to know is that the Swarm64 FPGA acceleration software operates on data stored in special, columnar-indexed Postgres Swarm64 foreign tables. For example:
CREATE FOREIGN TABLE t0(
col0 INT NOT NULL,
Swarm64 integrates with the Postgres query engine. When SQL statements arrive at the Postgres server seeking to query or update table
t0 (following the example above), Swarm64 plans a query that is optimized for parallel processing of columnar-indexed data, and then executes that on the FPGA.
Run the TPC-H benchmark on your Postgres instance
The Postgres server you just provisioned includes a TPC-H benchmarking kit. The TPC-H kit is an easy way to load benchmarking data at any scale if you don’t have your own data to experiment with.
./install_tpc_toolkit.sh as explained in the “Swarm64 DA Evaluation” section of the product docs.
Getting help and more free stuff…
Lastly, please contact us here if you would like to do any of the following:
- Get help with your installation
- Request an extension to your free 7-day Swarm64 trial
- Ask for free AWS credits to cover the cost of the AWS F1 instance on which Swarm64 runs.
We hope you enjoy your Swarm64 experience on Amazon! Let us know what you think.