Introduction

Snowflake is a purpose-built SQL cloud data warehouse provided as Software-as-a-Service (SaaS). Snowflake provides a data warehouse that is faster, easier to use, and far more flexible than traditional data warehouse offerings. With Snowflake’s unique and patented architecture, it’s easy to amass all of your data, enable rapid analytics, and derive data-driven insights for all of your users.

MoEngage <> Snowflake

MoEngage takes advantage of Snowflake’s Secure Data Sharing. Secure Data Sharing lets you share selected objects in a database in your account with other Snowflake accounts.

With Secure Data Sharing, no actual data is copied or transferred between accounts. All sharing uses Snowflake’s services layer and metadata store. Shared data does not take up any storage in a consumer account and therefore does not contribute to the consumer’s monthly data storage charges. The only costs to consumers are for the compute resources (i.e. virtual warehouses) used to query the shared data.

Benefits 

Reduce integration time 

• No more searching for the right ETL tool as MoEngage directly integrates with MoEngage.
• Long and complicated ETL pipelines are now replaced with a single-step Secure Data Sharing that gives MoEngage direct access to your data without actual data flow.
• This decreases the dependency on tech teams significantly.

Faster data processing

• The power of the Snowflake infrastructure enables us to store, process, and query massive amounts of data in near real-time.
• Any changes in the original schema are propagated immediately without having to change any configuration on MoEngage’s end which is a significant advantage over the traditional ETL pipelines.
• Since there is no need for ETL tools and external cloud providers, the cost of import via Secure Data Sharing is significantly lower than traditional data pipelines.

Integration use-cases

1. Keep your users in MoEngage up-to-date with the user data in your Snowflake account.
2. Import event data from your Snowflake account directly into MoEngage.
3. Import users into MoEngage and retarget them via custom segments.
4. Export event data from MoEngage into your Snowflake account for performing data analysis and gaining unique insights.

Integration

Scope

The integration allows you to

1. Import your users and events from Snowflake into MoEngage for segmentation and hyper personalised campaigns 

2. Export your campaign interaction data back to Snowflake  

Architecture diagram 

1. Import from Snowflake into MoEngage 

Step 1: Create a share on Snowflake 

Using Snowflake’s built-in roles and permissions capabilities, access to data shared from MoEngage can be controlled and governed using the access controls already in place for your Snowflake account and the data therein. Access can be restricted and monitored the same way as your own data.

Once the data share is provisioned, MoEngage will create a database from the incoming data share, making all the tables shared appear in our Snowflake instance and be queryable just like any other data you’re storing in your instance. 

Step 2: Import shared data into MoEngage 

On your MoEngage dashboard, navigate to Segment >> Imports. 

Select your snowflake data 

1. Click + Import on top right. Select users/events depending on your requirement
2. Select Snowflake as your source.
3. Fill in the following details - 
   • Import name - Name you would like to use to identify this import 
   • Import source - 
     • Snowflake account - Name of your snowflake account 
     • Snowflake share name - Name of the snowflake share that you had shared with MoEngage
4. Click Fetch Share. 
5. Select your database and table from the dropdown. 
   Note- You can also click on Preview to view top rows in your selected table. This will help you to ensure that you have selected the right data.
6. Click Next

Map your snowflake data to MoEngage Attributes

In this step, you need to map the snowflake data to the attributes present in MoEngage. All your columns will be shown one below the other:

1. Column name - This is the column name to be mapped. Below the column name, we also show a sample value (picked from the top row) for your reference.
2. Map attribute - Here you need to pick which MoEngage attribute you want to map. You can also choose to create a new attribute. Some attributes support ingestion from multiple data types, so you need to pick the data type of the column as well. For the "DateTime" columns, you also need to pick the format.
3. Actions - You can optionally choose to skip the column. The skipped column will not be imported.

Depending on the type of import, there are a few mandatory mappings required:

1. User imports - You need to select the column having:
   • Registered - User ID - Your data should mandatorily have a column that contains a unique user identifier (that identifies a user’s account in your system).
   • Anonymous - Anonymous ID - You need to mark a column (email, mobile number, etc.) as an identifier of these users.
2. Event imports - You need to select the column having:
   • User ID - This column will be used to match the users in MoEngage to your snowflake data.
   • Event Time - You need to map either Event Time (UTC) or Event Time (App Timezone). If you have chosen Event Time (UTC), then the "Event Time" of the imported event will be converted to the timezone chosen in your MoEngage Dashboard settings:

Once a mandatory mapping is marked, it will reflect against the column name in the mapping table and you will no longer be able to mark the column as skippable. 

We do have some additional optional capabilities like -

• Just like new events, you can also create a new attribute from the dropdown by clicking on "+ Create attribute":
• Manifest Files - Optionally, you can choose to auto-map these columns by uploading a manifest file. To upload a manifest file, click the "Upload mapping file" option on the top-right of the mapping table:

• Import Notifications - Choose to get notified about your imports' statuses via email. Select up to 10 emails to send the status emails to

      Once done, click Next. 

In this step, you can define when to import data. We support the following types of imports:

1. One-Time Imports: You can either choose to run the import as soon as possible or at a later date and time (scheduled)
2. Periodic: You can choose to run your imports daily, weekly or monthly, or with intervals and advanced configurations.

Click Done. Your snowflake import is now set up. And you will start seeing the Snowflake data in MoEngage depending on your import frequency.  

 

2. Export campaign interaction events to Snowflake  

On your MoEngage dashboard, navigate to App Marketplace >> Snowflake.

1. Click Integrate >> Add Integration 
2. Enter your Snowflake account details 
   • User - The username for a user with write access to your Snowflake database
   • Password - The password for the user.
   • Account - The account id of your cluster, not the url (for example, url: my-business.snowflakecomputing.com, account-id: my-business. 
   • Database - The database name 
3. Select the events that you want to export to Snowflake 
4. Click connect. 

Data would now start reflecting on your snowflake account in 15 minutes.