AWS Glue DataBrew, how does it compare to the AI & Analytics Engine’s Smart Data Preparation? 

Data preparation. Love it or hate it, one thing is certain — it can chew up hours...

Blowing out project timelines and adding a little grey to your hair along the way. It is no surprise then, that Amazon Web Services (AWS) recently announced the release of their new data preparation tool, AWS Glue DataBrew. We wanted to provide a comparative look at this new option in the market and identify the similarities and differences with the AI & Analytics Engine’s Smart Data Preparation feature. 

So if you are looking for an AutoML tool to help claw back time from data preparation or data wrangling, read on...

First, a little background on both options. 

 

 

Smart Data Preparation (The AI & Analytics Engine)

Within the AI & Analytics Engine, Smart Data Preparation is a fully interactive feature that lets users prepare their data at scale in a flexible manner. 

The premise is that the user is guided by smart recommendations during the recipe-creation process. It offers a variety of “actions” (data transformation steps) through an action catalogue that offers the ability to fully customize and edit their data-preparation recipes. It covers the four stages of data preparation commonly required in analytics and machine learning tasks: 

1. Cleaning
2. Structuring
3. Enriching 
4. Feature engineering

Rather than a standalone tool, the Smart Data Preparation feature is a tightly integrated functionality within the end-to-end user journey on the AI & Analytics Engine platform. It sits between the data import and app creation phases of the journey in a unified graphical user interface (GUI).

                                        

3. Inspect the Statistical Profile of the Finalized Dataset

4. Create app (Target variable selection, train/test split)

5. Select features

6. Select models to train

 

7. Deploy trained models

 

Similarities

The main similarity between the two tools is that they both cater to the need for an easy-to-use no-code interactive data preparation tool. Let's take a closer look: 

Data Ingestion and Outcome

Both tools are targeted at tabular data files in the following formats: CSV, JSON (lines), and PARQUET.

While DataBrew allows the import of data mainly from data lakes on the AWS cloud such as S3, Redshift, and RDS — The Smart Data Preparation (AI & Analytics Engine) allows a diverse set of options for importing data, such as HTTP (URLs), SQL and NoSQL databases. Users can still upload data from cloud storage services such as S3 or GCS (google cloud) by getting a pre-signed URL for their dataset and using the HTTP option.

The outcome of the recipe-building process is a re-runnable recipe that can be re-used to transform a larger dataset of the same input schema.

Interactive Recipe Building and Recommender

Both AWS Glue DataBrew and the Smart Data Preparation feature have an interactive recipe-building user interface. There are many similarities between the two:

1. Quick preview of the dataset being prepared
2. The list of actions (steps) selected so far in the recipe
3. The ability to edit or delete a step in the recipe

Validation of Recipe

Whenever the list of actions in a recipe is modified, a validation check is applied to ensure that the recipe is legitimate. It consists of the following among many checks:

• • Input columns are available in the schema and are of the correct data types
  • The combination of parameters is valid
  • Names of new columns output by an action are not colliding with existing column names

 

Advanced column selectors

Our Smart Data Preparation offers the ability for users to apply a transformation on multiple columns with a single action. To aid these, our graphical user interface and API provides the following modes of including/excluding input columns, column groups, or both into the selection:

• • By Name: explicitly list out the name
  • By Type: select by schema type
  • By Pattern: select by match in regex pattern of the name

The user is also allowed to combine multiple such criteria with the and/or operator. This enables full flexibility to let the user specify complex selection criteria such as “columns matching the name pattern ‘x_.*’, excluding non-numeric columns.”

Queuing and Committing actions to a recipe

In The AI & Analytics Engine, whenever an action is added, it is “queued” to be committed to the recipe. Like in DataBrew, the queued actions are run on a fixed-size sample (first 5k rows) of the full dataset and the result is displayed as a preview. This serves as visual feedback to users, allowing them to change the configuration of their action to ensure that the preview represents what they desire as the result.

Our platform also provides an additional functionality enabled for the user, called the “committing” of the queued actions to a recipe, before continuing to edit the recipe further. Committing of actions signals the platform to run the recipe actions on the full dataset (rather than on the sample) and then show a renewed sample preview. This results in a more accurate data preview, where the sample was first generated from the raw data before the recipe actions were applied.

Committing actions to a recipe also provides the platform to run intelligent algorithms with the fully processed data to provide users with good recommendations for the next set of actions in their recipe.

Recommendations

In DataBrew, recommendations are generated on a per-column basis, and are not available by default. The user needs to click on a particular column and request recommendations corresponding to one column.

The AI & Analytics Engine’s Smart Data Preparation also provides recommendations of the next set of actions likely to be helpful to the user. 

The key differences are that: 

1. These recommendations are generated automatically the first time the user starts a recipe and after every commit. 
   
   
2. The recommendations are based on a sample of the entire dataset rather than a single column. This makes it quite attractive for users who want to detect, for example, input columns with too little correlation with the target variable that needs to be dropped.
   
   
3. Every recommendation is accompanied by reasons why these actions are useful, showing charts and summary statistics to help the user understand and scrutinize their data. This is particularly useful for users without data science expertise who want to access the benefits of AI/ML.