Dataflow Automation Common Questions

What's covered:

• What is Dataflow Automation?
• What is a record?
• How is a record structured?

What is Dataflow Automation?

Dataflow Automation provides organizations Big Data capabilities including long term storage, data management, processing and integration without having to invest millions of dollars and a dedicated team.

Click here for more information.

What is a record?

Within Dataflow Automation, a single set of data is called a record. Think of a spreadsheet. In that instance, a record is a single row. The same principle applies here.

How is a record structured?

We use the JavaScript Object Notation (JSON) for the following reasons:

• Light syntax
• More flexible than tables
• Integrates with JavaScript
• Easily modified

At first JSON structured records can look intimidating, but after learning four design principles you'll be fluent in JSON.

Key-Value Pairs

At the heart of JSON is the idea that data (the value) is identified by a name (the key). Think of a dictionary where the word is the key and its definition is the value. In a JSON, keys are separated from values by a colon (:) and key-value pairs are separated from one another by a comma (,).

Objects

Now that we know a data value is organized as a key-value pair, we need to learn how a record starts and ends. In JSON, a set of key-value pairs is called an object. The beginning and end of an object is identified by curly braces {}.  

We're now prepared to create our first JSON record. The record will be two key-value pairs for first name and last name.

{ "first_name": "Michael", "last_name": "Phelps" }

But how do we identify the object?  We can use JSON for this, and since spacing does not affect the structure, let's spread out the text to make the record easier to read.

record: {
    "first_name": "Michael",
    "last_name": "Phelps"
}

Arrays

An important characteristic of key-value pairs inside a JSON object is that there is no explicit order. Regardless if first_name or last_name comes first, the object is effectively the same. For lists that require a specific order, we use arrays which start and end with square brackets [].

We would use an array to create a JSON object that holds the gold, silver, and medal winners.

medalists: ["Michael Phelps","Kosuke Hagino","Wang Shun"]

Array of Objects

So far so good? The final JSON principle is combining arrays, objects, and keys to create arrays of objects. Let's expand our swimming array from above to illustrate this.

medalists: [{
        "first_name": "Michael",
        "last_name": "Phelps"
    },
    {
        "first_name": "Kosuke",
        "last_name": "Hagino"
    },
    {
        "first_name": "Wang",
       "last_name": "Shun"
    }]

This may be confusing at first, so let's dissect what we have. First, we can identify the object we made at the beginning with first_name and last_name keys. Now that we understand that, we can condense those sections.

medalists: [{},{},{}]

That's more manageable! The above JSON looks very similar to our array example, but instead of a string like "Michael Phelps", we have objects as each value in the array list. The process of using one data structure inside another is called nesting. The ability to nest objects within one another allows JSON to better model records which are not flat.