What is Orchestrate?

Orchestrate makes databases simple by powering full-text search, events, graph, and K/V storage behind a REST API.


Receive Blog Updates

Share:

Adron Hall

Test Data Builder Symphonize.js With Chance.js (1/3)

Today I’m kicking off a series of articles about building solutions with specific data elements or structures. In this first article, I’m going to work through some basic sample data generation. In subsequent articles, I’ll be using the data and generation of data to outline some other domain spaces and related solutions. The key data structures that I’ll generate from the code here will be key value and graph. Over the course of the series, I may also add more data generation, but for now, here’s what I went through to build out some sample data.

Key/Value Basics

Key/Value data, sometimes referred to as dictionary or hash data types, is made up of two elements. One element is the key, often a globally unique identifier, which is used to lookup the data. The other element is the value, which can be anything like binary or text. Orchestrate.io requires the key to be URL safe and the value to be JSON.

Graph Basics

Graph data structures can be complicated to explain, so I’ll briefly touch on them. A graph structure is a finite, sometimes mutable set of pairs. These pairs have a connecting element that is called an “edge”. The edge defines the relationship between the pair. Often the edge contains user data, which serves as metadata about the pairing. For a deeper dive read about the abstract graph data typegraph mathematicsor on graph traversal.

Orchestrate.io makes graph data structures easy by providing a simple RESTful API that allows you to define edges (aka “relations”) between Key/Value pairs. This can be done within a collection or between Key/Value objects in separate collections.

Now that we’ve covered the basics, let’s get to coding up something that can help us generate all of this data!

Orchestrate.io, Chance.js, and Symphonize

Start a basic NPM library project, with an appropriate bin and lib directory as outlined in “How to Build an NPM Package, Beginning the Symphonize Project.” For the remainder of this blog I’ll be using the symphonies project as a base, starting from this commit.

In the packages.json file I’ve added the orchestrate npm library by Steve Kaliski (@stevekaliski). This we’ll use for storing the data in Orchestrate.io a little later on. The other library to add is the chance library, which generates basic random data, perfect for our library use case. It’s located at chance.js on github, with a full docs site and of course the NPM page.

    {
      "name": "symphonize",
      "version": "0.0.1",
      "description": "Spits out all sorts of random generic data of various sorts. Check README.md for more information!",
      "main": "./lib/main",
      "bin": {
        "replaceme": "./bin/symphonize"
      },
      "dependencies": {
        "orchestrate": "~0.0.1",
        "chance": "~0.5.2"
      },
      "devDependencies": {},
      "scripts": {
        "test": "echo "Error: no test specified" && exit 1"
      },
      "engines": {
        "node": "*"
      },
      "repository": {
        "type": "git",
        "url": "git@github.com:Adron/symphonize.git"
      },
      "keywords": [
        "orchestrate.io",
        "orchestrate",
        "key",
        "value",
        "graph"
      ],
      "author": "Adron Hall",
      "license": "Apache 2.0",
      "bugs": {
        "url": "https://github.com/Adron/symphonize/issues"
      }
    }

I alternatively could have added the dependency for the npm orchestrate library by using the npm command line to install it, and then have npm add it to the packages.json file with the --save setting.

npm install orchestrate --save

npm install chance --save

Now that we have the Orchestrate library all set, let’s get some sample data generator functions put together for export in an NPM module.

    var Chance = require('chance');

    exports.generateContent = function generateContent(objectCount, dataAbstraction) {
        var content = new Array();
        var chance = new Chance();

        for (var i = 0; i < objectCount; i++) {
            var iOSidentifier = new Object();
            iOSidentifier.guid = chance.guid();
            iOSidentifier.first = chance.first();
            iOSidentifier.last = chance.last();
            iOSidentifier.cell = chance.phone();
            iOSidentifier.work = chance.phone();
            iOSidentifier.birthday = chance.birthday();
            iOSidentifier.gender = chance.gender();
            iOSidentifier.long = chance.longitude();
            iOSidentifier.lat = chance.latitude();
            iOSidentifier.CFUUID = chance.guid();

            var identifier = JSON.stringify(iOSidentifier);

            var keyValue = new Object();
            keyValue.key = chance.guid();
            keyValue.value = identifier;

            content.push(keyValue);
        }
        return content;
    }

    exports.generateFileDump = function (fileName, objectCount) {
        var fs = require('fs');

        if(fileName === null || fileName === ''){
            fileName = 'dumpfile.json';
        }

        if(objectCount === null || objectCount < 1){
            objectCount = 4;
        }

        fs.writeFile(fileName, this.generateContent(objectCount), function (err) {
            if (err) return console.log(err);
            console.log('File written.');
        });
    }

    exports.writeIoOrchestrate = function(dataAbstraction){
        // Write data to orchestrate.io based on appropriate key value or graph storage.

        if(dataAbstraction === 'keyvalue'){
            createKeyValueData();
        }

        if(dataAbstraction === 'graph'){
            createGraphData();
        }
    };

    console.log(this.generateContent(10, 'keyvalue'));

Summary

This is the first run at some code to generate data. The code is broken out into some stubs for the write to Orchestrate while I test out some good ways to generate the data, and what data to generate. So what I’ve written above is broken out as:

  • writeIoOrchestrate – I set this up as a stub point for were the data will eventually be written, directly to the Orchestrate.io account of your choice, once I get a decent finalized data set being generated.
  • generateFileDump – This is a straight dump of the data to a file. I’ve got it in here to use for testing, and as another option when working locally, to get all of the data in a single place.
  • generateContent – This function is where the magic will be done. Eventually I’ll add more to this, but this is a good starting point in which to work from.

Next, I’ll dive into generating a full data set, and start discussing some of the options for how that data can be used with the orchestrate.js library.

@adron