What is Orchestrate?

Orchestrate makes databases simple by powering full-text search, events, graph, and K/V storage behind a REST API.


Receive Blog Updates

Share:

Asa Miller

Explore the Marvel Universe with Orchestrate

cover

Marvel recently opened up an API for its character and comic data, which includes all of the characters, what comics they appear in, and a lot more. After poking around with it for a little while, I decided I wanted a more visual way to explore the characters and the comics, and that this would be a great project to build off of Orchestrate. We could use graph search to explore the relationship between characters and comics, as well as full text search to explore the data.

This project was pretty fast to build. It took a little time to import the data from the Marvel API and to do the design and frontend work, but the actual backend code and database work was really fast (less than a day or so). Orchestrate makes it very easy to make the social connections we use and to get the data in and out.

What we’re building

A node.js app for searching and viewing all the Marvel Comics data. Since Marvel’s API is rate limited, we’ll store all the data in Orchestrate and periodically update it from the API. It took some time to get the data from Marvel’s API and I won’t go over that part here, but if you want to download the cached data, I’ve included a dump of it in the repo.

View the code on GitHub

View the finished app here

Screen-Shot-2014-04-07-at-9.49.38-AM
Screen-Shot-2014-04-07-at-9.49.50-AM

The Tech

Starting off

First we’ll start off with a basic Express app. We’ll create a web.js file for our express routes and a functions.js file for our (yup you guessed it) app functions.

In the web.js we’ll start out with this:

var express = require('express');  
var exphbs  = require('express3-handlebars');  
var querystring = require('querystring');

// our functions
var f = require('./functions.js');

var app = express();  
app.use(express.logger());  
app.use(express.static(__dirname + '/public'));

// HANDLEBARS HELPERS
var hbs = exphbs.create({ defaultLayout: 'main' });  
app.engine('handlebars', hbs.engine);  
app.set('view engine', 'handlebars');

//  ROOT SEARCH INPUT PAGES
app.get('/', function(request, response) {  
    response.render('search-characters');
});

// START THE SERVER
var port = process.env.PORT || 5000;  
app.listen(port, function() {  
    console.log('Listening on ' + port);
});

Viewing a character

Let’s start off with viewing a specific character’s page. We’ll use the character id provided from the Marvel API as our key.

Add this to your web.js file so we can respond to /character/123456 requests.

// GET A SINGLE CHARACTER
app.get('/character/:id', function(request, response) {  
    var id = request.params.id;

    f.getCharacter(id)
    .then(function (data) {
        response.render('character', {
            data: data
        });
    })
    .fail(function(error){
        console.log(error);
        response.render('error');
    });
});

Then in the functions.js file, create a new function that will get a character’s data from Orchestrate.

exports.getCharacter = function (id) {  
    return db.get('characters', id)
    .then(function(results){
        return results.body;
    });
}

Getting data from Orchestrate is super easy. This will return the data stored at the key we pass in. But we want to get more than just the character data. We also want to see the comics that the character appears in. We can use Orchestrate’s Graph search for that. When I imported the data, I set up a graph relationship between the character and the comic. The relationship looks like this:

character > in > comic

The relationship is really simple to build because we have the character id and comic id from the Marvel API, and we can make the connection like this:

db.newGraphBuilder()  
.create()
.from('characters', characterID)
.related('in')
.to('comics', comicID)

Ok, so back to our app. Now we are getting the character data and we want to get the comics too.

We’ll create a new function that gets the comics.

function getComicsByCharacter (id, offset, limit) {  
    var limit = limit || 20;
    var end = offset + limit;

    return db.newGraphReader()
    .get()
    .from('characters', id)
    .related('in')
    .then(function (results) {
        // limit it to a max number of results
        return {
            total: results.body.results.length,
            results: results.body.results.slice(offset, end)
        };
    });
}

This is pretty straight forward. One thing to note, I’m manually limiting and paginating the data. Orchestrate’s Graph search will return all of the results and for popular characters, this can be a thousand comics.

Now we can add this call into our getCharacter function. We’ll use the Q module to handle the asynchronous calls.

exports.getCharacter = function (id, page) {  
    var out = {};
    var comicResults;

    // Get the character data
    var character = db.get('characters', id)
    .then(function(results){
        out = results.body;
    });

    // get the comics for this character
    var comics = getComicsByCharacter(id, page)
    .then(function (results) {
        comicResults = results;
    });

    // join the results and return them
    return Q.all([ character, comics ])
    .then(function () {
        out.comics = comicResults;
        return out;
    });
}

Here we get the character data and the comic data, then when both calls are done, we add the comic data to the character data and return it.

Screen-Shot-2014-04-07-at-10.35.03-AMScreen-Shot-2014-04-07-at-10.35.11-AM

Full Text Search

Now we can view a character, but how about searching? Orchestrate does full text search out of the box using Lucene. This is really powerful because we can search on specific parts of our data, such as “gender=female”, or do a wider search of anything that contains “Charles Xavier”.

First create a new endpoint for the search in the web.js file.

app.get('/characters/search', function(request, response) {  
    var query = request.query.s;

    f.getCharacters({
        query: query
    })
    .then(function (data) {
        response.render('characters', {
            data: data.body
        });
    })
    .fail(function(error){
        console.log(error);
        response.render('error');
    });
});

Then add a new function getCharacters()

// SEARCH CHARACTERS
exports.getCharacters = function (options) {  
    var options = options || {};

    // if no term is specified, get everything
    if (!options.query) options.query = '*';

    return db.newSearchBuilder()
    .collection('characters')
    .query(options.query);
}

This is the very basic version of search. It will take a passed query and return the results. But what if we want to do a more complex search. For example, what if we only want to search on the description field, or only find characters that exist in the Marvel Ultimate universe. To do that we will build a more complex Lucene query such as:

value.description: Wolverine AND value.wiki.universe:  "Ultimate Universe"  

We tell it to search on a specific field with value.FIELDNAME: SEARCH

Ian goes into more detail about Lucene here. It’s definitely worth the read.

Screen-Shot-2014-04-07-at-11.13.41-AM

Back to our query, our search form will pass a few parameter. Query, field (limit the query to a specific field), gender, and reality (which Marvel universe to search). We then will put that together into a search query:

var queries = [];

if (options.field) {  
    queries.push( 'value.' + options.field + ': ' + options.query );
} else {
    queries.push( options.query );
}

if (options.gender) {  
    queries.push( 'value.gender: "' + options.gender + '"' );
}

if (options.reality) {  
    queries.push( 'value.wiki.universe: ' + options.reality + '' );
}

return db.newSearchBuilder()  
.collection('characters')
.query(queries.join(' AND '));

With this we populate an array with each query term and then join them all with “AND”.

Screen-Shot-2014-04-07-at-11.29.44-AM

Comics

The process to get a comic or search through them is very similar. Take a look at the source code on GitHub to learn more about searching for comics.

Next steps

Now our app can view and search. We use Graph for connecting comics to characters, but what else could we do? What about seeing which characters appear in comics together, or seeing which villans have appeared in the most comics? There’s a lot more we could do with this data.

Have an idea? I’d love to hear it, or better yet, code it up and send a pull request!

 

Orchestrate gives developers access to multiple NoSQL databases through a single, RESTful API, eliminating the need to run databases in production. Learn more