What is Orchestrate?

Orchestrate makes databases simple by powering full-text search, events, graph, and K/V storage behind a REST API.


Receive Blog Updates

Share:

Max Thayer

Resolving Conflicts in Orchestrate with Hextor

06.19.2014
boxing

Databases don’t always know what you want. As engineers, we do our best to build resilient, scalable systems that address your needs with intuitive interfaces, but sometimes, we don’t know how to handle what we’re given. The system does its best, but leaves behind a conflict so you can clarify how you want to handle this situation.

Many databases, including Orchestrate and CouchDB, generate conflictss when they don’t know how to handle something. Conflicts contain all the information you as a developer would need to resolve them how you see fit, such as what happened, and what documents were affected. Then you can fix them up to your heart’s content.

So, in Orchestrate, how do conflicts happen?

Say you’re writing an application that saves names, like this:

{
    names: ['Max Thayer', 'Mx. Maximiline', 'M. Thayer']
}

But one of your developers hasn’t dismantled the falsehoods programmers believe about names, so they insert a document like this:

{
    names: 'Falco, Antony'
}

But when they try to insert it, they get a 409 -- indexing_conflict. The document is saved, but it doesn’t show up in searches. What gives?

N.B.: This post gets into the nitty-gritty of Orchestrate as a piece of technology. We recognize this implementation falls short of our goal of effortlessness, and we’re working on it. In the meantime, we want to make clear how that implementation affects you, and how you can keep it painless for you.

Conflicts

In Orchestrate, conflicts occur when the database doesn’t know how to handle something. Specifically, it doesn’t know how you want a new document to be searched. Orchestrate does its best, but leaves behind a conflict that you can resolve at your leisure.

Whenever you insert a document, we analyze its contents to determine how to index it for searches. Based on the type of each field in the document, we create a mapping. When you insert more documents, we refer to this mapping for how to index fields, or update it when we encounter fields we haven’t seen yet in your data.

But when a document has data in a known field of an unexpected type, Orchestrate doesn’t know how to index it. Because it’s of a different type, we can’t index it along previous values, but the document is technically valid, so we don’t want to just reject it. So, we create a conflict detailing the difference between what we expected and what we saw, so you can resolve it as you please.

Hextor does this automatically.

Hextor

Hextor is a daemon that resolves conflicts. It watches your Orchestrate collection for conflicts, and whenever it sees them, executes your resolution strategy.

Documents with conflicts in them show up in searches with a field named conflicts with the value true, so a search for conflicts:true returns only documents with conflicts in them. While that search returns results, Hextor resolves them. If it doesn’t, it waits a while, and searches again.

Hextor has a default strategy that resolves conflicts by changing field names, but if that’s not what you want, you can implement a different strategy. Let’s start:

sudo npm install -g hextor
export ORCHESTRATE_API_KEY=...
export ORCHESTRATE_COLLECTION=...
hextor
# Now watching [collection] for conflicts...

That’s it! Now, whenever your collection sees a conflict, Hextor will resolve it, like a proper God of Conflict.

Hextor’s default strategy resolves conflicts by changing field names. If Orchestrate expects a string but sees an object, it will change the field’s name to [fieldname]_object, prompting Orchestrate to attempt to re-index it. This might not be perfect for your scenario, and that’s fine. Pop open defaults.js in Hextor’s source code and modify the strategy to your liking, like this:

git clone git@github.com:orchestrate-io/hextor.git
cd hextor
npm install
export ORCHESTRATE_API_KEY=...
export ORCHESTRATE_COLLECTION=...
# open up `defaults.js` and modify the default strategy
npm start

Now you’re resolving conflicts using your custom strategy. Way to go!

You can also deploy Hextor to Heroku, to autonomously resolve conflicts. To do that, you’ll need Heroku’s toolbelt. Then:

git clone git@github.com:orchestrate-io/hextor.git
cd hextor
heroku create
heroku config:set ORCHESTRATE_API_KEY=...
heroku config:set ORCHESTRATE_COLLECTION=...
# open up `defaults.js` and modify the default strategy if you want
git push heroku master
heroku ps:scale worker=1 web=0

Now Hextor will slay conflicts day and night.

The Future of Hextor

Our hope is to extend Hextor to resolve conflicts in other systems that experience them, like CouchDB. Whatever system you use, it should be effortless.

Likewise, we hope to introduce changes to our search indexing that categorically eliminate these kinds of conflicts in Orchestrate. Until then, take Hextor for a spin and let us know what you think, yeah?

Happy coding!