What is Orchestrate?

Orchestrate makes databases simple by powering full-text search, events, graph, and K/V storage behind a REST API.


Receive Blog Updates

Share:

James Estes

New Release: Updated Orchestrate Events API

04.24.2014
blog_feature_social

We’ve spent the last few weeks making some significant updates to our Events API.

Events in Orchestrate can be a powerful tool if you know how to use them. The power of Orchestrate Events comes from two rules:

  1. Events are always associated with a specific item in a collection.

  2. Events are always ordered by timestamp.

This means that events are a perfect fit for when you have time-ordered data you want to associate with a specific item in a collection. Audit history and activity trails are both great examples of events data in action. Before we continue with the events API update, let’s take a closer look at the kinds of features you can build with Events.

What You Can Build with Events Data

Audit History for an Item

Audit history is the ability to track all the changes to a particular item in a collection. In our sample app Monster Match, for example, we want to know about all the changes made to a user’s profile over time, and be able to look back and review all of those edits. For this we might have a collection called “profiles.” For simplicity, lets say Bigfoot is in that collection under “profiles/bigfoot”.

Tracking changes in a user profile or a document is a perfect fit for using Events. Monster Match could have a “changes” event type that will have an event added for every edit to Bigfoot’s profile, which would record who changed what in the profile: profiles/bigfoot/events/changes

When any change happens to Bigfoot’s profile, Monster Match would record the change along with a ‘changes’ Event with information about the change that Bigfoot can see later in a screen that shows him a history of all the changes to his profile (ordered most recent first).

Activity Streams for an Item

Activity streams keep track of activities that occur in a system that relate to a particular item. This is also a great fit for Orchestrate Events.

In our Monster Match example (or any social network) users like to see other users’ public activity, to get to know a little more about them. To make this happen, Monster Match developers might add an event type called “activities” to user profiles: profiles/bigfoot/events/activities

So, looking at a profile, they might like to see:

  • Links they’ve shared
  • Things they’ve liked
  • Activity on other social networks (Monster Match has fantastic integration with other services)
  • Comments or status updates they’ve made

Not only are users interested in what Bigfoot does, sometimes they’re also interested in how other users interact with Bigfoot:

  • Things of Bigfoot’s other people like
  • Comments made on Bigfoot’s shares
  • Mentions of Bigfoot in other peoples statuses.

Any interesting activity that Bigfoot does, or that is done to Bigfoot, Monster Match can record as an activity to Bigfoot’s activity stream so users can see it later when viewing his profile.

Monster Match Events

New Orchestrate Events API Features

We’ve listened to a lot of feedback from users about our Events API and have addressed many of them in a major new release this week.

Multiple Events per Timestamp

Items can now have multiple events of the same type occur within the same millisecond. For example, if you have a ‘profile_viewed’ event type on your Monster Match users, you will now be able to see that Nessie AND Capt. Zargon BOTH viewed Bigfoot’s profile at the exact same millisecond.

Previously, the timestamp was considered part of the identity of each Event. We received feedback that users would like to have better granularity in the event timestamps. So, events now have another field called ‘ordinal’ that will be used to further identify an event within the same millisecond (more on the ‘ordinal’ field next).

Insertion Order Preserved

Events are always ordered by their timestamp newest-to-oldest in range query results (reverse-chronological order). When multiple Events have the same timestamp (to the millisecond), these Events will still be in reverse-chronological order, but use the insert order to determine what the chronology is.

The ‘ordinal’ field value represents an insert order. For a specific item’s events, the ordinal will be always increasing as events are added. However, the ordinals will not be strictly sequential (there may be gaps). It is useful only as a mechanism for determining insert-order of events for a specific key: higher values were added after lower values. Ordinals should only be compared to other ordinals within the same item key (ie, the ordinal space may be different between item keys).

Timestamp Formats

Various event query API endpoints support specifying ‘timestamp’ parameters. These parameters are specified either as a query string parameters, or as part of the uri itself.

Previously, the timestamps could only be the formatted as milliseconds since epoch. We’ve now added support for multiple timestamp formats. Wherever a timestamp is expected, you can provide a value that is either:

  • A Long that is the milliseconds since epoch
  • ISO8601 (with or without the milliseconds portion)
  • RFC2616 Compatible Dates. This allows the following formats:
    • RFC1123 – Sun, 06 Nov 1994 08:49:37 GMT
    • RFC1036 – Sunday, 06-Nov-1994 08:49:37 GMT (2-digit date supported and uses 2000 as start year)
    • ASCTIME – Sun Nov  6 08:49:37 1994 (GMT Enforced)

POST to Add Events (PUT Deprecated)

For adding a new Event, we now have a POST endpoint. Events can be added by issuing a POST request to the event-type uri or the event-type/timestamp:

POST /v0/$coll/$key/events/$event-type

Here, the timestamp will be generated at insert time.

If you have a specific timestamp that you want the event to fall under (i.e. you manage timestamps or already have the event timestamp):

POST /v0/$coll/$key/events/$event-type/$timestamp

Get/Update/Delete an Individual Event

We now support manipulating an individual Event. For example, if you are using Events for Bigfoot activities:

10:02 call Nessie
10:02 check Nessie’s Profile
10:01 check Monster Match Profile
10:00 logged in to Monster Match

And later, Bigfoot is checking his activity, and wants to update one of those events. Now, you can add  tag:

10:02 call Nessie, tags:["funny"]

Or decides he doesn’t want people to know his login habits:

DELETE 10:00 logged in to Monster Match

To fetch an individual event, issue a GET to the fully qualified Event path:

GET /v0/$coll/$key/events/$event-type/$timestamp/$ordinal

The response will include only that single event. To update an event, issue a PUT to the same uri, with the new value as the request body:

PUT /v0/$coll/$key/events/$event-type/$timestamp/$ordinal

To delete an event, issue a DELETE to the uri and include a purge=true parameter. Events do not support full history, so the api will require the purge=true parameter here to make it clear that this is the behavior.

DELETE /v0/$coll/$key/events/$event-type/$timestamp/$ordinal?purge=true

Conditional Updates and Deletes

If you want to update an event, but only if it has not been modified by another request, you can use the If-Match header. Just like with KV Item conditional updates, the If-Match header value should be the event’s “ref” value. The “ref” value is returned as part of the Event itself in a GET request and it is also provided as the ETag header on the POST, PUT and individual GET requests (all requests to an individual event).

Improved Event Range Queries

Previously, event ranges could only be between start and end timestamps, where start was always inclusive (ie included events with this timestamp), and end was non-inclusive (i.e. up to, but not including, events with this timestamp).

We’ve now added support for a non-inclusive start (afterEvent) and an inclusive end (endEvent). Also, to be consistent with naming that is used in KV Item pagination, we have deprecated ‘start’ and ‘end’, and now use the following names:

startEvent – the start of the event range (inclusive)
afterEvent – the start of the event range (non-inclusive)
beforeEvent – the end of the event range (non-inclusive)
endEvent – the end of the event range (inclusive)

Note that the endEvent is inclusive whereas the former ‘end’ parameter was non-inclusive.

The results will still be returned in reverse-chronological order. The range parameters only restrict the range of Events that will be ultimately returned. So if you specify an endEvent for example, the first event in the returned results list will likely be that exact event, if it exists.

The format of this parameter is a timestamp and ordinal pair, since we need to be able to succinctly but fully identify a unique event for the range parameters. The format is: “[timestamp]/[ordinal]“. The ordinal may be left out if the query is only concerned with a time range, so you can continue to just specify a timestamp if that is what you need. For example:

GET /v0/$coll/$key/events/$event-type?startEvent=$timestamp/$ordinal

or if you just want to get events starting at a timestamp

GET /v0/$coll/$key/events/$event-type?startEvent=$timestamp

The timestamp values here can be any supported timestamp format mentioned above.

If you only want all events that have a specific timestamp, you can issue a GET to that timestamp:

GET /v0/$coll/$key/events/$event-type/$timestamp

And only events with that exact (ie to the millisecond) value will be included in the results.

Event Pagination

Sometimes you might have a LOT of events for an item, and you really would like to restrict how many will be returned for each query. We’ve now added pagination support for Event queries. By default, only the first 10 Events will be returned in the request. For larger pages, specify the ‘limit’ parameter with a value up to 100.  If there are more Events, the response will also provide a “next” field in the result list as well as a Link header with a rel=”next” whose value will be the uri that can be used to fetch the next page of results.

We hope you enjoy the latest updates and find them useful!

 

 

Orchestrate gives developers access to multiple NoSQL databases through a single, RESTful API, eliminating the need to run databases in production. Learn more