What is Orchestrate?

Orchestrate makes databases simple by powering full-text search, events, graph, and K/V storage behind a REST API.


Receive Blog Updates

Share:

Antony Falco

Voices: Tristan Sloughter on Erlang, NoSQL and Technology’s Effect on Economics and Culture

blog_feature_techlines

If you’ve been following the Orchestrate blog, we frequently reach out to folks in the broader community to get their take on a  range of topics important to developers and technophiles.  This week, we caught up with our friend, developer and economic pontificator, Tristan Sloughter (@t_sloughter).

Currently an infrastructure developer at Heroku, Tristan has  worked as a software engineer for companies including Orbitz and Mashape. He’s also quite active in the Erlang and open source communities, and recently wrote an Erlang client for Orchestrate. We picked his brain on a variety of topics in Erlang, NoSQL…. and even the economics and culture of technology today.

Erlang and NoSQL have both played a big role in work you’ve done. Do you see any commonalities – thematic, architecturally, even culturally – between functional programming (and programmers) and the recent spate of NoSQL databases?

Erlang and NoSQL definitely had a similar course, for better and worse. With the move to multiple cores Erlang had a major opening, having great SMP support since like 2005. It was a clear fit for the language. Next came not simply having to deal with large amounts of data but doing so on commodity hardware.

And this is where the goes to “the worse”. Both NoSQL and Erlang had a burst of use and interest but because they were seen as silver bullets. Soon people realized you couldn’t simply translate your imperative code to Erlang and see improvements but instead regressions. Additionally, throwing your relational data at a NoSQL databases caused the same.

I feel the NoSQL culture and programmers haven’t retracted to the core yet as much as Erlang. Though Erlang may see another surge of misuse and misinterpretation now with the sale of WhatsApp :).

Which NoSQL databases have you used?  Do you try to fit a project to databases you know or do you let requirements dictate, even if it means introducing something new?

I’ve mostly used CouchDB and Riak. When starting a project I feel it is important to start with something you know, unless you are 100% sure you need something different. Too often the assumptions we make when starting projects are wrong and course must change. I much prefer changing course from something that I understand, and thus truly know why it isn’t going to work, to something new, then having spent time trying something new (again, obvious caveats here) only to realize what I already knew was a perfect fit, or adequate fit even.

Unless, of course, the project is just to learn new things, and then try it all :)

You’ve worked on the plumbing of really big systems (Orbitz, Heroku) that have doubtless evolved to be very specialized.  Is there room for the new wave of infrastructure with public APIs – e.g. PagerDuty, Sendgrid, Orchestrate – to supplant or complement these internal systems or are requirements just too specific?

There is definitely room. When I was at Orbitz they were on the move to try new services. Orbitz started in 1999 and has grown greatly, making it harder to do so. Some tools that Orbitz developed for monitoring, Graphite for example, are still used by so many companies. But using services is much less about needing control than it is about trust. And in the sense of older companies like Orbitz, it is about the labor required to replace what is working with a service. Even if it would mean improvements, whether in reliability or less stressed engineers, those have to be weighed against how much it will take to engineer the move.

You contributed the Erlang client for Orchestrate (thanks, btw!). What, besides a free tier, can services offer to foster a community, and to make sure we do more than just extract profits?

Yes, a free tier and open API is a must! But it is important not to over commit to a free tier. If developers can’t easily try a service there won’t be new tools or uses the provider might not have even thought of developed. And when that developer ends up having a use for that tool in a real project they’ll have confidence proposing your service.

But an overcommitted free tier can become a mess and is no good for anyone. People have to be paid, so money has to be made, which everyone (ok, most everyone) understands, but people aren’t understanding about having what they thought was free taken away. If your service makes it possible for people to build real world apps on the free tier they will do it and they will be very upset if in the future that goes away because the free tier was over committed from the start.

One of the ways I got to know you is our shared interest in the rights of labor. Do you think it is possible to organize software engineers? Does the myth of meritocracy make organizing more difficult? How do we avoid cynicism?

I think it is definitely possible to organize engineers. But also very difficult. And the difficulties aren’t all unique to engineers. With high unemployment, and high under-employment, it is said you should feel lucky simply to have a job and to not ask for more. Which is a great tactic, if you want to keep the status quo of high under-employment and increasingly lopsided flow of wealth (from the bottom to the top).

Beyond these global issues you have sort of two-tiers of engineers and the engineer culture to overcome. Often it is forgotten just how many programmers there are. Instead the focus is on these select few companies and startups as the mass of programmers, when this is not even close to the scope. I’ve spoken to many developers who think golden jobs exist all over easily to grab by anyone who will work hard enough. This is not the case at all and the programmers in the companies with slides and free lunches must learn this and unite with the real world or risk finding themselves on the wrong side of labor battle in the future.

I think the push for everyone to learn to code shows what the future holds. More and more companies in all fields are having uses for programmers, and the easier it is to replace them with cheaper labor the better for their profits. A reserve army of labor is necessary for this, and is being built. So it may be very difficult right now to organize, but the conditions are changing.

We won’t be able to convert all the developers who dream of going Galt — and who builds the iPhone’s in Galt’s utopia anyway? But they are the few. More likely the mass of developers are on the “Right” but not because of libertarian ideology but simply being in the United States.

Today it seems technology creates disparities or imbalances of wealth, information, and access to services. Is this reversible? Or does the economic framework too strongly shape their use? If “better” means a fair distribution of resources, can technology really make the world a better place?

Technology has a huge part to play in creating a world of fair distribution and people being able to lead fulfilled lives. And advances can be made under the current economic model through struggle, as we have seen over the last 2 and a half centuries.

The first issue when discussing this is people think Internet apps are such technology advancements and forget about the washing machine. Technology like the washing machine greatly reduced the household labor required to be expended everyday. This wasn’t an issue for the rich as they had people to do their washing, but for the working class when this became available in laundromats and for some in the home it saved massive amounts of time that could now be spent elsewhere.

This is similar to how advancements in food production, when humans no longer had to spend their whole day collecting enough food to reproduce the labor they need to do the same the next day, they were able to develop science and art.

The Internet itself has created major changes in the world. And not all advancements have to be labor saving, Wikileaks is an example of this. I’m not talking about the leaks that upset the United States but those before, and to a lesser extent after, they became a target of the US. This spreading information to the people to expose the crimes of the rulers can have drastic effects.

Now this is where Stuart Varney rants that capitalism allowed for the washing machine to be developed and we should be thankful and want to strengthen the current system. This is typical Hayek philosophy and ignores what really happened. Simply developing a technology doesn’t change the world, but the social battles around it, either directly or indirectly, do. The market can’t do this for us. Without the fight for universal suffrage, a 40 hour work week, civil rights, and many more any technological benefit would always be minimal for the workers.

We can do a better job but it will require changes to the core of the political and economic system. The economy must be democratized and planned. We currently have no production issue but a distribution issue. There is enough food and housing, it simply isn’t being properly distributed, and this shows that technology itself can not provide a solution.

 

Orchestrate gives developers access to multiple NoSQL databases through a single, RESTful API. It’s free for up to one million API calls a month.