Ideas and Facts from scotland.js in Edinburgh

Test Driven Development

I have been one of the lucky attendants of scotland.js in Edinburgh recently.
It was a really nice & informative conference, thanks to all people that made it possible.
I did really like to see that TDD is reaching the front-end developers finally.
A lot of useful tools for this have been presented by James Shore, Bernard Kobos and Sebastian Golasch.
In ArangoDB, TDD is in action all day and I am looking forward to improving our front-end testing even further using these awesome tools.

 

Front-end Development meets NoSQL

Furthermore several talks focussing on front-end development have been given, e.g. by Gregor Martynus presenting Hoodie.js.
These front-end talks and my discussions with other attendees gave me the impression that front-end developers spend a lot of time deciding which database they should use.
However many databases focus on one special task and have been optimised for it. If the developers follow the vision of polyglot-persistence which is quite common in the NoSQL world, several different databases have to be installed in order to use the best tool for each job. However maintaining such a crowd of databases is quite an effort and it is much harder to decide which data should be persisted in which database.

In practice many front-end developers could benefit from a more general solution that supports several use-cases. People on the conference expressed their need for a more general NoSQL database, giving the benefits of document-stores, graph-databases and key-value stores, which can serve a broad range of use cases. Additionally, using just one database can make it easier for developers to wrap their head around as they only have to learn one technology.

That said, I am getting more and more convinced that we are going in the right direction with our developments of ArangoDB, which is more general and serves as a document-store as well as a graph-database while only having one interface and one query language for both.

 

Rapid API Development

Another interesting talk has been held about deployd – a web-based framework to create REST-APIs for MongoDB.

It is especially targeted for those applications where most of the program logic is in the front-end and the backend only has the task to persist data.
It offers many convenience functions to define access rights on REST-calls as well as on documents and even document attributes.
Furthermore it is shipped with an easy to use content management system.
The general idea of deployd reminded me of the FOXX-framework that has just been released for ArangoDB and I have been asked about the differences between FOXX and deployd.
For the curious ones I made a short list of main differences:

Feature deployd FOXX
Database MongoDB ArangoDB
Collections Bound to database Bound to app
Routes HTTP methods on collection names User defined routes
Hosting Cloud or own server Own server (cloud in progress)
App sharing Not built-in Central repository
Posted in Community, FOXX, General, Javascript | Leave a comment

ArangoDB 1.3.0 released

Features and Improvements

The following list shows in detail which features have been added or improved in ArangoDB 1.3. ArangoDB 1.3 also contains several bugfixes that are not listed here.

Changes to the Datafile Structure

As the datafile structure has changed, please read the upgrade manual carefully.

Rapid API Development with FOXX

A preview of the forthcoming Foxx is contained in 1.3. Please note that this is not the final version, Foxx is still experimental.

Foxx is a lightweight Javascript “micro framework” which allows you to build applications directly on top of ArangoDB and therefore skip the middleman (Rails, Django, Symfony or whatever your favorite web framework is). Inspired by frameworks like Sinatra Foxx is designed with simplicity and the specific use case of modern client-side MVC frameworks in mind.

The screencast at http://foxx.arangodb.org explains how to use Foxx.

Transactions

ArangoDB provides server-side transactions that allow executing multi-document and even multi-collection operations with ACID guarantees.

Transactions in ArangoDB are defined by providing a JavaScript object which needs to contain the transaction code, and some declarations about the collections involved in the transaction.

The transaction code will be executed by the server en bloc. If execution of any statement in the transaction code fails for whatever reason, the entire transaction will be aborted and rolled back.

Data modifications done by transactions become visible to following transactions only when a transaction succeeds. Data modifications that are performed by a still-ongoing transaction are not exposed to other parallel transactions. In fact, transactions on the same collection will be executed serially.

The following example will atomically transfer money from one user account to another:

db._create("accounts");
db.accounts.save({ _key: "john", amount: 423 });
db.accounts.save({ _key: "fred", amount: 197 });

db._executeTransaction({
  collections: {
    write: "accounts"
  },
  params: {
    user1: "fred",
    user2: "john", 
    amount: 10
  },
  action: function (params) {
    var db = require("internal").db;
    var account1 = db.accounts.document(params['user1']);
    var account2 = db.accounts.document(params['user2']);
    var amount = params['amount'];

    if (account1.amount < amount) {
      throw "account of user '" + user1 + "' does not have enough money!";
    }

    db.accounts.update(account1, { amount : account1.amount - amount });
    db.accounts.update(account2, { amount : account2.amount + amount });

    /* will commit the transaction and return the value true */
    return true; 
  }
});

Please refer to Transactions for more details and examples on transaction usage in ArangoDB.

New Administration Interface

ArangoDB 1.3 comes with a new administration front-end. The front-end is now based on backbone and uses repl.it, which allows for instance line editing when using the browser based ArangoDB shell.

Please note, that the “Application” tab belongs to the forthcoming Rapid API Development with FOXX Foxx. The functionality below this tab is neither stable nor complete. It has been shipped as a feature preview.

New Server Statistics

The server statistics provided by ArangoDB have been changed in 1.3.

Before version 1.3, the server provided a multi-level history of request and connection statistics. Values for each incoming request and connection were kept individually and mapped to the chronological period they appeared in. The server then provided aggregated values for different periods, which was implemented using a constant recalculation of the aggregation values.

To lower ArangoDB’s CPU usage, the constant recalculation has been removed in 1.3. Instead, the server will now only keep aggregate values per figure reported, but will not provide any chronological values.

Request and connection statistics values are 0 at server start, and will be increased with each incoming request or connection. Clients querying the statistics will see the accumulated values only. They can calculate the values for a period of time by querying the statistics twice and calculating the difference between the values themselves.

The REST APIs for the statistics in ArangoDB 1.3 can be found at:

/_admin/statistics
/_admin/statistics-description

The /_admin/statistics-description API can be used by clients to get descriptions for the figures reported by /_admin/statistics. The description will contain a textual description, the unit used for the value(s) and the boundary of slot values used.

The previoulsy available APIs

/_admin/request-statistics
/_admin/connection-statistics

are not available in ArangoDB 1.3 anymore.

AQL extensions

It is now possible to extend AQL with user-defined functions. These functions need to be written in Javascript, and be registered before usage in an AQL query.

arangosh> var aqlfunctions = require("org/arangodb/aql/functions");
arangosh> aqlfunctions.register("myfunctions:double", function (value) { return value * 2; }, true);
false
arangosh> db._query("RETURN myfunctions:double(4)").toArray();
[ 8 ]

Please refer to Extending AQL with User Functions for more details on this.

There have been the following additional changes to AQL in ArangoDB 1.3:

  • added AQL statistical functions VARIANCE_POPULATION, VARIANCE_SAMPLE, STDDEV_POPULATION, STDDEV_SAMPLE, AVERAGE, MEDIAN. These functions work on lists.
  • added AQL numeric function SQRT to calculate square-roots.
  • added AQL string functions TRIM, LEFT and RIGHT for easier string and substring handling.
  • the AQL functions REVERSE and LENGTH now work on string values, too. Previously they were allowed for lists only.
  • made “limit” an optional parameter in the NEAR function. The “limit” parameter can now be either omitted completely, or set to 0. If so, an internal default value (currently 100) will be applied for the limit.

Please refer to Functions for detailed information on the AQL functions.

Node Modules and Packages

ArangoDB 1.3 supports some of modules and packages from node. The most important module is maybe the Buffer support, which allows to handle binary data in JavaScript.

arangosh> var Buffer = require("buffer").Buffer;
arangosh> a = new Buffer("414243", "hex");
ABC
arangosh> a = new Buffer("414243", "ascii");
414243
arangosh> a = new Buffer([48, 49, 50]);
012

Supplying the Buffer class makes it possible to use other interesting modules like punycode. It enables us to support some of NPM packages available – for instance CoffeeScript.

arangosh> var cs = require("coffee-script");
arangosh> cs.compile("a = 1");
(function() {
  var a;

  a = 1;

}).call(this);

arangosh> cs.compile("square = x -> x * x", { bare: true });
var square;

square = x(function() {
  return x * x;
});

“underscore” is also preinstalled.

arangosh> var _ = require("underscore");
arangosh> _.map([1,2,3], function(x) {return x*x;});
[ 
  1, 
  4, 
  9 
]

The node packages can be installed using npm in the “share/npm” directory. If you find out, that a node package is also working under ArangoDB, please share your findings with us and other users.

Miscelleanous changes

  • Added server startup option --database.force-sync-properties to force syncing of collection properties on collection creation, deletion and on collection properties change.The default value is true to mimic the behavior of previous versions of ArangoDB. If set to false, collection properties are still written to disk but no immediate system call to sync() is made.

    Setting the --database.force-sync-properties to false may speed up running test suites on systems where sync() is expensive, but is discouraged for regular use cases.

  • ArangoDB will now reject saving documents with an invalid “type”.Previous versions of ArangoDB didn’t reject documents that were just scalar values without any attribute names.

    Starting with version 1.3, each document saved in ArangoDB must be a JSON object consisting of attribute name / attribute value pairs.

    Storing the following types of documents will be rejected by the server:

    [ "foo", "bar" ]
    1.23
    "test"

    Of course such values can be stored inside valid documents, e.g.

    { "data" : [ "foo", "bar" ] }
    { "number" : 1.23 }
    { "value" : "test" }

    User-defined document attribute names must also start with a letter or a number. It is disallowed to use user-defined attribute names starting with an underscore. This is due to name starting with an underscore being reserved for ArangoDB’s internal use.

  • Changed return value of REST API method /_admin/log :P reviously, the log messages returned by the API in the text attribute also contained the date and log level, which was redundant.

    In ArangoDB 1.3, the values in the text attribute contain only the mere log message, and no date and log level. Dates and log levels for the individual messages are still available in the separate timestamp and level attributes.

  • Extended output of server version and components for REST APIs /_admin/version and /_api/version:To retrieve the extended information, the REST APIs can be called with the URL parameter details=true. This will provide a list of server version details in the details attribute of the result.
  • Extended output for REST API /_api/collection/<name>/figures:The result will now contain an attribute attributes with a sub-attribute count. This value provides the number of different attributes that are or have been used in the collection.
Posted in General, Releases | Leave a comment

Foxx Screencast: Part 2

It has taken some time, but now part 2 of the Foxx Screencast is available. I talk about some more advanced topics of Foxx and also plans for the future.

Posted in General | Leave a comment

ArangoDB 1.3.alpha1 and the first Foxx Screencast

Starting today you can install the first Alpha of the upcoming release of ArangoDB version 1.3. Some of the new features are:

  • ArangoDB Foxx: A lightweight way to define APIs directly on top of ArangoDB
  • Traversals: Define traversals to explore your graphs stored in ArangoDB
  • A new and improved Frontend: Featuring a new design and various improvements
  • And more: Multi-Collection transaction support, user defined functions in AQL, more builtin AQL functions and AQL improvements, bug-fixes…

ArangoDB Foxx: The first screencast

If you want to get a first look at ArangoDB Foxx, you can now see our first screencast. Lucas talks about the motivation and goals of Foxx and introduces you to the basics. A second screencast will follow with more advanced techniques like our Repositories and Models:

How to install it?

  • If you are on a Mac and using Homebrew, you can install it now via
    brew install --devel arangodb.
  • If you are compiling ArangoDB by yourself, checkout the branch 1.3 from github and compile it.
  • if you are using Linux, we have create package for the usual distributions. You can download them from here.
Posted in General | Leave a comment

Foxx – a lightweight Javascript application framework for ArangoDB

For the last months we have been working on Foxx – a brand new Javascript framework on top of the upcoming version of the free and open source NoSQL database ArangoDB.
Foxx allows you to build APIs directly on top of the database and therefore skip the middleman (Rails, Django, Symfony or whatever your favorite web framework is). Foxx is designed with simplicity and the specific use case of modern client-side MVC frameworks in mind.
There are a couple of very interesting scenarios for Foxx:

  • You want to build a Single Page Web Application: most of the business logic is in the Javascript frontend. The backend’s most important task is to persist the data and send data to the client plus a bit of auth, modify, validate logic.  In this case ArangoDB plus a Foxx application is all you need as a backend.
  • Another use case is a mobile app either native or HTML5. Again you’ll have most of the logic already in the frontend (the app in this case). With Foxx you can create your API very easily. And as you don’t have a large full-size MVC stack included, your backend will be very, very fast.

You might want to consider Foxx even as lightweight, easy-to-learn alternative to Node.js.

How does it work?

Foxx apps are written in Javascript, the Javascript code is stored in production mode in the database. A Foxx app consists of a manifest file (a kind of central config file), the javascript application code, additional libraries like Backbone you might want to use, images and other assets.
As in similarly structured frameworks like Sinatra or Silex you define your own routes and assign functions to the routes. In these functions you have full access to the database.
Well, we all want to see code instead of reading long texts, so here is the not so awesome “Hello World” app in Foxx.

app = new FoxxApplication();
 
app.get("/hello/:name", function(req, res) {
 var name = req.params("name"),
  res.set("Content-Type", "text/plain");
  res.body("Hello " + name);
});
 
app.start(applicationContext);

Assuming we have a default ArangoDB installation and ArangoDB is running on port 8259 the browser can now access a url  http://localhost:8259/hello/{something} via REST. The database matches the url and returns “Hello {something}”. 

Is that all? Of course not. If you like to separate controller like logic from model related functionality, you can do this with Foxx as well. We don’t want to show this in detail in this very first announcement, but if you want to sneak a peek, have a look at Michael’s “todo app” implementation in Foxx.

What else is in Foxx?

  • Routes can have typed parameters (“foo” = string)
  • Routes can have annotations – the api documentation can be generated automatically from the annotations using Swagger
  • Foxx offers support for modules
  • Support for CoffeeScript
  • Support for the Underscore template engine is built in, other libs can be used as well.
  • The asset delivery system is easy to use and straight forward
  • “Format middleware”: React to certain request types (e.g. JSON) in a straight forward way
  • In development mode you edit your application files with your favorite editor/IDE. ArangoDB will read the files from the disk with each request. In production mode the code is stored inside of ArangoDB for faster response.
  • Using the setup/teardown feature in the manifest file.
  • You can run multiple Foxx apps at the same time – take a Foxx blog, a Foxx chat, a Foxx forum -> voilá, your community is ready
  • And: there will be a place to share Foxx apps

Is Foxx production ready?

This is a technical preview for the open minded – get the feature preview branch from Github, play around with it, give us some feedback and send us feature requests and muffins (we already have a real fox in the office, but if you have another one, come and bring it as a present as well). And yes, we are going to provide binaries very soon. And: not all features are done yet. On top of the list is the authentication feature, which will come next.

Installing the demo app

ArangoDB has now a “preview/Foxx” branch and a repository “ArangoDB-Apps” – The demo todo app called “aye_aye” is available there. First start arangosh.

arangosh&gt; var aal = require("org/arangodb/aal");

Look for the “aye_aye” app in the repository:

arangosh&gt; aal.printFishbowl("aye_aye");
Name: aye_aye
Author: Michael Hackstein
 
Description:
This a demo application implement a ToDo list based on TodoMVC.
 
1.0.7: aal.load("github", "mchacki/aye_aye", "v1.0.7");

There is one version listed above, so go ahead and load it:

arangosh&gt; aal.load("github", "mchacki/aye_aye", "v1.0.7");
{
  "path" : "/tmp/Foxx/share/arangodb/js/apps/aye_aye-1.0.7",
  "error" : false,
  "code" : 200
}

Check that it is loaded now:

arangosh&gt; aal.printAvailable()
 
AppID              Name                  Version    Path
=================  ====================  ========== ===================================
app:aardvark:1.0   aardvark              1.0        aardvark
app:aye_aye:1.0.7  aye_aye               1.0.7      aye_aye-1.0.7

Next we make it available using the “/todos” route:

arangosh&gt; aal.installApp("aye_aye", "/todos");
{
  "appId" : "app:aye_aye:1.0.7",
  "mountId" : "21447079"
}
 
arangosh&gt; aal.printInstalled()
 
MountID     AppID             Mount              Active  Devel
==========  =========         =================  ======  =====
21447079    app:aye_aye:1.0.7 /todos             yes     no

Open your browser at
http://localhost:8529/todos/

and enjoy!

To remove the app run:

arangosh&gt; aal.uninstallApp("/todos");
Posted in General | Leave a comment

Auto-increment values in ArangoDB

The most recent versions of ArangoDB offer more detailed control over what values will be auto-generated for the _key and _id attributes of documents. They also allow using predictable auto-increment sequences.
More »

Posted in General, Releases | Leave a comment

(German video) Nosql meets mobile.cologne

Last week Jan (core member of ArangoDB and the brain behind AQL and many other parts) was invited by “mobile.cologne” – a user group here in Cologne dealing with mobile development.

In this talk Jan gives a general overview on nosql databases and the different flavors. He explains how to query a nosql database and he evaluates how a nosql database can be used in a mobile app.

Warning: The talk is in German. If you want to hear it in English, let us know in the comments. :-)

“NoSQL meets mobile.cologne” by Jan Steemann.

Posted in Presentation | Leave a comment

Getting started with ArangoDB and Symfony 2 – part 1

This is part 1 (of 4) of an introduction to the use of ArangoDB together with Symfony 2. You’ll find the links to the other parts of this tutorial at the end of this text.

In this tutorial we will implement a very simple movie database as an example of how to use ArangoDB together with Symfony 2. We assume for the tutorial that you know the basic concepts of Symfony2. No prior ArangoDB knowledge is required.

The demo shows how to create, edit, update and delete movies, how to select a list of movies from the database in two different ways. We’ll use the simple query API for most examples. The “search for a topic” feature uses ArangoDB’s query language (AQL), a convenient sql-like way to query ArangoDB.

You can download the completed demo at Github.

More »

Posted in API, php | Tagged , , , | Comments Off

Getting started with ArangoDB and Symfony 2 – part 2

This is part 2 (of 4) of an introduction to the use of ArangoDB together with Symfony 2. You’ll find the links to the other parts of this tutorial at the end of this text.

You can download the completed demo at Github.

The data structure

Our app deals with movies: each movie has a title, a year of release and it is assigned to a genre. Each movie deals with none, one or many topics.

In a relational database we would probably create a table movie and another table topic and join them with a shared movie id.

ArangoDB follows like other document stores the “aggregate model” approach: In short terms this means that you should put data that is often needed together into one document. Remember: documents are a bit like rows in the relation world; in nosql world they can have a nested structure.

More »

Posted in API, php | Tagged , , , | Leave a comment

Getting started with ArangoDB and Symfony 2 – part 3

This is part 3 (of 4) of an introduction to the use of ArangoDB together with Symfony 2. You’ll find the links to the other parts of this tutorial at the end of this text.
You can download the completed demo at Github.

Working with forms

The next example shows the use of ArangoDB with Symfony’s form component.

This is how our form will look like. The topics are represented as a text field and the user is required to separate them by commas. This delivers questionable user experience, but leads to an interesting question: how can we transform our topics array from our movie document to a text field? And: we’ll need it at least two times, both for creating and editing movies.

More »

Posted in API, php | Tagged , , , | Leave a comment