[#168,doc][m]: backend docs including overview and spec of how a backend should function.

* model docs get some improvement and moved to docs/models.markdown
* change default layout to allow spec of where root is ...
This commit is contained in:
Rufus Pollock 2012-06-24 19:50:08 +01:00
parent f30c7d6273
commit 08fb55bd02
5 changed files with 375 additions and 304 deletions

View File

@ -9,7 +9,7 @@
<script src="http://html5shim.googlecode.com/svn/trunk/html5.js"></script>
<![endif]-->
<link rel="stylesheet" href="vendor/bootstrap/2.0.2/css/bootstrap.css" />
<link rel="stylesheet" href="{{ page.root }}vendor/bootstrap/2.0.2/css/bootstrap.css" />
{% if page.recline-deps %}
{% include recline-deps.html %}
@ -17,8 +17,8 @@
<!-- link rel="stylesheet" href="vendor/bootstrap/2.0.2/css/bootstrap-responsive.css" -->
<link href="css/site/pygments.css" rel="stylesheet" type="text/css" />
<link href="css/site/site.css" rel="stylesheet" type="text/css" />
<link href="{{ page.root }}css/site/pygments.css" rel="stylesheet" type="text/css" />
<link href="{{ page.root }}css/site/site.css" rel="stylesheet" type="text/css" />
</head>
<body>
<div class="navbar navbar-fixed-top">

122
docs/backends.markdown Normal file
View File

@ -0,0 +1,122 @@
---
layout: container
title: Backends
root: ../
---
<div class="page-header">
<h1>
Backends
<small>Connect to data sources</small>
</h1>
</div>
Backends come in 2 flavours:
1. Loader backends - only implement fetch method. The data is then cached in a Memory.Store on the Dataset and interacted with there. This is best for sources which just allow you to load data or where you want to load the data once and work with it locally.
2. Store backends - these support fetch, query and, if write-enabled, save. These are suitable where the backend contains a lot of data (infeasible to load locally - for examples a million rows) or where the backend has capabilities you want to take advantage of.
Backend modules must implement the following API:
{% highlight javascript %}
__type__: 'name-of-backend' // e.g. elasticsearch
// Initial load of dataset including initial set of records
fetch: function(dataset)
// Query the backend for records returning them in bulk.
// This method will be used by the Dataset.query method to search the backend
// for records, retrieving the results in bulk.
query: function(queryObj, dataset)
// Save changes to the backend
save: function(changes, dataset)
{% endhighlight %}
Details of each function below. Note that:
* Each backend function takes a dataset object. This is not a Dataset object
but is simple JS object representation resulting from calling
Dataset.toJSON().
It is required because the Dataset attributes contain details of specific
backend (e.g. url for ElasticSearch etc).
* Each function returns a promise API object - that is something conforming to
the jquery promise API and, in particular, having a done and fail function.
### fetch: function(dataset)
On success, promise callback must return an object with the following structure:
{% highlight javascript %}
{
// (optional) Set of record data
// Either an array of arrays *or* an array of objects corresponding to initial set of records for this object
// May not provided if data only returned by query
records: [...]
// (optional) Set of field data
// Either an array of string or an array of objects corresponding to Field specification (see `Field` above)
fields: { ... } // as per recline.Model.Field
// (optional) metadata fields to set on the Dataset object
metadata: { title: ..., id: ... etc }
// boolean indicating whether to use a local memory store for managing this dataset
useMemoryStore:
}
{% endhighlight %}
### query: function(queryObj, dataset)
`queryObj`: JS object following <a href="models.html#query-structure">Query specification</a> above.
#### Callbacks
On success must return a 'QueryResult' object which has the following structure:
{% highlight javascript %}
{
// total number of results (can be null)
total: ...
// one entry for each result record
hits: [
{
// JS object that can be used to initialize a Record object
}
],
// (optional)
facets: {
// facet results (as per <http://www.elasticsearch.org/guide/reference/api/search/facets/>)
}
}
{% endhighlight %}
The QueryResult is partially modelled on ElasticSearch - see <a
href="https://github.com/okfn/recline/issues/57">this issue for more
details</a>.
### save: function(changes, dataset)
<div class="alert alert-warning">The save function is still being revised and
its API and arguments are subject to change</div>
`changes`: an object with the following structure:
{% highlight javascript %}
{
creates: [ record.toJSON(), record.toJSON(), ... ]
updates: [ ... ]
deletes: [ ... ]
}
{% endhighlight %}
Each key has an array of records (as simple JS objects resulting from a call to
Record.toJSON()) that are in that state.
The backend should take appropriate actions for each case.

250
docs/models.markdown Normal file
View File

@ -0,0 +1,250 @@
---
layout: container
title: Models
root: ../
---
<div class="page-header">
<h1>
Models
</h1>
</div>
Models help you structure your work with data by providing some standard
objects. The key ones are Dataset and Record -- a Dataset being a collection of
Records. Additionally, there is a a Field object for describing the columns of
a Dataset, a Query object for describing queries, and a Facet object for
holding summary information about a Field (or multiple Fields).
# Models
All the models are Backbone models, that is they extend Backbone.Model. Note,
however that they do not 'sync' (load/save) like normal Backbone models.
## Dataset
A Dataset is *the* central object in Recline. Standard usage is:
{% highlight javascript %}
var dataset = new recline.model.Dataset({
// general metadata e.g.
id: ...
title: ...
// information about data source e.g.
url: http://url.to.my.data.endpoint/
// backend string or object
backend: the backend we are using - see below
});
// initialize dataset with data from the backend.
dataset.fetch();
// we will now have the following (and more) set up - see below for details
dataset.fields // collection of Fields (columns) for this Dataset
dataset.currentRecords // collection of Records resulting from latest query
dataset.docCount // total number of Records in the last query
{% endhighlight %}
### Key Attributes
* currentRecords: a collection of `Record`s currently loaded for viewing
(updated by calling query method) - note that this need <strong>not</strong>
be all the records in the dataset (for example, you may have connected to a
source where the complete dataset contains a million records but you have
only loaded a 1000 records)
* fields: (aka columns) is a Backbone collectoin of `Field`s listing all the
fields on this Dataset (this can be set explicitly, or, will be set by
Dataset.fetch()
* docCount: total number of records in this dataset
* backend: the Backend (instance) for this Dataset.
* queryState: a `Query` object which stores current queryState. queryState may
be edited by other components (e.g. a query editor view) changes will trigger
a Dataset query.
* facets: a collection of `Facet`s
### Querying
{% highlight javascript %}
dataset.query(queryObj)
{% endhighlight %}
`queryObj` an object following the <a href="#query-structure">query
pecification below</a>.
<h2 id="record">Record (aka Row)</h2>
A Record is a single entry or row in a dataset. A Record needs little more than
what is provided by the standard Backbone Model object. In general, you will
never create a Record directly -- they will get created for you by Datasets
from query results.
<h2 id="field">Field (aka Column)</h2>
A Field should have the following attributes as standard:
{% highlight javascript %}
var field = new Field({
// a unique identifer for this field- usually this should match the key in the records hash
id: 'my-field-id'
// (optional: defaults to id) the visible label used for this field
label: 'My Field Name',
// (optional: defaults to string) the type of the data in this field.
// Should be a string as per type names defined by ElasticSearch - see
// Types list on <http://www.elasticsearch.org/guide/reference/mapping/>
type: 'string',
// (optional - defaults to null) used to indicate how the data should be
// formatted. See below.
format: null,
// (default: false) attribute indicating this field has no backend data but
// is just derived from other fields (see below).
is_derived: false
{% endhighlight %}
#### Rendering, types and formats
One can customize the rendering of fields in the user interface and elsewhere
by setting a renderer function on the field. You do this by setting a field
attribute:
{% highlight javascript %}
myfield.renderer = myRenderFunction;
{% endhighlight %}
Your renderer function should have the following signature:
function(value, field, record)
Where the arguments passed in are as follows:
* `value`: the value of the cell (record value for this field)
* `field`: corresponding `Field` object
* `record : is the `Record` object (as simple JS object)
Note that implementing functions can ignore arguments (e.g. function(value)
would be a valid formatter function).
To guide the behaviour of renderers we have type and format information.
Example types and formats are:
* type=date, format=yyyy-mm-dd
* type=float, format=percentage
* type=string, format=markdown (render as markdown if Showdown available)
Default renderers are provided - see the source for details, but a few examples
are:
* type = string
* no format provided: pass through but convert http:// to hyperlinks
* format = plain: do no processing on the source text
* format = markdown: process as markdown (if Showdown library available)
* type = float
* format = percentage: format as a percentage
#### Derived fields
Some fields may be 'dervied' from other fields. This allows you to define an
entirely new value for data in this field. This provides support for a)
'derived/computed' fields: i.e. fields whose data are functions of the data in
other fields b) transforming the value of this field prior to rendering.
To use derived fields set a `deriver` function on the Field. This function will
be used to derive/compute the value of data in this field as a function of this
field's value (if any) and the current record. It's signature and behaviour is
the same as for renderer.
<h2 id="query">Query</h2>
Query instances encapsulate a query to the backend (see <a
href="backend/base.html">query method on backend</a>). Useful both
for creating queries and for storing and manipulating query state -
e.g. from a query editor).
<h3 id="query-structure">Query Structure and format</h3>
Query structure should follow that of [ElasticSearch query
language](http://www.elasticsearch.org/guide/reference/api/search/).
**NB: It is up to specific backends how to implement and support this query
structure. Different backends might choose to implement things differently
or not support certain features. Please check your backend for details.**
Query object has the following key attributes:
* size (=limit): number of results to return
* from (=offset): offset into result set - http://www.elasticsearch.org/guide/reference/api/search/from-size.html
* sort: sort order - <http://www.elasticsearch.org/guide/reference/api/search/sort.html>
* query: Query in ES Query DSL <http://www.elasticsearch.org/guide/reference/api/search/query.html>
* filter: See filters and <a href="http://www.elasticsearch.org/guide/reference/query-dsl/filtered-query.html">Filtered Query</a>
* fields: set of fields to return - http://www.elasticsearch.org/guide/reference/api/search/fields.html
* facets: specification of facets - see http://www.elasticsearch.org/guide/reference/api/search/facets/
Additions:
* q: either straight text or a hash will map directly onto a [query_string
query](http://www.elasticsearch.org/guide/reference/query-dsl/query-string-query.html)
in backend
* Of course this can be re-interpreted by different backends. E.g. some may
just pass this straight through e.g. for an SQL backend this could be the
full SQL query
* filters: array of ElasticSearch filters. These will be and-ed together for
execution.
#### Examples
<pre>
{
q: 'quick brown fox',
filters: [
{ term: { 'owner': 'jones' } }
]
}
</pre>
<h2>Facet <small>&ndash; Store summary information (e.g. values and counts) about a field obtained by some 'faceting' or 'group by' method</small>
</h2>
Structure of a facet follows that of Facet results in ElasticSearch, see:
<http://www.elasticsearch.org/guide/reference/api/search/facets/>
Specifically the object structure of a facet looks like (there is one
addition compared to ElasticSearch: the "id" field which corresponds to the
key used to specify this facet in the facet query):
{% highlight javascript %}
{
id: "id-of-facet",
// type of this facet (terms, range, histogram etc)
\_type : "terms",
// total number of tokens in the facet
total: 5,
// @property {number} number of records which have no value for the field
missing : 0,
// number of facet values not included in the returned facets
other: 0,
// term object ({term: , count: ...})
terms: [ {
"term" : "foo",
"count" : 2
}, {
"term" : "bar",
"count" : 2
}, {
"term" : "baz",
"count" : 1
}
]
}
{% endhighlight %}

View File

@ -1,188 +0,0 @@
---
layout: container
title: Models and Backends
---
<div class="page-header">
<h1>
Models and Backends
</h1>
</div>
Models help you structure your work with data by providing some standard objects. The key ones are Dataset and Record -- a Dataset being a collection of Records. Additionally, there is a a Field object for describing the columns of a Dataset, a Query object for describing queries, and a Facet object for holding summary information about a Field (or multiple Fields).
# Models
## Dataset
A Dataset is *the* central object in Recline. It has the following key attributes:
* currentRecords: a collection of `Record`s currently loaded for viewing (updated by calling query method) - note that this need <strong>not</strong> be all the records in the dataset (for example, you may have connected to a source where the complete dataset contains a million records but you have only loaded a 1000 records)
* fields: (aka columns) is a Backbone collectoin of `Field`s listing all
the fields on this Dataset (this can be set explicitly, or, will be
set by Dataset.fetch()
* docCount: total number of records in this dataset
* backend: the Backend (instance) for this Dataset.
* queryState: a `Query` object which stores current queryState.
queryState may be edited by other components (e.g. a query editor
view) changes will trigger a Dataset query.
* facets: a collection of `Facet`s
<h2 id="record">Record (aka Row)</h2>
A Record represents a single entry or row in a dataset. As Record fits very nicely with the default behaviour of a Backbone Model object it has little additional functionality.
<h2 id="field">Field (aka Column)</h2>
A Field should have the following attributes as standard:
{% highlight javascript %}
var field = new Field({
// a unique identifer for this field- usually this should match the key in the records hash
id: 'my-field-id'
// (optional: defaults to id) the visible label used for this field
label: 'My Field Name',
// (optional: defaults to string) the type of the data in this field.
// Should be a string as per type names defined by ElasticSearch - see
// Types list on <http://www.elasticsearch.org/guide/reference/mapping/>
type: 'string',
// (optional - defaults to null) used to indicate how the data should be formatted. See below.
format: null,
// (default: false) attribute indicating this field has no backend data but is just derived from other fields (see below).
is_derived: false
{% endhighlight %}
#### Rendering, types and formats
One can customize the rendering of fields in the user interface and elsewhere by setting a renderer function on the field. You do this by setting a field attribute:
{% highlight javascript %}
myfield.renderer = myRenderFunction;
{% endhighlight %}
Your renderer function should have the following signature:
function(value, field, record)
where value is the value of this cell, field is corresponding field
object and record is the record object (as simple JS object). Note that
implementing functions can ignore arguments (e.g. function(value) would
be a valid formatter function).
To guide the behaviour of renderers we have type and format information. Example types and formats are:
* type=date, format=yyyy-mm-dd
* type=float, format=percentage
* type=string, format=markdown (render as markdown if Showdown available)
Default renderers are provided - see the source for details, but a few examples are:
* type = string
* no format provided: pass through but convert http:// to hyperlinks
* format = plain: do no processing on the source text
* format = markdown: process as markdown (if Showdown library available)
* type = float
* format = percentage: format as a percentage
#### Derived fields:
* deriver: a function to derive/compute the value of data
in this field as a function of this field's value (if any) and the current
record. It's signature and behaviour is the same as for renderer. Use of
this function allows you to define an entirely new value for data in this
field. This provides support for a) 'derived/computed' fields: i.e. fields
whose data are functions of the data in other fields b) transforming the
value of this field prior to rendering.
<h2 id="query">Query</h2>
Query instances encapsulate a query to the backend (see <a
href="backend/base.html">query method on backend</a>). Useful both
for creating queries and for storing and manipulating query state -
e.g. from a query editor).
**Query Structure and format**
Query structure should follow that of [ElasticSearch query
language](http://www.elasticsearch.org/guide/reference/api/search/).
**NB: It is up to specific backends how to implement and support this query
structure. Different backends might choose to implement things differently
or not support certain features. Please check your backend for details.**
Query object has the following key attributes:
* size (=limit): number of results to return
* from (=offset): offset into result set - http://www.elasticsearch.org/guide/reference/api/search/from-size.html
* sort: sort order - <http://www.elasticsearch.org/guide/reference/api/search/sort.html>
* query: Query in ES Query DSL <http://www.elasticsearch.org/guide/reference/api/search/query.html>
* filter: See filters and <a href="http://www.elasticsearch.org/guide/reference/query-dsl/filtered-query.html">Filtered Query</a>
* fields: set of fields to return - http://www.elasticsearch.org/guide/reference/api/search/fields.html
* facets: specification of facets - see http://www.elasticsearch.org/guide/reference/api/search/facets/
Additions:
* q: either straight text or a hash will map directly onto a [query_string
query](http://www.elasticsearch.org/guide/reference/query-dsl/query-string-query.html)
in backend
* Of course this can be re-interpreted by different backends. E.g. some
may just pass this straight through e.g. for an SQL backend this could be
the full SQL query
* filters: array of ElasticSearch filters. These will be and-ed together for
execution.
**Examples**
<pre>
{
q: 'quick brown fox',
filters: [
{ term: { 'owner': 'jones' } }
]
}
</pre>
<h2>Facet <small>&ndash; Store summary information (e.g. values and counts) about a field obtained by some 'faceting' or 'group by' method</small>
</h2>
Structure of a facet follows that of Facet results in ElasticSearch, see:
<http://www.elasticsearch.org/guide/reference/api/search/facets/>
Specifically the object structure of a facet looks like (there is one
addition compared to ElasticSearch: the "id" field which corresponds to the
key used to specify this facet in the facet query):
<pre>
{
"id": "id-of-facet",
// type of this facet (terms, range, histogram etc)
"_type" : "terms",
// total number of tokens in the facet
"total": 5,
// @property {number} number of records which have no value for the field
"missing" : 0,
// number of facet values not included in the returned facets
"other": 0,
// term object ({term: , count: ...})
"terms" : [ {
"term" : "foo",
"count" : 2
}, {
"term" : "bar",
"count" : 2
}, {
"term" : "baz",
"count" : 1
}
]
}
</pre>
# Backends
1. Data is held in an in memory store on the Dataset object.
2. Data is transparently sourced from a backend store.

View File

@ -1,113 +0,0 @@
// # Recline Backends
//
// Backends are connectors to backend data sources and stores
//
// This is just the base module containing a template Base class and convenience methods.
this.recline = this.recline || {};
this.recline.Backend = this.recline.Backend || {};
// ## recline.Backend.Base
//
// Exemplar 'class' for backends showing what a base class would look like.
this.recline.Backend.Base = function() {
// ### __type__
//
// 'type' of this backend. This should be either the class path for this
// object as a string (e.g. recline.Backend.Memory) or for Backends within
// recline.Backend module it may be their class name.
//
// This value is used as an identifier for this backend when initializing
// backends (see recline.Model.Dataset.initialize).
this.__type__ = 'base';
// ### readonly
//
// Class level attribute indicating that this backend is read-only (that
// is, cannot be written to).
this.readonly = true;
// ### sync
//
// An implementation of Backbone.sync that will be used to override
// Backbone.sync on operations for Datasets and Records which are using this backend.
//
// For read-only implementations you will need only to implement read method
// for Dataset models (and even this can be a null operation). The read method
// should return relevant metadata for the Dataset. We do not require read support
// for Records because they are loaded in bulk by the query method.
//
// For backends supporting write operations you must implement update and delete support for Record objects.
//
// All code paths should return an object conforming to the jquery promise API.
this.sync = function(method, model, options) {
},
// ### query
//
// Query the backend for records returning them in bulk. This method will
// be used by the Dataset.query method to search the backend for records,
// retrieving the results in bulk.
//
// @param {recline.model.Dataset} model: Dataset model.
//
// @param {Object} queryObj: object describing a query (usually produced by
// using recline.Model.Query and calling toJSON on it).
//
// The structure of data in the Query object or
// Hash should follow that defined in <a
// href="http://github.com/okfn/recline/issues/34">issue 34</a>.
// (Of course, if you are writing your own backend, and hence
// have control over the interpretation of the query object, you
// can use whatever structure you like).
//
// @returns {Promise} promise API object. The promise resolve method will
// be called on query completion with a QueryResult object.
//
// A QueryResult has the following structure (modelled closely on
// ElasticSearch - see <a
// href="https://github.com/okfn/recline/issues/57">this issue for more
// details</a>):
//
// <pre>
// {
// total: // (required) total number of results (can be null)
// hits: [ // (required) one entry for each result record
// {
// _score: // (optional) match score for record
// _type: // (optional) record type
// _source: // (required) record/row object
// }
// ],
// facets: { // (optional)
// // facet results (as per <http://www.elasticsearch.org/guide/reference/api/search/facets/>)
// }
// }
// </pre>
this.query = function(model, queryObj) {}
};
// ### makeRequest
//
// Just $.ajax but in any headers in the 'headers' attribute of this
// Backend instance. Example:
//
// <pre>
// var jqxhr = this._makeRequest({
// url: the-url
// });
// </pre>
this.recline.Backend.makeRequest = function(data, headers) {
var extras = {};
if (headers) {
extras = {
beforeSend: function(req) {
_.each(headers, function(value, key) {
req.setRequestHeader(key, value);
});
}
};
}
var data = _.extend(extras, data);
return $.ajax(data);
};