Sample App Backend
Edit this article in GitHub
Version 2.4

Sample App Backend

The Node.js SDK tutorial bridges the gap between simple and advanced concepts by walking through a complete web application.

The full source code for the tutorial is available on GitHub: http://github.com/couchbaselabs/try-cb-nodejs. The primary focus of the tutorial is to explain the function and theory behind the Couchbase Node.js client and how it works together with Couchbase Server, and especially new features in versions 4.0/4.5 like N1QL, FTS and sub-document. It makes use of the travel-sample data set. The code that generates the web application is provided with the source code.

Specific Node.js prerequisites and set up

In addition to the prerequisites mentioned in Sample Application, you'll need:
  • Node.js and NPM
To get set up for the tutorial proper, follow these steps:
  • git clone https://github.com/couchbaselabs/try-cb-nodejs.git or download the source.
  • Install necessary libraries: npm install.
  • Start the application: npm run start.
  • Run the app: http://localhost:8080.
  • If you don't want to connect to Couchbase on localhost, change the configuration in index.js.
  • Press CONTROL+C to stop the application when finished.
Tip: This tutorial focuses on querying through N1QL and FTS rather than map-reduce views. If you want information about using views, see the following resources:

Walking Through the API

The following sections lead you through the primary functions of the sample application. All of the REST API application code is in the index.js file. The following shows you how to program with the various features and services of Couchbase including: connecting to a cluster and bucket, key/value iteraction, document query through N1QL and full text searches.

Configure and Bootstrap the SDK

Where: couchbase.Cluster()

Goals: Connecting to the Cluster and getting a reference to a Bucket, learn to reuse it.

Relevant Documentation Topics: Managing Connections Using the Node.js SDK with Couchbase Server

The first step is to let the application connect to your cluster and obtain a reference to a Bucket (this is your entry point for the whole storage API).

Connecting to the Cluster and Bucket
var cluster = new couchbase.Cluster('couchbase://localhost');
var bucket = cluster.openBucket('travel-sample');

The couchbase.Cluster() method instantiates a single cluster object which contains buckets. To open a bucket, use the cluster object cluster.OpenBucket().

Both bucket and cluster can be managed through the SDK as well (eg. add views or create new buckets), see Managing Clusters using the Node SDK with Couchbase Server for more information.

Managing Users using Key/Value API

Where: '/api/user/signup' and '/api/user/login'

Goals: Use Bucket operations and discover the Document API.

Relevant Documentation Topics: CRUD Document Operations using the Node.js SDK with Couchbase Server, Asynchronous Progamming Using the Node.js SDK with Couchbase Server, Sub-Document Operations

Couchbase Server is a document oriented database which provides access to your data both through its document ID (for high performance access), as well as through views, N1QL and FTS.

This is noticeable in the API, where the methods reflect Key/Value operations ( upsert,lookupIn, etc...) and work with a Sub Document interface that has uses id and requests specific content from the document. The default Document implementation accepts a simple representation of JSON as its content.

Creating New Users

A new user first goes through the signup method which does a POST of username and password encoded as a JSON object.

userrec = {'username': user, 'password': password}
A JWT token is also used through further interaction to validate the user session.
Tip: The "user::" prefix is arbitrary to this application, this is just a convention that the app uses to obtain unique keys and have additional information in it, but the key could have been anything else (even sequence numbers or UUIDs).
Here comes the part where Couchbase API is used to store the document, it's rather simple, using an insert method
bucket.insert(userDocKey, userDoc, callbackFn)

The resulting JSON data with JWT token and confirmation is sent back to the client with a success or an exception if it failed.

Tip: When it comes to storing a document, you have a choice of three methods:
  • insert will only work if no document currently exists for the given ID, otherwise an error couchbase.errors.KEY_EEXISTS will be returned.
  • replace on the contrary will only work if the document does already exist otherwise an error couchbase.errors.KEY_ENOENT will be returned.
  • upsert will always work, replacing or creating the document as needed.

So the result in fact just contains a JWT (Json WebToken) to identify the new user. If there is a problem, a Not OK status code will be returned to the client along with the error message.

Checking User Login
In the login route, in order to check a User's credential, the corresponding user document has to be retrieved. The user documents are identified by prefixing their username with user::, fetching the document using get api and verifying if the passwords match in the callback to validate the user login.
bucket.get(userDocKey, callbackFn)

If that particular key doesn't exist, the get method returns couchbase.errors.KEY_ENOENT. That's useful to check if the user exists at all.

Otherwise it's just a matter of checking the password with the one provided by the user, and responding accordingly.

A First N1QL Query: Finding Airports

Where: '/api/airports'

Goals: Use N1QL and the DSL to perform your first SELECT on Couchbase.

Relevant Documentation Topics: N1QL Queries Using the Node.js SDK with Couchbase Server.

In the SDK, there is a query method that accepts all variants of querying with Couchbase (views, spatial/geo views, N1QL and FTS). For N1QL, the couchbase.N1qlQuery is used to create the query.

Tip: N1QL is a super-set of SQL, so if you're familiar with SQL you'll feel at ease.

Statements can be provided either in String form or using the DSL.

The airports route is expected to return a reponse containing a List several matching rows.

Just the airport name has to be selected from relevant documents in the bucket and filtered based on a criteria that depends on the input length, so let's just do the SELECT and FROM clauses first:
queryprep = "SELECT airportname FROM `travel-sample` WHERE "

Then just the correct fields can be selected to look into depending on the length of the input. The user can enter either a ICAO or FAA code or a full name of an airport to search for, so each scenario can be accommodated as the N1QL statement is built. There can also be wildcards in the statement to give a free form expression:

var qs;
if (searchTerm.length === 3) {
    // FAA code
    qs = "SELECT airportname from `travel-sample` WHERE faa = '" + searchTerm.toUpperCase() + "';";
} else if (searchTerm.length === 4 &&
      (searchTerm.toUpperCase() === searchTerm ||
        searchTerm.toLowerCase() === searchTerm)) {
    // ICAO code
    qs = "SELECT airportname from `travel-sample` WHERE icao = '" + searchTerm.toUpperCase() + "';";
} else {
    // Airport name
    qs = "SELECT airportname from `travel-sample` WHERE LOWER(airportname) LIKE '%" + searchTerm.toLowerCase() + "%';";
}
The statement is ready! You can execute this statement by wrapping it in a couchbase.N1qlQuery.fromString(). Here it is very simple, no placeholders and no particular tuning of the query is necessary, so the simple method will be used:
bucket.query(q, callbackFn) 

The results of the query are returned in a list of records in the callback, which then is added to the response:

function(err, rows) {
    if (err) {
      res.status(500).send({
        error: err
      });
      return;
    }

    res.send({
      data: rows,
      context: [qs]
    });
  }

More Complex Queries: Finding Routes

Where: '/api/flightPaths/:from/:to'

Goals: Let the DSL guide you into making more complex N1QL queries.

Relevant Documentation Topics: N1QL Queries Using the Node.js SDK with Couchbase Server.

In this class, there are two more complex queries. The first aims at transforming the human-readable airport name for the departure and arrival airports to FAA codes:

SELECT faa AS fromAirport FROM `travel-sample` WHERE airportname = "Los Angeles Intl"
  UNION SELECT faa AS toAirport FROM `travel-sample` WHERE airportname = "San Francisco Intl"

The second aims at constructing the result set of available flight paths that connect the two airports:

SELECT a.name, s.flight, s.utc, r.sourceairport, r.destinationairport, r.equipment
  FROM `travel-sample` AS r
  UNNEST r.schedule AS s
  JOIN `travel-sample` AS a ON KEYS r.airlineid
  WHERE r.sourceairport = "LAX" AND r.destinationairport = "SFO" AND s.day = 6
  ORDER BY a.name ASC
Tip: Yes, you read that right, N1QL can do joins (on a single bucket or on several). It works as long as the "foreign key" described by ON KEYS clause can be mapped to a document's key in the joined bucket.

A specificity of N1QL that can be seen in the second statement is UNNEST. It extracts a sub-JSON and puts it at the same root level as the bucket (so its possible to do joins on each element in this sub-JSON as if they were entries in a left-hand side bucket).

For this final step, try to obtain the equivalent of these statements via the DSL and see how it guides you through the possibilities of the query language.

Indexing the Data: N1QL & GSI

Goals: Use the Index DSL to make sure data is indexed for N1QL to query it.

Index management is a bit more advanced (and is already done when loading the sample), so now that you've learned about N1QL, you can have a look at it. There is no code example in this application, but some is presented below for your reference.

For N1QL to work, you must first ensure that at least a Primary Index has been created. For that you can use the DSL from the Index class:

bucketmanager.createPrimaryIndex(options, callbackFn)

You can also create secondary indexes on specific fields of the JSON, for better performance:

bucketmanager.createIndex(options, callbackFn)

Use options to give a name to your index and also specify the fields in case of secondary index.

Full Text Search: Finding Hotels

Where: '/api/hotel/:description/:location'

Goals: Use FTS to search for matching Hotels. Use sub-document API to fetch the relevant data for each hit.

Relevant Documentation Topics: Full Text Search (FTS) using the Node.js SDK with Couchbase Server, Sub-Document Operations.

In this service, the hotels can be looked up using more fuzzy criteria like the content of the address or the description of a hotel, using FTS. Once there are some results, fetch only the relevant data for each result to be displayed in the UI using the sub-document API.

The '/api/hotel/' route accepts two parameters, location and description, which are the two possible refining criteria for a hotel search. To find hotels, use:
couchbase.SearchQuery.term('hotel').field('type')

A ConjunctionQuery allows you to combine multiple FTS queries into one, as an AND operation. That search always includes an exact match criteria that restricts it to the hotel data type (as reflected in the type field of the JSON document).

If the user provided a location keyword, add a second component to the FTS query that will look for that keyword in several address-related fields of the document. That is done in an OR fashion, using a Disjunction this time:

var qp = couchbase.SearchQuery.conjuncts(couchbase.SearchQuery.term('hotel').field('type'));
if (location && location !== '*') {
    qp.and(couchbase.SearchQuery.disjuncts(
        couchbase.SearchQuery.matchPhrase(location).field("country"),
        couchbase.SearchQuery.matchPhrase(location).field("city"),
        couchbase.SearchQuery.matchPhrase(location).field("state"),
        couchbase.SearchQuery.matchPhrase(location).field("address")
    ));
  }

Similarly, if a description keyword was provided by the user, the search query can use the freeform text description field and name field of the document:

if (description && description !== '*') {
    qp.and(
        couchbase.SearchQuery.disjuncts(
            couchbase.SearchQuery.matchPhrase(description).field("description"),
            couchbase.SearchQuery.matchPhrase(description).field("name")
        ));
  }

The MatchPhraseQuery can contain several words and will search for variations of the words (e.g. including plural forms or words with the same root...).

The compound FTS query is now ready to be executed. Build a SearchQuery object out of it, which also determines which FTS index to use ("hotels") and allows to set various parameters (like a limit of maximum 100 hits to return). The query is logged (and kept for narration) then executed, returning a SearchQueryResult object:

couchbase.SearchQuery.new('travel-search', qp).limit(100);
The FTS results are then iterated over, and the document corresponding to each result is fetched. In actuality, only the parts of the document that will be displayed in the UI are required. This is where the sub-document API comes in.

The sub-document API allows you to fetch or mutate only a set of paths inside a JSON document, without having to send the whole document back and forth. This can save network bandwidth if the document is large and the parts that the application is interested in are small. So here the results of the FTS search are iterated over and appropriate subdoc calls are triggered:

for (var i = 0; i < rows.length; ++i) {
      (function(row) {
        bucket.lookupIn(row.id)
            .get('country')
            .get('city')
            .get('state')
            .get('address')
            .get('name')
            .get('description')
            .execute(function (err, docFrag) {
              if (totalHandled === -1) {
                return;
              }
              var doc = {};
              try {
                doc.country = docFrag.content('country');
                doc.city = docFrag.content('city');
                doc.state = docFrag.content('state');
                doc.address = docFrag.content('address');
                doc.name = docFrag.content('name');
              } catch (e) { }

              // This is in a separate block since some versions of the
              //  travel-sample data set do not contain a description.
              try {
                doc.description = docFrag.content('description');
              } catch (e) { }

              results.push(doc);
                            totalHandled++;

              if (totalHandled >= rows.length) {
                res.send({
                  data: results,
                  context: []
                });
              }
            });
      })(rows[i]);
    }

Each FTS result is represented as a row dictionary with each document's id. The sub-document API is dedicated to fetching data (bucket.lookupIn(documentId).get('country').get('city')...) with specific fields to be looked up: country, city, state, address, name and description. In the rest of the code, the address-related fields are checked for some empty values and then aggregated together and the data obtained is returned as a JSON document for the browser.