The process of splitting up data and storing portions of data on different machines is called sharding.  By splitting up data across machines, it becomes possible to store more data and handle much more load without requiring large or powerful machines, e.g., machines that consist of powerful CPU’s and/or massive amounts of RAM.

Two types are sharding can occur.  Manual sharding and autosharding.

In manual sharding, the application code manages storing different data on different servers and querying the appropriate server to get it back.  Manual sharding can be done with virtually any database software package.

In MongoDB autosharding, some of the administrative overhead required in manual sharding is eliminated.  The cluster of database servers, or shards, handles splitting up of data and rebalancing of data automatically.

Autosharding

The basic concept behind MongoDB’s sharding is to break up collections into smaller chunks, or documents.  These documents can be distributed across shards so that each shard is responsible for a subset of the total dataset.

As an example, consider the following.  When you set up sharding you choose a key from a collection and use that key’s values to split up the data.  This key is called the shard key.

Suppose we had a collection of contacts.  If we chose “lastName” as our shard key, one shard could hold documents where “lastName” starts with A-F, the next shard could hold last names from G-P, and the final shard could hold last names Q-Z.  As you add or remove shards, MongoDB would rebalance this data so that each shard was getting a balanced amount of traffic and a practical amount of data.

So when should you decide to start sharding?  Consider the following reasons:

• When you’ve run out of disk space on your current machine.
• You want to write data faster than a single mongod can handle.
• You want to keep a larger portion of data in memory to improve performance.

Setting up Sharding

Three different components are involved in sharding as follows:

shard

A shard is a container that holds a subset of a collection’s data.  A shard is either a single mongod server (for development/testing), or a replica set (for production).

mongos

This is the router process.  It routes requests and aggregates responses.  It doesn’t store any data or configuration information, although it does cache information from the config servers.

config server

Config servers store the configuration of the cluster.  For example, which data is located on which shard.  Used by mongos to determine request routing.

Starting the Servers

First we need to strat up our config server and mongos.  We need to start a config server because mongos uses it to get its configuration.

$ mkdir -p ~/dbs/config
$ ./mongod --dbpath ~/dbs/config --port 20000

Now we start a mongos process for an application to connect to.  Routing servers do not even need a data directory, but they need to know the location of the config server.

$ ./mongos --port 30000 --configdb localhost:20000

Shard administration is always done through a mongos.

Adding a Shard

Start a normal mongod instance (or replica set), since this is what a shard naturally is

$ mkdir -p ~/dbs/shard1
$ ./mongod --dbpath ~/dbs/shard1 --port 10000

Now connect to the mongos process started earlier and add the shard to the cluster.

First, start up a shell connected to the mongos process as follows:

$ ./mongo localhost:30000/admin

Now add this shard with the addshard database command:

> db.runCommand({addshard : "localhost:10000", allowLocal : true})
{
    "added" : "localhost:10000",
    "ok" : true
}

The “allowLocal” key is necessary only if you are running the shard on localhost and lets MongoDB know that you’re in development and know what you are doing.

Sharding Data

In order to allow MongoDB to distribute data, you have to explicitly turn sharding on at both the database and collection levels.  For example, the following enaables sharding for the database acme:

> db.runCommand({"enablesharding" : "acme"})

Once you’ve enabled on the acme database, a collection is sharded by running the shardcollection command as follows:

> db.runCommand({"shardcollection" : "acme.products", "key", : {"_id" : 1}})

Now the collection will be sharded by the “_id” key.  When data is added to acme, it will be automatically distributed across the shards based on the values of “_id”.

I hope this post enlightens you on the possibilities that MongoDB’s auto-sharding feature provides for ease of scaling.

• easy learning curve
• faster development than with traditional relational databases
• high availability and easy horizontal scalability
• extremely fast access times
• powerful indexing and querying capabilities, similar to its relational database counterparts
• an ideal database platform for both cloud and desktop computing
• schemaless, document repositories, via collections and documents, that naturally fit application domains
• an ideal database platform for embracing agile development

A general set of MongoDB supported Use Cases, both well suited and less well suited are found here.

It should be noted that MongoDB shares the NoSQL spotlight with CouchDB.  Since MongoDB and CouchDB are both document-oriented databases with schemaless JSON-style object data storage, folks are naturally asking questions.  Please click here to compare MongoDB and CouchDB.

MongoDB Features

Here is a list of MongoDB features.  Click on the links for details.

• Document-oriented storage via JSON-style documents with dynamic schemas
• Full Index Support
• Replication & High Availability
• Auto-Sharding
• Querying
• Fast In-Place Updates
• Map/Reduce
• GridFS
• Commercial Support

The MongoDB Philosophy

MongoDB was not created to be just another database that tries to do everything for everyone.

Instead, MongoDB was created to work with documents rather than rows, was extremely fast, massively scalable, and easy to use.

In order to accomplish this, some features were excluded, namely support for transactions.  Therefore, MongoDB may not be a great fit for developing accounting applications.  However, it is quite common to strategize software architectures by using a hybrid of technologies.  For example, a relational database works well with accounting and transactional components.  While MongoDB is a great technology for storing and retrieving complex data and document storage.

At a high-level, MongoDB provides the following (this is not an exhaustive list):

• A Rich Data Model
• Easy Scaling
• Simple Administration
• Features such as indexing, embedded document references, stored javascript, fixed-size collections, file storage, aggregation, replication, auto-sharding, and more.
• And all of the Above without Sacrificing Speed

MongoDB is a document-oriented database.  It is not a relational one.  The primary reason for moving away from the relational model to the document-oriented model is to make scaling out easier, but additional advantages are found as well.

The fundamental idea is replacing the concept of a row with the more flexible model, the document.  By supporting embedded documents and arrays, the document-oriented approach makes it possible to represent complex hierarchical relationships with a single record.  This fits naturally with how data needs are addressed by developers using modern object-oriented languages.

More coming soon!

Installing and Running MongoDB on Ubuntu

I highly recommend installing MongoDB via the default repositories available in Ubuntu.  These repositories contain MongoDB, but may contain out-of-date versions.  Software installation from repositories is done through aptitude.  In order to obtain the latest versions of MongoDB, simply add the following line to your repository list found in /etc/apt/sources.list

deb http://downloads.mongodb.org/distros/ubuntu 10.4 10gen

where 10.4 is the version of Ubuntu I run.  Make sure to supply the version of Ubuntu you run.

Next, you need to tell aptitude to retrieve the new repositories as follows:

$ sudo aptitude update

And now install MongoDB using the following command:

$ sudo aptitude install mongodb-stable

As the last step in the installation process, you will need to create the data directory yourself.  Make sure and do this as a non-root user.

$ sudo mkdir -p /data/db/
$ sudo chown `id -u` /data/db

You can start the MondoDB service as follows, and again, make sure and do this as a non-root user:

$ sudo start mongodb
$ sudo status mongodb

And start the MongoDB shell with the following command:

$ mongo

Once in the MongoDB shell, some basic commands include the following:

> show dbs
> show collections
> show users
> use <db name>

Quick Reference to the MongoDB Upstart Scripts

Here is a quick reference to the commands that control the execution of the mongod server process:

$ sudo status mongodb
$ sudo stop mongodb
$ sudo start mongodb
$ sudo restart mongodb

Important in Case of Error Connecting to the Server

If upon starting the MongoDB shell, you happen to get an error message connecting to the server similar to the following:

Error: couldn’t connect to server 127.0.0.1} (anon):1137

You may also notice that even though you issued the command to start the service, it actually did not start.  The next step is to verify that you are unable to start MongoDB.

Run sudo start mongodb.  It will report

mongodb start/running, process XXXX

no matter what.  However, when you run sudo status mongodb again, you’ll get

mongodb stop/waiting instead of mongodb start/running

Note:  This condition is largely due to an unclean shutdown, and results in the creation of a lockfile /var/lib/mongodb/mongod.lock.

The fix is a quick two-step process as follows:

1. Remove the lockfile.
2. Run the repair script.

This is accomplished as follows:

$ sudo rm /var/lib/mongodb/mongod.lock
$ sudo -u mongodb mongod -f /etc/mongodb.conf --repair

Now when you run sudo start mongodb.  It will report

mongodb start/running, process XXXX

and when you run sudo status mongodb again, you’ll get the expected

mongodb start/running, process XXXX

Understanding Data Modeling under MongoDB

A MongoDB database is non-relational and schemaless.  Therefore a MongoDB database is not bound to columns and data types like relational databases are.  One of the primary benefits of a flexible schemaless design is that you are not restricted when programming in a dynamically typed programming language such as Python.  Keep in mind that even though MongoDB is schemaless, the data structure is not completely devoid of schema as you still define collections and indexes, as we will discuss later.  Nevertheless you will not need to predefine any data structure for any of your documents you will be adding.

The fundamental components of a MongoDB database are documents and collections.

Document is an item that contains the actual data, similar to a row in a SQL database table.

Collection is a collection of documents, similar to a table in a SQL database.  Unlike a SQL database, two or more completely different documents can co-exist in a single collection.

More coming soon!

Developing Applications under MongoDB using Python with Example Code

Python is a simple programming language that provides the natural ability to develop code that is perfectly readable.  Here are some links to Python code examples:

• Catalog of Python Code Examples
• Simple Python Programs

In this section, we will develop simple, clear, and powerful code that works with MongoDB through the Python driver known as the PyMongo driver.

Installing the PyMongo Driver

Before we begin to write Python code to access MongoDB databases, we first need to install the PyMongo driver.

I’m assuming you have Python 2.7 or greater up and running.  The steps for Python installation are simple once you obtain the source from python.org/download and extract the contents from the tar file, and are commonly listed as follows:

$ ./configure
$ make
$ make test
$ sudo make install

Of course, you should always consult the readme file to obtain up-to-date instructions.

The steps required to install the PyMongo driver are as follows:

Step 1

Obtain the Python setuptools egg for your version of Python.  For example, I obtained setuptools-0.6c11-py2.7.egg from http://pypi.python.org/pypi/setuptools#using-setuptools-and-easyinstall.

Step 2

Now execute the downloaded egg as if it were an acual shell script by typing (note your setuptools egg filename may be different depending on the version of Python you are using).

$ sudo sh setuptools-0.6c11-py2.7.egg

Step 3

$ sudo easy_install pymongo

That’s it!  But let’s test to ensure that our installation was a success.

From the Python shell, type the following:

>> import pymongo

You should be greeted with an empty cursor >>.  That’s success.

To take it one step further, we’ll insert data into a database and also retrieve it back.  Remember in MongoDB, if we try to retrieve a database that does not exist, MongoDB creates it for us automatically, as is the case with mytestdb below.

Type the following in the Python shell:

>> from pymongo import Connection
>> c = Connection()
>> db = c.mytestdb
>> collection = db.items
>> item = { "Title" : "Test Data", "Value1" : "1", "Value2" : "2"}
>> collection.insert(item)
>> collection.find_one()

{u'_id': ObjectId('4d432adc1d41c85d8a000000'), u'Value1': u'1', u'Value2': u'2', u'Title': u'Test Data'}

An Introduction to Working with MongoDB using Python via the PyMongo Driver

Now that you have the PyMongo driver installed and running, it is time to learn the basics in developing Python code that works with MongoDB.

Instead of reinventing an already great tutorial, I provide this link to the tutorial located on the mongodb.org site.

Enjoy.

Advanced:  Sharding

Since sharding in MongoDB is a key feature and large topic to discuss, I created a separate post found here.  This 15 minute high speed post provides a detailed overview of autosharding in MongoDB and, specifically, how to create shards supporting autosharding in MongoDB.

Enjoy.

This post discusses how to ensure your production CRM environment is tuned and optimized for top performance.

My knowledge in this area has been largely obtained by working closely with Robert Shurtleff, the Microsoft CRM Dynamics Architect at Ciber.

Therefore I supply a link to Robert’s post on this subject.

Enjoy!

Conversely, administering some common advanced performance tactics targeted at your specific usage can upgrade performance metrics to well above acceptable standards.

More coming soon.

This program has known compatibility problems… check online to see if solutions are available from the Microsoft website…

Well, I did find another Sql Server 2008 Express Edition SP1 download link here.  After learning this, I thought my life was looking up.  Yeah, right!

To get to the point, I Googled around a bit and found this post from someone who had the exact same problem as me.  After reading his post and some of his readers’ comments, I determined I really should use the Microsoft Web Platform Installer.  It provides for the installation of many web development express versions, such as Sql Server 2008 Management Studio Express Edition, and even Silverlight.  And, best of all, you don’t need to worry about the install order of dependent components!

Worked like a charm!

As always, work smarter. Not harder.

My example is based on a simple Message entity that has a fishhook association to another Message entity.  The key here is that the relationship defines a 0..1 aggregation and may associate to another Message entity, or not.  When we query a list of Message entities we need to return all Message entities, including both

• those that contain associations to other Message entities, and
• those that do not contain associations to other Message entities

My entities are generated from the Entity Framework, and I use a DTO to return the actual data.  The context is established through a “BusinessEntities”object (as shown in line 3 below).  The outer join is demonstrated in lines 6 – 10 below.  Here is the code example:

public IList<DomainModel.Entities.DTOs.Message> Messages(int recipientPartyId)
{
   var context = new BusinessEntities();
   IQueryable<DomainModel.Entities.DTOs.Message> entities =
   from m in context.Message
   let leftouter = (from msgRef in m.MessageReference
       select new
       {
           MsgRef = msgRef,
       }).FirstOrDefault()
   join recipient in context.Party on m.PartyRecipient.Id equals recipientPartyId
   where
       m.MessageStatusType.Id != MessageSendStatuses.Sent &&
       m.MessageStatusType.Id != MessageSendStatuses.Draft
   select new DomainModel.Entities.DTOs.Message
   {
       Id = m.Id,
       PartySenderId = m.PartySender.Id,
       SenderFirstName = m.PartySender.FirstName,
       SenderLastName = m.PartySender.LastName,
       PartyRecipientId = m.PartyRecipient.Id,
       RecipientFirstName = m.PartyRecipient.FirstName,
       RecipientLastName = m.PartyRecipient.LastName,
       Subject = m.Subject,
       Body = m.Message,
       SentAt = m.CreatedAt,
       MessageTypeId = m.MessageType.Id,
       MessageStatusTypeId = m.MessageStatusType.Id,
       MessageRecipientStatusTypeId = m.MessageRecipientStatusType.Id,
       MessageReference = leftouter.MsgRef != null ? new DomainModel.Entities.DTOs.Message(leftouter.MsgRef) : null
   };

   return entities.OrderByDescending(m => m.SentAt).ToList();
}

In line 30 above, if leftouter.MsgRef != null, we return an initialized Message DTO representing the associated Message entity.  Otherwise, leftouter.MgsRef == null, therefore we return null.

Hope this helps!

Every type of user interface needs a way to page through lists longer than a typical user interface can display in one screen.  Mobile platforms are no different, and in fact, may need this more than others.  Smartphones such as Droids, Palm Pre’s, Blackberrys, and iPhones contain small screens allowing for only a handful of list items to be displayed at once.  Some lists, such as your typical contacts list, reside directly on the device itself, and can be quickly navigated from front to back, and back to front again by a flick of your finger.  However, some lists, such as those residing in enterprise systems (and thus requiring over-the-network server requests), may be large enough to benefit from paging techniques when navigating those lists on a smartphone.  The reasons for paging are really no different for mobile clients than web pages.

In order to understand how to implement paging, the concept of paging must be understood.  Given a list of 50 items, and a page length of 10, we can break this list into 5 pages.  We commonly allow access to the pages through prev, next, and page number links.

Now let’s discuss the parameters required by the links to effectively execute paging.

A page is represented by the following criteria:

• Resource name or identifier
• Page number
• Page size

The resource name or identifier is obvious.  If you plan on creating a service that implements paging, you must define the page number and page size.

Example

Let’s move on with creating an actual service that supports paging.  We will create our simple RESTful service in ASP.NET MVC.  I will also provide this working service on my demo site (see actual service calls located after code).  Please note that we could easily use Rails, Grails, or other appropriate framework.

We create an EmployeesController that provides an Index(…) method.  The Index(…) method uses the parameters page number and page size.  If page number is null, we set it to 0 to indicate page 0.  This method calls the retrieve(…) method on the EmployeeRepository supplying the page number and page size parameters.

And lastly, the retrieve(…) method loads the employee records, using the LINQ extension methods Skip(…) and Take(…) to load the specific page requested.  Note that Skip(…) requires the number of records to skip, not number of pages.  So we use .Skip(pageNum * pageSize), as shown in line 43 below.

That’s it!  And here is the code demonstrating paging.

namespace SOAServices.Controllers
{
    [WebApiEnabled]
    public class EmployeesController : Controller
    {
    //
    // GET: /Employees/

    public ActionResult Index(int? pageNum, int? pageSize)
    {
        pageNum = pageNum != null ? pageNum : 0;
        pageSize = pageSize != null ? pageSize : 5;
        IEmployeeRepository repository = new EmployeeRepository();

        return View(
            repository.Retrieve((int)pageNum, (int)pageSize)
            .AsSerializable<Entities.DTOs.Employee>());
        }
    }
}

namespace SOAServices.Models.Abstract
{
    public interface IEmployeeRepository
    {
        // define basic CRUD methods
        IList<Entities.DTOs.Employee> Retrieve(int pageNum, int pageSize);
        //Entities.DTOs.Employee Create();
        //Entities.DTOs.Employee Update(Entities.DTOs.Employee emp);
        //void Delete(int employeeId);
    }
}

namespace SOAServices.Models.Concrete
{
    public class EmployeeRepository : IEmployeeRepository
    {
       private EmployeeBusinessEntitiesDataContext _db = new EmployeeBusinessEntitiesDataContext(ConfigurationManager.ConnectionStrings["EmployeeDatabaseConnectionString"].ConnectionString);
       public IList<Entities.DTOs.Employee> Retrieve(int pageNum, int pageSize)
       {
           var employees = _db.Employees
              .OrderBy(emp => emp.LastName + emp.FirstName)
              .Skip(pageNum * pageSize)
              .Take(pageSize)
              .Select(emp =>
                new Entities.DTOs.Employee
                {
                  FirstName = emp.FirstName,
                  LastName = emp.LastName,
                  Email = emp.Email,
                  Title = emp.Title,
                  Phone = emp.Phone,
                  Id = emp.Id,
                  DeptId = emp.DeptId,
                  SalaryGradeId = emp.SalaryGradeId,
                  TakesLongLunches = emp.TakesLongLunches
                });
            return employees.ToList();
        }
    }
}

namespace SOAServices.Entities.DTOs
{
    [Serializable]
    [DataContract]
    public partial class Employee
    {
        [DataMember]
        public string FirstName { get; set; }
        [DataMember]
        public string LastName { get; set; }
        [DataMember]
        public string Email { get; set; }
        [DataMember]
        public string Phone { get; set; }
        [DataMember]
        public string Title { get; set; }
        [DataMember]
        public int Id { get; set; }
        [DataMember]
        public int DeptId { get; set; }
        [DataMember]
        public int SalaryGradeId { get; set; }
        [DataMember]
        public bool TakesLongLunches { get; set; }
    }
}

Let’s look at the real working code on my demo site.  We will use a page size of five and we will demonstrate using these employee records.

Pay special attention to the pageNum and pageSize url parameters as they specify the page to return.

Since the main topic of this post is to support paging in Android and our service is using the webapi declaration in ASP.NET MVC providing multiple serializable formats, we’ll first look at XML.

Page 1, Page 2, Page 3

And now Json.

Page 1, Page 2, Page 3

As you can see, it is very simple to support paging.  Next we will discuss how to work with Android’s ListView to support our new paging model.

More coming soon!