Cookie policy: This site uses cookies (small files stored on your computer) to simplify and improve your experience of this website. Cookies are small text files stored on the device you are using to access this website. For more information on how we use and manage cookies please take a look at our privacy and cookie policies. Some parts of the site may not work properly if you choose not to accept cookies.


Planning for scale with Enterprise NoSQL

  • Print
  • Share
  • Comment
  • Save

All applications have inherent limits in what they can support. As organizations grow, so do their expectations on the apps they use. Development and operations teams need to plan for large scale from day one — not react to it when it finally happens.

When you hit the limitations of your system, you’ll need to determine how best to scale up. This can be done by changing the cluster hardware configuration, patching or replacing the database software, re-designing schemas, migrating data to other services, or a host of other tasks that often exacerbate issues in the short term or prolong downtime. The process can be costly and error-prone as organizations learn what pitfalls their database management systems (DBMS) vendor hasn’t told them about or hasn’t yet discovered.

The limitations of the traditional, relational approach to the DBMS are becoming more exposed as enterprise cloud services become more ingrained in daily business. NoSQL databases offer a solution to large scale but can be complex to implement. So what do you need to know?

Design tradeoffs
Traditional relational databases are designed for consistency. There is a raft of NoSQL databases that target different use cases, and often emphasise availability of the data. The differences lie in either using expensive hardware that is powerful enough to provide database consistency at large scale (‘vertical scaling’), or using new database software that supports availability across groups of commodity servers (‘horizontal scaling’).

Popular relational databases like MySQL scale well vertically but are complex to scale horizontally. As demand grows for enterprise cloud applications, there are good reasons for using NoSQL databases that are designed to scale horizontally:

  1.  Size of data — if your data set doesn't fit on one machine, you need two
  2.  Concurrency — if a server can handle 10,000 req/sec and you get 20,000, you need two machines

When a database needs more than one machine to handle demand, you hit CAP theorem. Posed by Eric Brewer, CAP theorem states that a distributed database system can prioritize two of three properties: consistency, high availability, and partition tolerance (more on Brewer’s CAP theorem can be found at

While horizontally scaled databases can effectively and easily scale with cloud applications, multi-machine system introduces the problem of partitioning events; nodes will fail, networks will get cut off. Partitioning leads us to a choice between high availability partitions (AP) and consistency partitions (CP). With CP, all nodes must agree before data is written. This means that availability is hampered during a network partition as no one is able to write. For AP, consistency is sacrificed to allow the storage layer to remain available and usable to the application. The drawback is that different clients may see different data during periods when nodes haven’t all received the latest data.

Plan for success
Worrying about scaling issues when they hit is no longer appropriate, especially in a world where demand can no longer be controlled (e.g. the ‘app store effect’). Users hit servers immediately, with no limits on when and how people are using apps. In fact, the only way to assert control in the app store era is to lock down an application. You either make users wait until servers can handle more load, or pull the application entirely. Such draconian measures hardly qualify as planning for success.

Large, heavily used systems bring with them a high probability that a portion of the system will fail. A database engineered around this assumption that prioritises availability and eventual consistency is better suited to keeping your application online.

ATMs are a great example. Inconsistent banking data is why it’s still possible to overdraft money without realizing it. It is unrealistic to present a consistent view of your account balance throughout the entire banking system if every node in the network needs to halt and record this figure before continuing operations. It’s better to make the system highly available.

Enterprise collaboration software poses a new problem in the age of mobile data. When mobile devices lack network access and go offline, there are essentially two disconnected systems on which users are updating data. Allowing them to have mutable data on their phones or tablets while the network is offline would be an important feature. Syncing these updates for thousands of users when these devices come back online is a huge problem.

Deal with NoSQL complexity now
The view that we take with Cloudant’s database-as-a-service, along with CouchDB and other NoSQL databases, is that it’s better to expose developers to the complexities of large scale early in the design process. Address scaling issues head on, so that you don’t have to solve them at 3am or right before that important demo.

Building these systems is challenging, which is why there are plenty of vendors who offer consulting and support for managing distributed NoSQL databases, and others who will host and manage these systems for you. Whether you decide to have help or build it yourself, when you’re launching a new cloud application for your business, keep application availability at large scale in mind. The enterprise version of the app store effect is coming.


Related images

  • cloud application

Have your say

Please view our terms and conditions before submitting your comment.

  • Print
  • Share
  • Comment
  • Save


  • Overhead Power Distribution – Best Practice in Modular Design

    Tue, 10 Nov 2015 16:00:00

    Overhead power distribution in your data center offers many attractive possibilities but is not without its challenges. Join UE Corp’s Director of Marketing, Mark Swift, and CPI’s Senior Data Center Consultant, Steve Bornfield, for an exploration of the options and some of the pitfalls, supported by real-life examples from the field.

  • Overcoming the Challenges of High Power Density Deployments

    Wed, 4 Nov 2015 19:00:00

    Increasing rack power densities saves space and energy, and improves both OPEX and CAPEX. But it can also create unintended problems that could bring your data center to a screeching halt. Join Raritan’s VP of Products & Marketing, Henry Hsu, and DCD’s CTO Stephen Worn, as they reveal the three key challenges in deploying a high density cabinet, and explain how to: Reduce operating costs, Increase up-time , Improve mean time to repair, become more energy-efficient manage existing capacity and plan for growth.

  • Squeezing the Lemon - The Power to do More with Less

    Tue, 20 Oct 2015 08:00:00

    Energy costs rising, manpower resources falling – managing a data center is getting more stressful by the day. One cold night could be all it takes to tip your power supply over the edge. And let's not forget the never-ending demands from IT for additional space. More information on its own is not the answer. Join Rittal's webinar to understand how to: • Lower your power consumption and OPEX charges, with 'smart' power distribution • Identify issues before they become problems, with intelligent PDUs' monitoring capabilities • Expand your DC as your business grows, with modular PDUs • Profile your power requirements to help you plan and make better-informed decisions REGISTER NOW Note: All attendees will receive a free copy of the latest White Paper from Rittal.

  • Live Customer Roundtable: Optimizing Capacity (12:00 EST)

    Tue, 8 Sep 2015 16:00:00

    The biggest challenge facing many data centers today? Capacity. How to optimize what you have today. And when you need to expand, how to expand your capacity smarter. Learn from the experts about how Data Center Infrastructure Management (DCIM) and Prefabricated Modular Data Centers are driving best practices in how capacity is managed and optimized: - lower costs - improved efficiencies and performance - better IT services delivered to the business - accurate long-range planning Don;t miss out on our LIVE customer roundtable and your chance to pose questions to expert speakers from Commscope, VIRTUS and University of Montana. These enterprises are putting best practices to work today in the only place that counts – the real world.

  • Power Optimization – Can Your Business Survive an Unplanned Outage? (APAC)

    Wed, 26 Aug 2015 05:00:00

    Most outages are accidental; by adopting an intelligent power chain, you can help mitigate them and reduce your mean-time to repair. Join Anixter and DatacenterDynamics for a webinar on the five best practices and measurement techniques to help you obtain the performance data you need to optimize your power chain. Register today!

More link