Meteor - Scaling / Performance tunning

meteor

What is the purpose of Kadira?

It’s a performance monitoring tool, similar to New Relic. You register for an online account, add the package to a Meteor project, and then, from a web-based interface, you’ll gain insight into the various performance bottlenecks and other things to consider. You can add Kadira to your project with the following command:

meteor add meteorhacks:kadira

If you have no idea where to start when it comes to performance montioring, I’d suggest checking out Bulletproof Meteor — an interactive course, designed by the creators of Kadira, that dives deep into this precise topic.

One of the most useful feature to be the response time tracing. This feature allows you to drill down and really see what is slowing down the system.

What is the purpose of Compose?

Compose has a dedicated section for slow queries and an entire Monitoring dashboard. The slow queries section will mainly show you queries that are not supported by indexes. The Monitoring dashboard gives you more insight into the performance of the MongoDB deployment itself.

What is the purpose of the Spinner package?

Even with all of the performance tweaks in the world, some things will simply take some time to load, and to make this loading process seem less broken, we can use the Spinner package. This package can quickly add one of those classic, spinning loading symbols to an application. To add Spinner to your project, use this command:

meteor add sacha:spin

Then use a “spinner” template whenever it needs to be used:

{{> spinner}}

You can also configure it with a number of options:

Meteor.Spinner.options = {
    lines: 13, // The number of lines to draw
    length: 10, // The length of each line
    width: 5, // The line thickness
    radius: 15, // The radius of the inner circle
    corners: 0.7, // Corner roundness (0..1)
    rotate: 0, // The rotation offset
    direction: 1, // 1: clockwise, -1: counterclockwise
    color: '#fff', // #rgb or #rrggbb
    speed: 1, // Rounds per second
    trail: 60, // Afterglow percentage
    shadow: true, // Whether to render a shadow
    hwaccel: false, // Whether to use hardware acceleration
    className: 'spinner', // The CSS class to assign to the spinner
    zIndex: 2e9, // The z-index (defaults to 2000000000)
    top: 'auto', // Top position relative to parent in px
    left: 'auto' // Left position relative to parent in px
};

The easiest way to use the Spinner package though is to integrate it with the Iron Router package.

How can we keep memory consumption low?

The first step is making sure you only publish the data you need (assuming you don’t need to go through too many contortions to get there).

This is good practice all around: not only will this use less resources for the connection between client and server, but it will also require less browser memory on the user’s computer. And as we’ve seen, it will also have a vital impact on server-side memory use, and thus how many online users each web node can support.

The second take away is that it becomes very important to consider how long users are connected for. Because even if a user isn’t doing anything (for example, having a browser tab open in the background), this will still result in a certain server load.

What are some scaling strategies?

  1. Attempting to minimize the number of unique-per-user subscriptions will allow Meteor to pool work across users as much as possible.
  2. Using simpler queries will usually mean there’s less work for Meteor to do to decide if a change affects it.
  3. Constraining your database writes (especially against heavily observed collections) will limit the amount of work Meteor needs to do to keep observers up to date.

There’s also a few monitoring tools that can help you get a better picture of your app’s performance and locate bottlenecks. One of them is the Server Info package, which provides a password protected URL to get stats on the current number of open LRSes against each collection.

The Facts package (part of the Meteor core since 0.7.0) is also a great tool to gather data about open observers, both polling and oplog-based. And keep an eye out as well for Arunoda’s very promising MeteorAPM performance monitoring service.

Finally, to test your app’s scalability, the best tool right now is Adrian Lanning’s Load Test.

It’s a good idea to put a caching server in front of Meteor. This will reduce the load on Meteor to serve static content. Node.js (where Meteor runs) does not work well when it comes to static file serving, so using a caching server improves performance. The caching server shouldn’t cache the first HTML page loaded from Meteor into the browser. This is the only HTML content Meteor loads, and it contains a JavaScript variable called serverId that is used to compare versions in hot code reload logic.

The first thing to optimize is database indexes. Meteor doesn’t auto-magically add indexes–they need to be added explicitly. Indexes can provide a large performance gain. Subscriptions and queries can also be optimized, with notable performance gains.

MongoDB is a core component of Meteor, so it needs to be prioritized where scaling and performance are concerned. Generally, MongoDB performs quite well. It supports both vertical and horizontal scaling. It can be run on a more powerful server, or use MongoDB sharding to scale horizontally. Although MongoDB comes with good sharding tools, care needs to be taken when using sharding.

Why should we ensure all publish function queries are supported by an Index?

At a fundamental level, indexes help MongoDB more efficiently return the result of query by allowing it to setup a special data structure to represent the underlying documents. There are many types of indexes allowed by MongoDB, all of which can be found in their documentation of indexes. By default, the _id property of a document is supported by an index, however publish functions often use queries specifying fields other than the _id. For these queries, setting up an index will improve performance tremendously. At Differential, we like to put the ensureIndex (createIndex in MongoDB 3.0) calls directly after any publish function to show that the query is supported by an index. Here’s an example:

Meteor.publish('data', function (arg) {  
  return Data.find({level: arg});
});

Meteor.startup(function () {  
  Data._ensureIndex({level: 1});
});

Why should we use Oplog for all publish function queries?

In a MongoDB replica set, the oplog (short for operation log) keeps a record of all operations that modify data inside of the replica set. This special log is used by secondary (slave) members in a replica set to stay synchronized with the primary (master). To allow Meteor to utilize this log for observe changes to data, you can simply set up a MONGO_OPLOG_URL environment variable. Further information on Meteor’s oplog integration an be found here, and below is an example of the environment variable structure used. The MONGO_URL is provided directly from Compose as the Replica Set URI in the Admin section for a particular database.

// MONGO_URL
mongodb://[username]:[password]@[candidate_url]:[port_number]/[database_name]?replicaSet=set-[uniqueId]

As long as the username is setup to have oplog access, the key to setting up the oplog url is to update the database name to point to local and setting the authSource to the database name in the mongo url.

// MONGO_OPLOG_URL
mongodb://[username]:[password]@[candidate_url]:[port_number]/local?authSource=[database_name]&replicaSet=set-[uniqueId]

Why should we use fields, limits, and sorts in publish function queries?

Sending the minimum amount of data to the client helps make the sub/pub process faster, and helps protect against inadvertent data exposure. This is why specifying only the fields the client needs is a best practice for publish functions. Limits are wise to setup, especially for views that theoretically could have an infinite list. When limiting the result set of a query, typically the query is also sorted. So keeping indexing in mind is critical for performance.

Why should we increase the memory for MongoDB?

One key to determining whether or not your MongoDB deployment requires increased memory is by monitoring the page faults your database is experiencing. Page faults occur when MongoDB tries to read or write to parts of the database that are not already loaded into physical memory. Compose gives you insight into page faults visually within their Monitoring section. We saw spikes in page faults right after launching BRAVE and found that the memory allocation provided out of the box by Compose was not sufficient. Unfortunately, there is no way to increase memory from the dashboard view of Compose, so we had to contact their support team directly to get it setup.

What is MongoDB Replica Set?

Replica Set is MongoDB’s replication strategy to address high availability. It’s a good idea to deploy a Replica Set for any production MongoDB deployment. Replica Set comprises one or many MongoDB servers (nodes) replicating the same set of data. It is recommend to run a Replica Set of at least 3 nodes. But a Replica Set should not contain more than 12 nodes.

One node acts as the Primary and others are known as Secondaries. Primary election is fully automatic and if the current Primary goes down, a new “Primary” will be elected from the Secondaries. But there are many more options to customize the default behavior.

Write operations can be directed to Primary only, but you can read from any node. If you read from a Secondary, you may get data that is inconsistent with the Primary data.

Replica Sets are used mainly for high availability. But some people use Replica Sets for backups and doing background analytic jobs using Secondaries.

What is the minimum recommended number of nodes in a replica set?

3.

What is the maximum recommended number of nodes in a replica set?

12. Replica Set should not contain more than 12 nodes.

What is OpLog?

Oplog is the heart of a MongoDB Replica Set. It’s a stream of all the write operations happening inside the Primary. Actually, oplog is a capped collection named oplog.rs, located on the local database. It’s created for you automatically when initializing the Replica Set.

Capped collections can be tailed (with queries) and monitored for new additions. So, it can be used as a stream. That’s what exactly Secondaries do. They are tailing oplog on the “Primary” and make a copy of that while applying write operations(entries in the oplog) into the data that Secondary maintains. That’s how Secondaries keep updated with the Primary.

How much disk space does OpLog take?

By default, oplog takes 5% of the disk size available for the MongoDB. But you can change that via the –oplogSize option.

How does Meteor uses Oplog?

Meteor’s default observe driver is based on polling and that makes Meteor super-slow and causes it to consume a lot of server resources. As a solution for that, Meteor uses oplog to detect data changes and trigger observers. Using oplog to trigger observers is very efficient compared to the polling approach.

Meteor acts like a Secondary in the sense that it is tailing the oplog of the Primary. Meteor keeps a cache of data inside the memory and applies changes while triggering observers.

Meteor won’t cache all the data comes through the oplog, just the data relevant to observers. Actually, most of the work is done on the Meteor framework itself and you simply need to ask Meteor to enable oplog support.

For that, you need to set the following MONGO_OPLOG_URL environmental variable, pointing to the Mongo url of your Replica Set’s local db. Here is an example:

MONGO_OPLOG_URL=mongodb://user:pass@host1:port,host2:port,host3:port/local

Keep in mind that oplog contains changes for all the DBs available in your Replica Set. By contrast, Meteor only tails changes for the db you’ve specified via the MONGO_URL. So MONGO_URL and MONGO_OPLOG_URL should point to the same Replica Set.

How might OpLog make thing worst?

Even though oplog makes your Meteor app faster in general, it might make things worst. In its current version, Meteor is tailing all the data changes happening in your DB. So, Meteor will receive all the data changes for the db whether you really need them or not. Because of that, if you have a lot of writes in your DB that do not trigger any observers you might be in trouble. Some examples of this are offline pre-aggregations and writes from other apps. Right now, the only available solution is to move those collections into a separate database until Meteor implements a fix.

How can we optimize our queries for Oplog?

Normally, most of your queries can work with oplog without additional steps. But you will need to do some tweaks for some of them. To learn which queries require additional steps —and how to implement them— you can refer this Kadira Academy article on Optimize Your Meteor App for Oplog

How can we deploy a replica set?

Deploying a production- ready MongoDB Replica Set is not that simple. And if you’ve successfully deployed a Replica Set, managing it is not a trivial task. Use some cloud solutions like compose.io or mongolab to deploy your Replica Set. They offer very affordable plans now. If you are still looking to deploy on your own, I recommend following official MongoDB tutorials on Replica Set deployment.

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License