Meteor - Mongodb

meteor
mongodb

Designing MongoDB structure

https://kadira.io/academy/optimize-your-app-for-oplog/
http://stackoverflow.com/questions/39002192/is-meteor-using-the-mongo-oplog
https://blog.compose.io/meteor-the-oplog-and-elastic-deployment/
http://blog.mongolab.com/2014/07/tutorial-scaling-meteor-with-mongodb-oplog-tailing/
http://docs.mongodb.org/manual/administration/replica-set-deployment/

How does Meteor use MongoDB?

Meteor is built top of the MongoDB and Meteor is totally depending on MongoDB at this moment. But this can be changed in the future. But MongoDB is not a real-time database; but Meteor is realtime. Meteor makes MongoDB realtime using two techniques as shown below.

  1. Polling MongoDB every ~10 secs
  2. Using MongoDB oplog

Polling is very expensive operation and that’s why Meteor includes another option (using oplog). But it needs some additional setup and is not possible with shared MongoDB hosting services.

What is the default port that Meteor use for its Mongo database in the development environment?

3001. Meteor comes with its own instance of MongoDB. Each time you start your server with the 'meteor run' command, a dedicated database server is also started and listens for connections on port 3001. By default, Meteor use this instance as its database engine and stores all content inside a database named 'meteor'. There is no need to define any database connection string, but you can use environment variables like MONGO_URL to your Meteor server to another database instance.

How can we setup oplog tailing?

The oplog is stored in a system database called 'local'. If you haven't defined any replica sets, you won't have a collection called oplog.rs. To initialize the collection you must define a replica set, but you don't need to add multiple members, you can use a single primary member only. Each mongod instance has its own configuration. First open the mongodb configuration file /etc/mongodb.conf. At the very end of the file, add two lines:

replSet=rs0
oplogSize=100

If you're running multiple mongod processes on the same machine, make sure you're editing the correct config file. Next, restart the mongod process and open a mongo shell. You can use either a tool such as Robomongo or the command line. Once you're connected, switch to using the local database:

use local

The next step is to initialize the replica set and thereby enable the oplog:

rs.initiate({
  _id: "rs0",
  members: [{
    _id: 0,
    host: "localhost:27017"
  }]
});

You can always check the status of the current replica set by using rs.status() and view the full configuration and member list using rs.config(). After a successful initialization, these commands should show a replica set with a single member. As you can see, two additional collections will be created (oplog.rs and system.replset). Also, the shell prompt will change to reflect the replica set name and its member status (rs0:PRIMARY).

MongoDB now automatically tracks all write operations in the oplog.rs collection. Once the specified size is reached, it'll purge old entries.

How can we set up an oplog user?

By default, MongoDB doesn't require a user to authenticate. In such an environment, you can also access the oplog without credentials, so you could skip this step. But in production environments you should add users to provide a mean of access control. For the purpose of tailing the oplog, you need a dedicated user that's allowed to access the 'local' database, which is where the oplog.rs collection is.

Even though the oplog user has access to the local database, it's technically created inside the 'admin' database. This is because the 'local' database doesn't allow any users to be created inside it. Create an oplog user with the following command:

db.createUser({
  user: 'oplog',
  pwd: 'password',
  roles: [
    { role: 'read', db: 'local' }
  ]
});

What is a Replica Set?

Replica Set is MongoDB’s replication strategy to address high availability. It’s a good idea to deploy a Replica Set for any production MongoDB deployment. Replica Set comprises one or many MongoDB servers (nodes) replicating the same set of data. It is recommend to run a Replica Set of at least 3 nodes. But a Replica Set should not contain more than 12 nodes.

One node acts as the Primary and others are known as Secondaries. Primary election is fully automatic and if the current Primary goes down, a new “Primary” will be elected from the Secondaries. But there are many more options to customize the default behavior. Write operations can be directed to Primary only, but you can read from any node. If you read from a Secondary, you may get data that is inconsistent with the Primary data.

Replica Sets are used mainly for high availability. But some people use Replica Sets for backups and doing background analytic jobs using Secondaries.

What is the recommended minimum number of nodes in a Replica Set?

It is recommend to run a Replica Set of at least 3 nodes. But a Replica Set should not contain more than 12 nodes.

What is the recommended maximum number of nodes in a Replica Set?

It is recommend to run a Replica Set of at least 3 nodes. But a Replica Set should not contain more than 12 nodes.

What is Oplog?

Oplog is the heart of a MongoDB Replica Set. It’s a stream of all the write operations happening inside the Primary. Actually, oplog is a capped collection named oplog.rs, located on the local database. It’s created for you automatically when initializing the Replica Set. Capped collections can be tailed (with queries) and monitored for new additions. So, it can be used as a stream. That’s what exactly Secondaries do. They are tailing oplog on the “Primary” and make a copy of that while applying write operations(entries in the oplog) into the data that Secondary maintains. That’s how Secondaries keep updated with the Primary.

How does Meteor use Oplog tailing?

Meteor’s default observe driver is based on polling and that makes Meteor super-slow and causes it to consume a lot of server resources. As a solution for that, Meteor uses oplog to detect data changes and trigger observers. Using oplog to trigger observers is very efficient compared to the polling approach.

Meteor acts like a Secondary in the sense that it is tailing the oplog of the Primary. Meteor keeps a cache of data inside the memory and applies changes while triggering observers.

Meteor won’t cache all the data comes through the oplog, just the data relevant to observers. Actually, most of the work is done on the Meteor framework itself and you simply need to ask Meteor to enable oplog support.

For that, you need to set the following MONGO_OPLOG_URL environmental variable, pointing to the Mongo url of your Replica Set’s local db. Here is an example:

MONGO_OPLOG_URL=mongodb://user:pass@host1:port,host2:port,host3:port/local

Keep in mind that oplog contains changes for all the DBs available in your Replica Set. By contrast, Meteor only tails changes for the db you’ve specified via the MONGO_URL. So MONGO_URL and MONGO_OPLOG_URL should point to the same Replica Set.

What may impact performance of Meteor oplog tailing?

Even though oplog makes your Meteor app faster in general, it might make things worst. In its current version, Meteor is tailing all the data changes happening in your DB. So, Meteor will receive all the data changes for the db whether you really need them or not. Because of that, if you have a lot of writes in your DB that do not trigger any observers you might be in trouble. Some examples of this are offline pre-aggregations and writes from other apps.

Right now, the only available solution is to move those collections into a separate database until Meteor implements a fix.

How can we determine the URL for the mongo database?

  1. Start meteor if it is not already running: meteor
  2. Launch the meteor mongo shell: meteor mongo

We will get something like:

MongoDB shell version: 2.2.1
connecting to: 127.0.0.1:3001/meteor

If our app was deployed to the meteor.com free server, we need to:

  1. cd yourapp
  2. meteor login
  3. meteor mongo yourapp.meteor.com —url

How can we backup our mongo database?

mongodump -h 127.0.0.1 --port 3001 -d meteor

Dumps will be located under the dumps folder in the folder you executed the above command.

mongodump -h production-db-b1.meteor.io --port 27017 --username client-ID  --password password-3be8-f6c5-50a9-password -d yourapp_meteor_com

This backup the entire remote database into a default dump/ folder.

How can we restore our mongo database from a backup dump?

mongorestore -h 127.0.0.1 --port 3001 -d meteor dump/meteor
mongorestore -h 127.0.0.1 --port 3001 -d meteor dump/<some directory>
Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License