Meteor - Performance

meteor

https://ngrok.com/
http://galaxy-guide.meteor.com/apm-getting-started.html

What is the purpose of Meteor.setTimeout()?

Sending email is typically slow. Inside a Method, we may want to wrap the slow operation inside a Meteor.setTimeout:

followers.forEach(function(follower) {
  Meteor.setTimeout(function() {
    Email.send({
      from: "hello@todoapp.com",
      to: follower,
      subject: "New todo added",
      text: "here's the todo item: " + title
    });
  }, 0);
});

Wrap the code for sending email inside a Meteor.setTimeout(). Then the time taken to send the email will not affect the method's response time.

What is the purpose of Meteor.defer()?

As the name implies, it defers the execution. Here's the method signature:

Meteor.defer(function() {
  // deferred logic goes here
});

Do we have to send email using SMTP?

Not entirely necessary. When we're sending emails with Email.send(), it uses the SMTP protocol behind the scenes. Instead, set the MAIL_URL environment variable before sending emails. Check this gist for more information.

If you are sending a lot of emails, SMTP is not a good protocol, and it will take a lot of time to send emails. But most email providers use their own HTTP APIs to send emails. This way, we can send emails very quickly. Here are a few such HTTP APIs:

  1. MailGun
  2. Gmail
  3. SendGrid
  4. You can also send emails by using a service like customer.io, which allows you to use email templates and gives you better control when sending emails.

How does Meteor process requests?

In any Meteor app, we invoke methods and subscribe to publications from the client. Meteor executes all these in a sequence, so if one method or publication takes a lot of time to process, it will increase the time spent by other methods/publications as well.

Meteor is based on NodeJS, or JavaScript which is basically single-threaded. On a multi-core machines, we can run one process on each core, but to a certain limit, what was mention in the previous paragraph still hold true.

By default, Meteor processes all DDP messages, including method calls and subscriptions for a single client (browser tab), one by one. When it is processing a single message, all the other messages of the processing client are queued. The period of time that a message stays in the queue is known as the wait time.

If we have a method that is slow and we cannot improve it further, we can use this.unblock().

You can use this.unblock() on top of a method call and ask Meteor not to block other DDP messages because of this method. Once Meteor detects the this.unblock() API call, it will pick a DDP message from the queue and start to process it.

For Meteor, any method call or subscription is a DDP message. That's why we use the term "DDP message" to refer to both method calls and subscriptions.

What is the purpose of this.unblock()?

If we have a method that is slow and we cannot improve it further, we can use this.unblock().

You can use this.unblock() on top of a method call and ask Meteor not to block other DDP messages because of this method. Once Meteor detects the this.unblock() API call, it will pick a DDP message from the queue and start to process it.

For Meteor, any method call or subscription is a DDP message. That's why we use the term "DDP message" to refer to both method calls and subscriptions.

If you need to learn more about this.unblock and wait time, please refer to this article.

Unfortunately, we can't use this.unblock() on every method, even though it's technically possible. I highly recommend the following articles if you want to learn more about wait time and this.unblock.

What is the difference between Meteor.defer and this.unblock()?

We use Meteor.defer to execute something after the method completes. It's a kind of background work: it's independent of the execution of the Meteor method. Therefore, the time taken to execute code inside Meteor.defer does not count towards the method's response time.

The code this.unblock() asks Meteor to pick the next DDP message and execute it in a separate fiber, so other methods and publications don't need to wait for this method. That's how this.unblock() helps to reduce the response time of the app.

Unlike Meteor.defer, this.unblock() does not reduce the response time of the running method.

How do we typically invoke subscription?

In Meteor, we normally invoke subscriptions in two places, either on the router or inside templates (or React components). With a router, when a user navigates to a route, Meteor creates a set of new subscriptions. Then when the user navigates to a different page, it destroys the previous set of subscriptions and creates a new set of subscriptions.

With templates (or components), subscriptions will be destroyed at the same time as the templates are destroyed. When new templates are getting created, your app will invoke subscriptions again.

End users usually navigate back and forth a lot. That creates a lot of very short term (short lifespan) subscriptions. The same set of subscriptions will get created and destroyed again and again, wasting time.

To identify the issue, we are going to use Discover Meteor's Microscope project. It's a lightweight clone of HackerNews built with Meteor and using Iron Router as the router.

Watch https://youtu.be/-ZcaZDxwIN4

From the Kadira Debug event stream filter, select "Sub" and "LiveUpdates" events, then play with the app.

As you've identified already, when we are navigating between pages, Meteor sends new data from the server every time. But the client already had that data. We can fix this behavior by simply using SubsManager. SubsManager is a tiny project we've created to cache subscriptions on the client and manage them as a group.

Add SubsManager into the app:

meteor add meteorhacks:subs-manager

Now open the router file of the app. (It is located at lib/router.js). Simply replace all instances of Meteor.subscribe in routes with SubsManager. It's very simple. First, create a SubsManager instance.

var subscriptions = new SubsManager();

Now, replace Meteor.subscribe with subscriptions.subscribe. You don't need to do this for subscriptions defined under Router.configure.

Even though you can use SubsManager instead of Meteor.subscribe, it is not a drop-in replacement. It has a few limitations and known issues:

  1. https://github.com/meteorhacks/subs-manager#limitations
  2. https://github.com/meteorhacks/subs-manager/issues/15
  3. https://github.com/meteorhacks/subs-manager/issues/10

Check out all the issues on GitHub.

Because of the above limitations and issues, you have to select which subscriptions you need to power with SubsManager. In Kadira APM, visit the Pub/Sub Dashboard and sort the publications by "shortest lifespan." All the subscriptions with "short lifespan" and "high SubRate" are candidates for SubsManager.

How can we cause Meteor to use Oplog?

Although Meteor uses MongoDB, it's not a real-time database. Meteor does a lot of tricks to make MongoDB real-time and reactive. By default, Meteor uses a MongoDB driver that polls the db for changes. It works well, but it's not suitable for production use. That's why Meteor came up with another driver that tails the MongoDB oplog.

If you are running a production Meteor app, you should use the oplog driver. Otherwise, you'll face a lot of performance issues, especially with the CPU. To enable oplog support, you have to export the environment variable shown below before starting your Meteor app.

MONGO_OPLOG_URL=mongodb://auth:password@host:port/local

When you are running your app locally with Meteor, it runs with the above environment variable and is oplog ready. But when you deploy your app into *.meteor.com, it's not oplog ready. To learn more about oplog: https://meteorhacks.com/mongodb-oplog-and-meteor.html

Meteor tries to use oplog for most queries (observers), but it can't do it for some of them. As you already know, if you specify a limit without a sort specifier, then Meteor can't use oplog for that query. The solution is to add a sort specifier like the one below:

Meteor.publish('todos', function () {
  return Todos.find({}, {limit: 10, sort: {date: -1}});
});

Learn more about oplog: https://kadira.io/academy/optimize-your-app-for-oplog/

Why should we send less data?

With Meteor, it is very simple to send data to the client: simply create a publication and subscribe to it from the client. As we discussed earlier in the "Subscription Caching" lesson, CPU usage increases as you send more data. Interestingly, your app will send a lot of data when it has more connected clients.

If you send a lot of data from a publication, it not only affects the CPU but slows down your app as well (especially on mobile devices that have lower bandwidth). For this reason, it's very important to send only the data you really need to the client.

How can we send less data?

  1. Field filtering
  2. Server-side transformation
  3. Pagination
  4. Removing unnecessary subscriptions
  5. Change your database structure inside MongoDB and modify your application

How can we do field filtering?

Meteor.publish('getRecentWikis', function () {
  var options = {
    sort: {date: -1},
    limit: 50,
    fields: {topic:1, content: 1}
  };
  return Wiki.find({}, options);
});

In situations like this, we can filter out fields that will be sent to the client. For that, we can use the field filtering functionality of Mongo, which is supported by Meteor. Use the following links to learn about field filtering:

  1. http://docs.mongodb.org/manual/tutorial/project-fields-from-query-results/#projection
  2. https://docs.meteor.com/#/full/fieldspecifiers

What is server-side transformation?

Normally, a single wiki contains over 3,000 characters of content, with the preview having fewer than 200 characters. With the current solution, we are sending some unnecessary data to the client. We can optimize the above publication by generating the summary on the server.

To do that, we can use the transform documents on the server before sending to the client. For that you've got to write a custom publication, as shown below:

// function that transform the document and create the summary field
function transform(wiki) {
  var newWiki = {
    topic: wiki.topic,
    summary: wiki.content.substring(0, 100)
  };

  return newWiki;
}

Meteor.publish('getRecentWikis', function () {
  var sub = this;
  var options = {
    sort: {date: -1},
    limit: 50,
    fields: {topic:1, content: 1},
  };

  // creates the cursor
  var cursor = Wiki.find({}, options);

  // observe the cursor and send changes manually
  cursor.observeChanges({
    added: function (id, doc) {
      // wikis is the collection name (in mongodb)
      sub.added('wikis', id, transform(doc));
    },
    changed: function (id, doc) {
      sub.changed('wikis', id, transform(doc));
    },
    removed: function (id) {
      sub.removed(id);
    }
  });

  // making sure to mark subscription as ready
  sub.ready();
});

Now the client will receive a new field called "summary" that can be displayed directly, rather than generating a summary from the content.

Meteor has an option to transform when using collection.find(). However, this option does not work when you are returning a cursor from a publication.

There are some other ways to achieve the above outcome. For example, you can generate the summary while the user is creating or updating the wiki. For that, you can build the summary inside the client or use something like collections hooks to do it inside the server.

How can we implement pagination?

There are several ways we can paginate. The simplest way is to ask for the number of documents needed via an argument from the publication. This is what our modified code looks like now:

Meteor.publish('getRecentWikis', function (limit) {
  limit = limit || 10;

  var options = {
    sort: {date: -1},
    limit: limit,
    fields: {topic:1, content: 1},
    transform: function (wiki) {
      var newWiki = {
        topic: wiki.topic,
        summary: wiki.content.substring(0, 100)
      };

      return newWiki;
    }
  };
  return Wiki.find({}, options);
});

This is a very simple way to implement pagination on the server side. There are also some packages that provide advanced pagination solutions for your app. Check atmosphere for such projects.

What is https://ngrok.com/?

Not sure. You can also use a service like ngrok to get the actual user experience. If you are using ngrok or something similar, make sure to run your app with the —production flag:

meteor --production

Otherwise, ngrok loads all your individual source files and source maps, which takes a lot of time to load.

How can we reduce the application's initial loading time?

For your Meteor app,

  1. First, load the initial HTML that contains Meteor, all your packages, and your app. That's when you'll start to see the loading screen.
  2. Then the browser will start connecting to the Meteor (DDP) server.
  3. After that, your app on the browser will start to send subscriptions.
  4. Finally, the server will send the data.

That's when the loading screen goes away and you can see the content of your app. It takes some time to complete processes 2-4. That's why there is always a loading screen. You can verify the above with the following network diagram, made with Chrome Dev Tools when loading crater.io. You can see how it's showing a six-second loading time with a decent mobile internet connection (4 Mbps).

The most viable solution for this issue is to send the initial data that is required to render the page along with the initial HTML. Then, just after the initial HTML loads, Meteor can start to render the page, and there will be no loading screen.

That's exactly why we created the Fast Render package. It does a lot of tricks behind the scenes to get your data with the initial HTML and start rendering.

Fortunately, you don't actually need to change your app to integrate Fast Render. Simply add an option to your Iron Router routes, and those routes will be fast rendered. First, add Fast Render to the app:

meteor add meteorhacks:fast-render

Then, open Microscope's router file, located at lib/router.js. Then simply add the fastRender option to the route, as shown below:

Router.route('/', {
  name: 'home',
  controller: NewPostsController,
  fastRender: true
});

You can see some low-level details by enabling Fast Render logs. To do that, paste the following code into the browse console and reload the page.

FastRender.debugger.showLogs();

You can hide logs with:

FastRender.debugger.hideLogs();

To verify that FastRender works, apply the following command in the browser console:

FastRender.debugger.blockDDP();

It will reload the app. You'll see that the initial posts are rendered, but now all the DDP messages are blocked. Now you can turn off Fast Render support by applying following code:

FastRender.debugger.disableFR()

You'll see that there is nothing rendered on the screen. Now, you might wonder how Fast Render magically knows which data to pick and send with the initial HTML. There is no magic. Fast Render tries to understand which subscriptions are used when loading the page and selects data for that subscription on the server side. These are the ways in which Fast Render detects subscriptions:

  1. Subscriptions defined in the route's waitOn function.
  2. Subscriptions defined in the route controller's waitOn function.
  3. Subscriptions defined in the route controller's subscriptions method.

Sometimes, it's not possible for Fast Render to detect subscriptions. Then you have to define them yourself. Microscope's /new route is something like that. This is how we tell Fast Render about subscriptions for that route. Add the following code at the end of the lib/router.js:

if (Meteor.isServer) {
  FastRender.route('/new/:limit?', function (params) {
    this.subscribe('posts', {sort: {}, limit: params.limit})
  });
}

Learn more about FastRender: https://github.com/meteorhacks/fast-render

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License