Meteor - SEO - Spiderable


What are the things that we need to do?

  1. Add a meta tag
  2. Add the required packages

What is the meta tag that we need to add to make spiderable package work?

<meta name="fragment" content="!">

What do we need to append to the end of the URL?


How can we verify that the spiderable package work correctly?

The meta tag tell the crawler to append "?_escaped_fragment_=" to the end of the url. Add that string to the end of the URL. You won't see much difference, but a look at the code will show nice proper html5.

What packages do we need to install to work with spiderable?

meteor add lookback:seo
meteor add ongoworks:spiderable
meteor add dfischer:phantomjs
meteor add gadicohen:sitemaps

How can we add a sitemap?

Create server/sitemaps.js which contains something like:

sitemaps.add('/sitemap.xml', function() {
  // required: page
  // optional: lastmod, changefreq, priority, xhtmlLinks, images, videos
  return [
    { page: '/x', lastmod: new Date() },
    { page: '/y', lastmod: new Date(), changefreq: 'monthly' },
    { page: '/z', lastmod: new Date().getTime(), changefreq: 'monthly', priority: 0.8 },
    { page: '/pageWithViedeoAndImages',
      images: [
        { loc: '/myImg.jpg', },        // Only loc is required
        { loc: '/myOtherImg.jpg',      // Below properties are optional
          caption: "..", geo_location: "..", title: "..", license: ".."}
      videos: [
        { loc: '/myVideo.jpg', },      // Only loc is required
        { loc: '/myOtherVideo.jpg',    // Below properties are optional
          thumbnail_loc: "..", title: "..", description: ".." etc }
    { page: 'lang/english', xhtmlLinks: [
      { rel: 'alternate', hreflang: 'de', href: '/lang/deutsch' },
      { rel: 'alternate', hreflang: 'de-ch', href: '/lang/schweiz-deutsch' },
      { rel: 'alternate', hreflang: 'en', href: '/lang/english' }

In the above code, our function returns an array of object, each object represent a page. The first page has the URL of /x and the last modification date specified. The second page has changefreq of monthly. The /pageWithViedeoAndImages page has images and videos on the page so those resources should be included in the sitemap. The last page has the URL of lang/english, and it has the xhtmlLinks attribute which is an array of alternative languages.

You can call sitemaps.add() as many times as you like. More details on the format below. Note that the url is automatically added to the data served from /robots.txt (since 0.0.4, using the robots.txt smart package).

The above example uses a brand new Date() for every link. This is just for demonstration purposes. Of course you should use the real date of the last page update (updatedAt from the database?). If you always use the current time, Google will penalize you (or at the very least, ignore this field on future crawls).

What URLs can be included in a sitemap?

Note that the location is important. A sitemap can only reference other URLs in its own path or descendant paths. e.g. /sitemap.xml can reference all URLs on the site. /articles/sitemap.xml can only reference other pages in the /articles/ directory/path/route.

Can we call the sitemap.add method multiple times?


How can we implement a sitemap base on a collection?

sitemaps.add('/mw_AllPages_sitemap.xml', function() {
  var resultArray = [], pages = WikiPages.find().fetch();
  _.each(pages, function(page) {
      page: 'read/' +,
      lastmod: page.lastUpdated
  return resultArray;

How can we override Meteor.absoluteUrl with sitemap.config?

Anywhere where a url can be provided, you can provide a "relative URL" (with or without a leading /), and Meteor.absoluteUrl() will be prepended. You can override this by calling sitemaps.config('rootUrl', 'myRootUrl'). For individual links, providing an absoluet URL (beginning with http:// or https://) will avoid this behaviour. URI components are escaped for you.

Why might we need to add swap space for the spiderable package to work?

When you deploy with meteor up it will set up phantomjs for you if you choose to. The problem is that with the cheaper providers, such as the $5 Digital Ocean droplet, or modulus's lower packages, phantom will take up too much memory and crash. To solve this we use the dfischer:phantomjs package and set up a memory swap.

Setting up swap space for web application is usually not a good thing because when it start to swap, we may not be able to access the machine. So, we probably should use a VM with decent amount of memory.

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License