Merging Dynamic and Static Sites

Over the past year or so I’ve been fascinated by static site generators. I have a long history of working with dynamic web sites, and static sites were an intriguing change of perspective for me. While they certainly can’t handle every situation web developers will encounter, they provide a powerful alternative in some cases and offer one of the best benefits a developer can hope for – simplicity.

As nice as they are, one of the issues I’ve run into with SSGs (Static Site Generators) is that they still require some level of technical knowledge to use. Your clients may appreciate the fact that their infrastructure is much simpler with a SSG and their deployment costs are next to nothing, but when they come to you asking to add their latest super exciting press release to the site, you may find yourself wishing you had a ‘traditional’ dynamic web app with a proper CMS (Content Management System).

This lack of suppport for non-technical users is probably the biggest issue SSGs face. There are a few companies now offering services to help mitigate this concern. Cloudcannon, for example, offers a web app front-end to a Jekyll SSG back-end.

I thought it might be fruitful to explore building such a solution myself. In theory, it should be possible to build a CMS that can output to a static site and then automate the act of publishing. You can combine the ease of use for the client when dealing with editing tasks, while still keeping the benefits of a static site for the public site. You can even place the CMS behind a firewall to completely lock down who has access to the data.

If this works, you can truly get the benefits of both worlds! Let’s examine how a solution like this could be built.

Step One – Building the CMS

To begin, let’s create an ultra simple CMS system to handle the basic CRUD (Create, Read, Update, Delete) tasks for our data. For my proof of concept, I decided on the following features:

  • Node.js for the app server.
  • MongoDB for the database. I used mongolab to host the database to make it easier. Obviously, if you are keeping everything in house, you would run Mongo locally. My decision to use mongolab was completely arbitrary. I had never heard of it before and wanted to check it out. (And folks curious – it is pretty darn cool!)
  • For the sake of the demo, I skipped user authentication. Basically, if you can hit the web app, you can use it. I also skipped form validation because this is a demo and I’m lazy.

Our demo is a simple blog. Each blog entry will consist of a title, a body, and a publication date that is automatically created when the content is written. Since this isn’t an article about building Node.js apps, I’m simply going to share a few screen shots of the app in action so you can get an idea as to how it works. If you want to see the initial version of the app, check out the “cmsapp initial” folder in the zip file download.

First, the home page, which simply lists articles:

List of entries

As you can see, you’ve got a button to add a new entry as well as links to edit and delete. (And again, I’m keeping this app as simple as possible. A ‘proper’ CRUD form here would use pagination and a nice little search box to help find older entries.)

When you add a new entry or edit, you get a basic form:

Entry form

And that’s it. You could add more types of data of course. For example, maybe I want to let the client edit blocks of text that relate to various pages on the site. That’s just more CRUD form fields and wouldn’t be difficult to add to the existing demo.

The point is – we now have our simple CMS for the client to come in and edit content as they see fit. We could expand on this to allow for “draft” status, different properties for blog entries, and the like. We can also host this on their own network or externally if we actually added login security to the app.

Step Two – Designing the Static Site

We have numerous choices for which SSG we can use. Personally, my two favorites are HarpJS and Jekyll. If these don’t float your boat, you’re welcome to peruse a list of the nearly 400 other available options.

I think Jekyll is the best I’ve encountered yet, but for this article I’m going to use HarpJS. For one, HarpJS is somewhat simpler than Jekyll. Secondly, Harp actually supports running within a Node.js application as well. That makes it appealing for me as I can directly integrate it within my CMS.

I began by creating a new Harp site. While Harp can create a site with a boilerplate template, it works just fine with any plain directory. I simply made a folder, “forharp,” and began creating my site there. I’m not going to spend a lot of time explaining the particulars of Harp (you can find out more at their documentation site), but I created a simple index page that would display all the blog entries with link to each particular article. Harp supports Jade and EJS, and since I believe that Jade is the template language of the devil, I naturally chose EJS. Here’s the home page:

<h1>Blog</h1>

<% for(articleId in public.articles._data) { %>
    <%
        article = public.articles._data[articleId];
        //lame date formatting
        published = article.publishedDate.split("T")[0];
    %>
    <p>
    <a href="/articles/<%= articleId %>.html"><%= article.title %></a> 
    - <%= published %>
    </p>
<% } %>

This is not the most elaborate home page, but hopefully you get the idea. I’ve got a header and then I loop over all the articles and create links.

Harp supports a basic metadata system where you can use flat files of JSON data to represent your site content. In my case, I know the data from my CMS, blog entries, will be stored in a simple JSON file. Harp will read that JSON and make it available to my template. While creating this demo, I simply hard coded my JSON. Here is an example:

{"test":
    {"title":"test 5",
    "publishedDate":"2015-07-01T12:18:40.308Z",
    "body":"I want to test <strong>html</strong> in my content.\r\n"
    }
}

Later on, when we return to the CMS, we’ll add code to generate this JSON file directly from the MongoDB data.

Next, I added code to render my article. In my articles folder I created a file called test.ejs. The file name is important – “test” in this case matches the key I used in my JSON file above. When Harp sees a match between the URL that you are using and a key in the JSON file, it automatically makes the associated data available to your template.

Again, this is a particular feature of Harp, but most SSGs work in a similar way. My test file then used this template to render the blog entry:

<%
    //lame date formatting
    published = publishedDate.split("T")[0];
%>


<h1><%= title %></h1>
<p>
Published on: <%= published %>
</p>

<%- body %>

In case your curious, the <%- %> block used at the end there is a special EJS marker that allows HTML in the output. I expect blog content to contain HTML tags so I need to remember to specifically allow it there. For simple string values, like the title, you can see I used <%= %> instead.

For the final part of my Harp site, I added the layout. By default, Harp will look for layout.ejs (or layout.jade, but only evil spawns of Satan use Jade) and pass along the template content to be rendered within the site. Here’s my template:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
    <title>Basic CMS</title>

    <!-- Bootstrap -->
    <link href="/css/bootstrap.min.css" rel="stylesheet">
    <link href="/css/app.css" rel="stylesheet">

    <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
    <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
    <!--[if lt IE 9]>
      <script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
      <script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
    <![endif]-->
  </head>
  <body>

    <div class="container">
     <%- yield %>

      <% if(current.source != "index") { %>
        <p>
          <a href="/">Home</a>
        </p>
      <% } %>
    </div>

    <!-- jQuery (necessary for Bootstrap's JavaScript plugins) -->
    <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.3/jquery.min.js"></script>
    <!-- Include all compiled plugins (below), or include individual files as needed -->
    <script src="/js/bootstrap.min.js"></script>
  </body>
</html>

For the most part, this is just another Bootstrap template. But make note of this line:

<%- yield %>

This will render the page content inside the template. And then note this block:

<% if(current.source != "index") { %>
    <p>
    <a href="/">Home</a>
    </p>
<% } %>

Harp passes a variable to your templates that represents the current “location” of the page. This is useful for adding links based on where you are in the site. As you see in the code above, I’ve basically said that, if I’m not on the home page, I want a link back to home.

At this point, I’ve got a very basic Harp site with fake content. Using harp serve, I can open it up and ensure things are working ok.

First, the home page, in all its simple glory:

Static site home page

Next the article page:

Static site blog entry

You may not be terribly thrilled by my design skills, but you get the idea.

Integrating Node.js

Now it’s time to integrate my Node.js app with the Harp project. I began by adding a new button to my CMS to start the process:

Generate button

Next, I started building a Node.js app to process this new action.

First, I added Harp to my package.json file and installed it via npm:

{
  "name": "CMSApp",
  "description": "Simple CMS app",
  "version": "0.0.1",
  "dependencies": {
    "body-parser": "^1.13.1",
    "express": "^4.13.0",
    "express3-handlebars": "^0.5.2",
    "mongoose": "^4.0.6",
    "harp":"*"
  }
}

Then in my app.js file, I required the Harp library:

var harp = require("harp");

I then created my route:

app.get('/generate', function(req, res) {

});

To make this work, I needed to do a few things.

  • First, I need to get all my data. That’s pretty simple with the Mongoose library I’m using to integrate with MongoDB.
  • Once I have my data, I need to make a new JSON file that represents my content.
  • I also need to make a physical file for each particular blog entry. Harp requires this so that it can generate a final static version of the site.
  • I’ll also need to clear out any old articles. Since my CMS lets the client delete entries, I want to ensure I remove things from the public web site.
  • The last thing I’ll do is ask Harp to generate a static site from the templates.

So let’s break this down bit by bit. First, we’ll clean the articles folder:

//clear out forharp/articles
//credit: http://stackoverflow.com/a/19145894/52160
fs.readdirSync('./forharp/public/articles/').forEach(function(fileName) {
    //don't erase the template
    if(fileName != "_article.ejs") fs.unlinkSync('./forharp/public/articles/'+fileName);
});

You’ll notice I’m deleting everything but a file called _article.ejs. I’ll get to that in a few second.

Next, let’s fetch all the articles:

BlogEntry.find(null, null, {sort:{publishedDate:-1}}, function(err, entries) {
...

Once we have them all, we can then loop over the data. From the list above, we need to do two things: help create one JSON file for all the data and make a physical file. The physical file for the article entry is what defines how the blog entry looks. Instead of writing out HTML from my Node.js app, I decided to make use of a feature in HarpJS called partials.

Partials are simply a way for a file to include another. I took my Article template code from the listing above and put it in _article.ejs. When generating static sites, Harp will not render any file that begins with an underscore. By using the partial to handle my layout, I can tweak how my static site looks without having to mess around with my Node.js code. Now you understand why our “delete everything in the articles folder” code had a simple conditional there to avoid deleting the template.

I also needed a way to name my files. For that, I used the title of the blog entry minus spaces and non-alpha-numberic characters.

Finally, I added the article data to an object (articleData) that I’d then write out to _data.json for Harp to use when generating the site. Here is the complete code behind everything I described above.

var content = "<%- partial('_article') %>";

for(var i=0;i<entries.length;i++) {

    //write out one article per entry

    //my slug is based on my title
    var slug = entries[i].title.replace(/ /g, "_");
    slug = slug.replace(/[\W]/g,"");

    //filename is forharp/public/articles/slug.html
    fs.writeFileSync('./forharp/public/articles/'+slug+'.ejs', content);

    //save for data
    articleData[slug] = {title:entries[i].title,publishedDate:entries[i].publishedDate,body:entries[i].body};
}

//write out the _data.json
fs.writeFileSync('./forharp/public/articles/_data.json', JSON.stringify(articleData));

The net result of all this is that my folder containing my Harp site now contains content based on my MongoDB data. All I have to do is ask Harp to compile it. That takes one line:

harp.compile('./forharp', '../forharp_output', function() {
...

This is an asynchronous process so I have to use a callback to handle the result. But let’s stop for a minute and consider what we have now.

Our CMS lets the client add, edit, and delete content. We built a Harp site that is then dynamically populated by that data and is then converted to flat HTML files by Harp. As you can probably tell from the code snippet above, the folder “forharp_output” will contain the site.

Step Three – Publication

Ok, time to wrap this baby up. We’ve got our static site and now we need to get it online. There’s numerous options for hosting static sites.

One of the coolest new ways of deploying and hosting static sites is Surge. Surge is a CLI that lets you publish a folder to a public web site in seconds. It is really cool, but unfortunately it doesn’t work well in a Node.js app yet. I’ve filed an ER with them to make this process easier in the future.

For my demo, I decided to use Amazon S3. S3 supports static web site hosting so it is a good choice for deployment. I also found a great S3 library to make the process even easier. For information on how to use S3 for static sites, check out Amazon’s docs on the process.

I made my bucket and then got my access key and my secret key from the Amazon console. This is required by the S3 library I’m using. Luckily, the library is fairly simple to use. You begin creating a client with your credentials:

var s3 = require("s3");
var client = s3.createClient({
  s3Options: {
    accessKeyId: credentials.s3.accessKeyId,
    secretAccessKey: credentials.s3.secretAccessKey
  },
});

And then you can use the API to sync a folder to a bucket:

var params = {
    localDir: "./forharp_output",
    deleteRemoved: true, 
    s3Params: {
        Bucket: "dynstatic1.raymondcamden.com",
        Prefix: ""
    },
};

var uploader = client.uploadDir(params);
    uploader.on('error', function(err) {
    console.error("unable to sync:", err.stack);
});

uploader.on('end', function() {
    console.log("done uploading");
    res.redirect('/?published=1');  
});

As you may notice, I’m not doing a good job of reporting errors to the client, but in general, if things work well, it is a fairly simple process.

So now with one button, the entire process of converting the database into flat template files, converting the templates to flat HTML, and then publishing, can be done by the client! Below you can see a video of this in action:

You can download a zip file with the sample app here. You can also see the live site on my S3 bucket at http://dynstatic1.raymondcamden.com.s3-website-us-east-1.amazonaws.com/

Conclusion

Obviously the demo and code here is somewhat rough, and should be considered simply a proof of concept. However, I think the idea is sound. In cases where you really want the simplicity and performance of static pages in production but need to offer a user-friendly editing interface for the client, the combination of the both a dynamic and static solution can be a powerful one.

Header image courtesy of newtonapple

Comments

  • Jen Looper

    Really interesting article, Ray! Now I want to take a look at Surge. And I love this: “Jade is the templating language of the devil” – no joke!!

    • Brett Andrews

      Could you explain your hatred of Jade for me? I’ve never used the templating side of it but I think it’s cleaner than writing HTML.

      • Personally, I don’t like Jade, because “now you have two problems”. That is, first you need to know HTML and then you need to know Jade’s special DSL – and now you (and everyone else on your team) have the quirks of both.

      • I think Les explains it well. Other template languages “extend” HTML, by adding looping, conditionals, and basic variable substitution. Jade replaces HTML too and to me – just makes it too darn confusing.

      • astroanu

        template engines in common are just making things over complicated. Learning another syntax is too much work i guess. Unless you’re using a frontend frameworks like angular i dont think a template layer is that necessary. For example blade in laravel does automatic escaping but when it comes to debugging, blade is terrible.

        • I guess we have to agree to disagree. I think template engines help solve a common issue – dynamically generating HTML from data.

          • astroanu

            PHP is a template engine

  • webhooked

    Might want to look into the Webhook CMS. Webhook.com

    We’re still pretty new but we largely solved this problem. Code is all MIT open source. Essentially the site runs off of a grunt task locally and connects to a remote Firebase. The deployed site is completely static and because the CMS is purely Firebase data, users can access and manipulate it live behind auth through a one-page Ember app that is also served statically. You get the best of all worlds.

  • Very cool. As time passes static is looking like the only way to manage front-ends. Great article.

  • Pingback: Dew Drop – July 17, 2015 (#2056) | Morning Dew()

  • The more things change, the more they stay the same 🙂 “Bake, Don’t Fry” from the late aaronsw in 2002 – http://www.aaronsw.com/weblog/000404

    Also, last fall I switched over my blog from WordPress to a set of Gulp tasks that publish to S3. It’s a pretty fast & efficient process. No pesky CMS waiting to be hacked out there on my server. http://blog.lmorchard.com/2014/10/20/static-blog-generation-with-gulp/

  • The more things change, the more they stay the same 🙂 “Bake, Don’t Fry” from the late aaronsw in 2002 – http://www.aaronsw.com/weblog/000404

    Also, last fall I switched over my blog from WordPress to a set of Gulp tasks that publish to S3. It’s a pretty fast & efficient process. No pesky CMS waiting to be hacked out there on my server. http://blog.lmorchard.com/2014/10/20/static-blog-generation-with-gulp/

  • The more things change, the more they stay the same 🙂 “Bake, Don’t Fry” from the late aaronsw in 2002 – http://www.aaronsw.com/weblog/000404

    Also, last fall I switched over my blog from WordPress to a set of Gulp tasks that publish to S3. It’s a pretty fast & efficient process. No pesky CMS waiting to be hacked out there on my server. http://blog.lmorchard.com/2014/10/20/static-blog-generation-with-gulp/

  • Brett Andrews

    Could you explain your hatred of Jade for me? I have never used the templating side of it but I think it’s cleaner than writing HTML.