The Beginner's Guide to Bundler and Gemfiles

While learning Ruby, you will inevitably come across Bundler and Gemfiles. You will see instructions to add things to the Gemfile and run bundle commands, but they won't always explain how it works, and how to fix any issues you might run into. Understanding how Bundler and Gemfiles work together will make you feel more comfortable figuring things out on your own when you feel stuck. Using a real example of documentation that has confused beginners, we'll learn about Bundler and Gemfiles while working through the errors.

One of the ways you can publish a website generated by Jekyll for the world to see is by using GitHub Pages. GitHub documents its features on its GitHub Docs site, and in there you can find an article for creating a GitHub Pages site with Jekyll.

Depending on when you're reading this post of mine, the GitHub article might no longer contain the confusing instructions that we'll be going over. That's because I submitted a pull request to fix the article. Here's a screenshot of steps 7 through 10 from before my changes:

GitHub docs before my changes

At this point, the article assumes you already have a working Ruby development environment with Jekyll and Bundler installed. If you want to follow along in your terminal, the easiest way to get set up is to use my script, which automatically installs Bundler. To install Jekyll, run this command in your terminal:

gem install jekyll

To try things out in the terminal, I keep a folder called playground to differentiate it from real projects. Let's create a new bundler-tutorial folder there to try the commands from the GitHub article:

cd ~
mkdir playground && cd playground
mkdir bundler-tutorial && cd bundler-tutorial

Now let's try running the first command in the GitHub article:

bundle exec jekyll 3.9.0 new .

Uh oh, we get this error:

Could not locate Gemfile or .bundle/ directory

Bundler cannot work without a file called Gemfile, which is why it's complaining. What does Bundler do? Why does it need a Gemfile? Here's the first sentence on the Bundler site:

Bundler provides a consistent environment for Ruby projects by tracking and installing the exact gems and versions that are needed.

In other words, Bundler lets you organize and manage gems separately for each of your Ruby projects. For example, the gems you need for a Rails app will be different from the gems for a Jekyll site. And if multiple projects use the same gem, they might require different versions of the gem. It also makes it much easier to share your project with others. Without Bundler, if multiple people were working on the same project, they would have to coordinate the installation of gems on their computers. Every time Sally wants to add a new gem, or upgrade an existing one to a new version, she would need to tell Harry to manually install or upgrade that gem as well. Working on multiple projects that require different versions of gems would be a nightmare.

The way Bundler allows us to keep gems from different projects separate from each other is by having a file called Gemfile within each project that lists all the gems needed by that project. We can either create that file manually, or let Bundler do it for us:

bundle init

This will create a file called Gemfile with the following contents:

# frozen_string_literal: true

source "https://rubygems.org"

git_source(:github) {|repo_name| "https://github.com/#{repo_name}" }

# gem "rails"

The source line tells Bundler to fetch gems from Rubygems, which is the official site where Ruby gems are published. The git_source line provides a shortcut to make it easier to point to gems on GitHub instead of Rubygems. This is not relevant for us in this guide. I might go over it in the future in a more advanced guide.

To start adding gems to your Gemfile, you can either do it manually by typing the name of the gem within the Gemfile, following this format:

gem "name of gem"

For example:

gem "jekyll"

Or you can use Bundler commands in the terminal:

bundle add jekyll

The advantage of using bundle add is that it automatically adds the version number to the Gemfile, and it also automatically installs the gem. With the manual route, you would need to run bundle install after making any changes to the Gemfile.

Now that we have a Gemfile with Jekyll added to it, we can try that first command again:

bundle exec jekyll new .

Note that I didn't include the version number this time because that's a documentation error. We can't add a version number after jekyll because Jekyll is expecting a subcommand after jekyll, and 3.9.0 (or any other Jekyll gem version number) is not a valid subcommand. To see valid subcommands, we can use the Jekyll help subcommand:

bundle exec jekyll help

We can also use the --help option:

bundle exec jekyll --help

Most command line programs have a --help option, which is a good way to learn more and troubleshoot.

Let's go back to the command we wanted to run:

bundle exec jekyll new .

Oh no! Another error!

Conflict: /Users/moncef/playground/bundler-tutorial exists and is not empty.
          Ensure /Users/moncef/playground/bundler-tutorial is empty or else
          try again with `--force` to proceed and overwrite any files.

What's going on here? Some Ruby gems that provide a lot of functionality, like Rails and Jekyll, have a new subcommand that can quickly generate a new Rails app or a new Jekyll site with all the various required files already filled out for you. One of those files that is automatically generated is the Gemfile since Rails apps and Jekyll sites require specific gems. When you create a brand new Jekyll site, it assumes that you want to create it in a brand new empty folder. But because we added our own Gemfile with bundle init earlier, Jekyll is being helpful and warning us that the bundler-tutorial folder is not empty. To allow Jekyll to overwrite our original Gemfile with the one it generates, we can pass in the --force option, like this:

bundle exec jekyll new . --force

The period after new means the current directory. If we're already inside the folder where we want to generate the Jekyll site, then we use the period to tell Jekyll to install it where we already are. Otherwise, if we were in the playground directory, we could pass in the name of the new folder we want to create, and it would both create the folder and generate the site in it. For example:

cd ~/playground
jekyll new jekyll-test

Wait a minute! What was the point of using Bundler if generating a new Jekyll site is going to overwrite our Gemfile anyway? That's a great question. I have no idea why that is even mentioned in the GitHub docs, which is why I proposed to simplify step 7 to just jekyll new . in my pull request.

So when does it make sense to start a new Ruby project with Bundler? When you are writing it from scratch, or using gems that don't generate their own Gemfiles.

What about Jekyll or Rails? If we shouldn't use Bundler to generate new projects with them, does that mean Bundler isn't needed to continue building the Rails app or Jekyll site? No, you will definitely use Bundler going forward. That is how you continue to manage the gems by installing new ones or upgrading existing ones. The benefit of having the Gemfile in the project is that anytime Sally makes a change to the Gemfile, all that Harry needs to do is run bundle install, and they will both be using the same gems with the same versions.

Now we can move on to step 8 in the GitHub article. It says to open the Gemfile that was created (when we ran jekyll new .) and to follow the instructions in the Gemfile's comments to use GitHub Pages, and then the article has a screenshot of the section of the Gemfile related to GitHub Pages. This might be clear if you are experienced, but not for beginners. Some people didn't understand that they had to make changes to the Gemfile, or didn't know how to, and they tried to run bundle update github-pages that is mentioned in the Gemfile screenshot. This results in this error:

Could not find gem 'github-pages'.

Similarly, running bundle install will not install the github-pages gem because it's not enabled by default in the Gemfile. In Ruby, any code that is preceded with a # is considered a comment, and will not be run. When the instructions say to "uncomment the line below", they mean to remove the # so that the line looks like this in the Gemfile:

gem "github-pages", group: :jekyll_plugins

And then step 9 says to specify a version number for github-pages. As of today, the latest version, which you can always check on Rubygems, is 209, so we update the line to look like this:

gem "github-pages", "~> 209", group: :jekyll_plugins

In addition, the instructions say to remove the "gem "jekyll"" above. Why is that? Don't we need the jekyll gem to run a Jekyll site? Well, let's see what happens if we don't remove that line. Now that both the github-pages and jekyll gems are enabled in our Gemfile, let's try to install github-pages:

bundle install

We get a conflict!

Fetching gem metadata from https://rubygems.org/..........
Fetching gem metadata from https://rubygems.org/.
Resolving dependencies...
Bundler could not find compatible versions for gem "jekyll":
  In snapshot (Gemfile.lock):
    jekyll (= 4.2.0)

  In Gemfile:
    jekyll (~> 4.2.0) x86_64-darwin-19

    github-pages (~> 209) x86_64-darwin-19 was resolved to 209, which depends on
      jekyll (= 3.9.0)

Running `bundle update` will rebuild your snapshot from scratch, using only
the gems in your Gemfile, which may resolve the conflict.

What this means is that the github-pages gem depends on a specific version of the jekyll gem. It has to be exactly 3.9.0, which is what the = sign means. But we also have a line in the Gemfile that says that we want any version of jekyll that is at least 4.2.0, but less than 4.3. That is what the ~> sign means. If it said ~> 4.2, that would mean any 4.x version greater than or equal to 4.2. The technical term for ~> in Ruby is the "pessimistic operator".

Another thing we can learn from this is that a gem listed in the Gemfile is not necessarily a single gem. Gems often depend on other gems, so when you are installing Jekyll, you are also installing a bunch of other gems. You can see the full list of gems required by your Ruby project by looking at the file called Gemfile.lock, which gets automatically generated and updated when you install and update gems with Bundler.

Since the github-pages gem only works with Jekyll version 3.9.0, we can't specify a greater version, which is why we need to remove the gem "jekyll" line from the Gemfile in order to install the github-pages gem. As an aside, Jekyll 4.0 was released over a year ago, but the github-pages gem still hasn't been updated to be compatible with the latest Jekyll versions, so I wouldn't recommend it as a way to publish your Jekyll site on GitHub Pages.

There are better options, as I've written in my step-by-step guide that shows how to use the latest version on Jekyll with GitHub Actions.

OK, so let's remove the gem "jekyll" line from the Gemfile, or just add a # at the beginning of the line to disable it, then let's try to bundle install again.

Drat! More conflicts:

Bundler could not find compatible versions for gem "terminal-table":
  In snapshot (Gemfile.lock):
    terminal-table (= 2.0.0)

  In Gemfile:
    github-pages (~> 209) x86_64-darwin-19 was resolved to 209, which depends on
      terminal-table (~> 1.4)

    minima (~> 2.5) x86_64-darwin-19 was resolved to 2.5.1, which depends on
      jekyll (>= 3.5, < 5.0) x86_64-darwin-19 was resolved to 4.2.0, which depends on
        terminal-table (~> 2.0) x86_64-darwin-19

The jekyll gem is still being resolved to 4.2.0 because that's what was initially installed when the Jekyll site was first generated. You can verify this by looking at Gemfile.lock. To fix this, we can run bundle update like the Bundler error message suggests.

Now we should be able to run and view our Jekyll site:

bundle exec jekyll serve

Visit http://localhost:4000 and you should see your site.

Why bundle exec jekyll serve, and not just jekyll serve? By prepending bundle exec to any commands, we ensure that we're using the specific version of the gem that is defined in our Ruby project. This is why you can have multiple versions of the same gem installed on your computer, yet still be able to use specific versions within different projects without worrying about conflicts. In this case, we are using version 3.9.0 of Jekyll, but if we run jekyll serve, it will use the version that we installed at the beginning of this guide when we ran gem install jekyll. At the time of this writing, the latest version is 4.2.0. You can tell which version it's trying to use by looking at the error:

$ jekyll serve
Traceback (most recent call last):
  10: from /Users/moncef/.gem/ruby/2.7.2/bin/jekyll:23:in `<main>'
   9: from /Users/moncef/.gem/ruby/2.7.2/bin/jekyll:23:in `load'
   8: from /Users/moncef/.gem/ruby/2.7.2/gems/jekyll-4.2.0/exe/jekyll:11:in `<top (required)>'
   7: from /Users/moncef/.gem/ruby/2.7.2/gems/jekyll-4.2.0/lib/jekyll/plugin_manager.rb:52:in `require_from_bundler'
   6: from /Users/moncef/.gem/ruby/2.7.2/gems/bundler-2.2.1/lib/bundler.rb:149:in `setup'
   5: from /Users/moncef/.gem/ruby/2.7.2/gems/bundler-2.2.1/lib/bundler/runtime.rb:26:in `setup'
   4: from /Users/moncef/.gem/ruby/2.7.2/gems/bundler-2.2.1/lib/bundler/runtime.rb:26:in `map'
   3: from /Users/moncef/.gem/ruby/2.7.2/gems/bundler-2.2.1/lib/bundler/spec_set.rb:148:in `each'
   2: from /Users/moncef/.gem/ruby/2.7.2/gems/bundler-2.2.1/lib/bundler/spec_set.rb:148:in `each'
   1: from /Users/moncef/.gem/ruby/2.7.2/gems/bundler-2.2.1/lib/bundler/runtime.rb:31:in `block in setup'
/Users/moncef/.gem/ruby/2.7.2/gems/bundler-2.2.1/lib/bundler/runtime.rb:302:in
`check_for_activated_spec!': You have already activated i18n 1.8.5, but your
Gemfile requires i18n 0.9.5.
Prepending `bundle exec` to your command may solve this. (Gem::LoadError)

Again, Bundler is helpful by recommending to prepend bundle exec.

I hope this was a useful introduction to Bundler and Gemfiles. If anything wasn't clear, please let me know and I will try to explain it better. Next, you might want to read about how Gemfile.lock works and why it's important.