Looking at Gemfiles

rubygems is the overall ecosystem, which includes the main rubygems.org database of shared packages, and how they depend upon each other. At the heart of it is a gemspec file which describes the package, files included in it, dependencies, and other metadata.

bundler is the tool that helps you download and install specific gems for your application. It uses a Gemfile to define what you want, and then resolves all the dependencies and stores that into a Gemfile.lock file that contains a list of the exact versions that you are developing against.

First, let's create a project, so we are all working off of the same thing. We'll add bundler as normal, and then irb from git directly, so we can see how that changes the Gemfile.lock. So, 3 gems, 2 from rubygems.org, and one directly from github.

1
2
3
4

bundle init
bundle add bundler --version "~>2.1"
bundle add irb --git "https://github.com/ruby/irb"
bundle add gems --version "1.1.1"

Lets also add in a ruby "2.6.6" line that will tell bundler which version of Ruby we want to use. This isn't always used, but let's put it in to flesh out what is generated in the lock file. Feel free to replace this with whatever version is on your local machine.

This gives us a Gemfile that looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# frozen_string_literal: true

source "https://rubygems.org"

git_source(:github) {|repo_name| "https://github.com/#{repo_name}" }

ruby "2.6.6"
# gem "rails"

gem "bundler", "~> 2.1"

gem "irb", git: "https://github.com/ruby/irb"

gem "gems", "1.1.1"

Doing a bundle add adds the specification to the Gemfile, downloads the gems and all of their dependencies, resolving specific versions and downloading those as well. This is specified in the lock file, which has the main sections:

1. GIT which is the first remote source on the list, here telling which commit that we are depending upon specifically
2. GEM which list out the the gems that are installed, with versions
3. PLATFORMS which is for platform information (for native builds)
4. DEPENDENCIES which is what comes out of the Gemfile
5. RUBY VERSION the version of Ruby, including the patch level, that this was bundled with
6. BUNDLED WITH version information about bundler

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

GIT
  remote: https://github.com/ruby/irb
  revision: 9bb6562d0d841fd294e95bbb690e812cbd5090ce
  specs:
    irb (1.2.4)
      reline (>= 0.0.1)

GEM
  remote: https://rubygems.org/
  specs:
    gems (1.1.1)
      json
    io-console (0.5.6)
    json (2.3.1)
    reline (0.1.4)
      io-console (~> 0.5)

PLATFORMS
  ruby

DEPENDENCIES
  bundler (~> 2.1)
  gems (= 1.1.1)
  irb!

RUBY VERSION
   ruby 2.6.6p146

BUNDLED WITH
   2.1.4

In normal usage you would have both a Gemfile and a Gemfile.lock. You'd start up your program using bundle exec which would validate that the right things were installed, load the gems into the Ruby environment, and you'd be good to go.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

  #!/usr/bin/env ruby
  require 'bundler'

  context = Bundler.load

  puts "Dependencies"
  context.dependencies.each do |x|
    printf "%-20s %s\n", x.name, x.requirements_list.to_s
  end

  puts
  context.specs.each do |s|
    code = s.metadata['source_code_url'] || s.metadata['source_code_uri']
    puts "Gem from #{s.source.to_s}"
    printf "%-20s %-10s %-40s %s\n", s.name, s.version.to_s, code, s.homepage
  end

Which yields:

Dependencies
irb                  [">= 0"]
bundler              ["~> 2.1"]

Gem from the local ruby installation
bundler              2.1.4      https://github.com/bundler/bundler/      https://bundler.io
Gem from rubygems repository https://rubygems.org/ or installed locally
gems                 1.2.0                                               https://github.com/rubygems/gems
Gem from rubygems repository https://rubygems.org/ or installed locally
io-console           0.5.6      https://github.com/ruby/io-console       https://github.com/ruby/io-console
Gem from rubygems repository https://rubygems.org/ or installed locally
reline               0.1.4                                               https://github.com/ruby/reline
Gem from https://github.com/ruby/irb (at master@9bb6562)
irb                  1.2.4                                               https://github.com/ruby/irb

So some gems have source_code_uri or source_code_url set, and some don't have it at all though they point to a github page where presumably we can figure out where the code is loaded.

For the case of irb the repo is listed in the remote source itself.

The metadata, homepage, etc, are from the gemspec files.

Let's now look at how to parse a Gemfile.lock of a different project, where we don't have the original Gemfile handy.

When we load in the lockfile using Bundler::LockfileParser we only have the gem name, version, and the source from which to get it. We may or may not have the gemspec on our local machine, and so additional metadata like the homepage, summary, codeurl, etc haven't been loaded yet. Let's first print out what we have and then figure out how to get gem information from the source, name, and version.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

#!/usr/bin/env ruby
require 'bundler'

def describe_lockfile file = Bundler.default_lockfile
  context = Bundler::LockfileParser.new( Bundler.read_file( file ) )
  puts "Bundler version"
  puts context.bundler_version.to_s

  puts
  puts "Dependencies"
  context.dependencies.each do |name,x|
    printf "%-20s %s\n", x.name, x.requirements_list.to_s
  end

  puts
  puts "Gems"
  context.specs.each do |s|
    printf "%-20s %-10s\n", s.name, s.version.to_s
    printf "%-10s %s\n", s.source.class, s.source.to_s
  end
end

describe_lockfile

Bundler version
2.1.4

Dependencies
bundler              ["~> 2.1"]
gems                 ["~> 1.2"]
irb                  [">= 0"]

Gems
gems                 1.2.0     
Bundler::Source::Rubygems rubygems repository https://rubygems.org/ or installed locally
io-console           0.5.6     
Bundler::Source::Rubygems rubygems repository https://rubygems.org/ or installed locally
irb                  1.2.4     
Bundler::Source::Git https://github.com/ruby/irb (at master@9bb6562)
reline               0.1.4     
Bundler::Source::Rubygems rubygems repository https://rubygems.org/ or installed locally

Another thing that's interesting to note is that the version of bundler itself doesn't show up in the specified gems, though it is available from the bundler_version method.

If you are working with a matching set of Gemfile and Gemfile.lock files, which most people are, there's a nifty command bundle outdated. This looks though the gems that you have to see if there's a later version released.

Let's recreate this using our Gemfile.lock only method so we can look at lock files and see which gems have updated code.

First we will pull the specs out of the lockfile. Then we will loop over them, pulling in the infomation from rubygems or wherever the remote was specified. We aren't doing anything special with git sourced gems.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

  #!/usr/bin/env ruby
  require 'bundler'
  require 'gems'

  def specs_from_lockfile file = Bundler.default_lockfile
    gems = {}
    context = Bundler::LockfileParser.new(Bundler.read_file(file))
    context.specs.each do |info|
      gems[info.name] = {version: info.version.to_s, source: info.source.class }
      if info.source.is_a? Bundler::Source::Rubygems
        gems[info.name][:remote] = info.source.remotes.first.to_s 
      elsif info.source.is_a? Bundler::Source::Git
        gems[info.name][:remote] = info.source.uri 
        gems[info.name][:ref] = info.source.ref
        gems[info.name][:revision] = info.source.revision
      else
        puts "Not sure how to process #{info.source.class}"
      end
    end
    gems
  end

  def add_rubygems_versions info
    info.each do |name, spec|
      if spec[:source] == Bundler::Source::Rubygems
        gems_client = Gems::Client.new( { host: spec[:remote] } )
        spec[:info] = gems_client.info( name )
      elsif spec[:source] == Bundler::Source::Git
        spec[:info] = Gems.info( name )
      else
        puts "Not sure of the source of #{name}"
      end
    end
  end

  specs = specs_from_lockfile ARGV[0] || Bundler.default_lockfile
  add_rubygems_versions( specs )

  printf "%15s  %-8s %-8s %3s %s\n", "Name", "Current", "Latest", "Old", "Info"

  specs.each do |name,info|
    info[:info] ||= {}
    current_version = info[:version]
    new_version = info[:info]["version"]
    printf "%15s  %-8s %-8s %-3s %s\n", 
           name,
           current_version,
           new_version,
           current_version == new_version ? "" : "Y",
           info[:info]["info"][0..50]
  end

           Name  Current  Latest   Old Info
           gems  1.1.1    1.2.0    Y   Ruby wrapper for the RubyGems.org API
     io-console  0.5.6    0.5.6        add console capabilities to IO instances.
            irb  1.2.4    1.2.4        Interactive Ruby command-line tool for REPL (Read E
           json  2.3.1    2.3.1        This is a JSON implementation as a Ruby extension i
         reline  0.1.4    0.1.4        Alternative GNU Readline or Editline implementation

This is a very simple project, but we can see that the one gem that we held back to 1.1.1 instead of the (current) latest of 1.2.0 is outdated.

Looking at the Gemfile.lock file we're able to see which versions of the gems are installed, and we can pull down the git repos of most of them using the metadata. (Or homepage, for a lot of them.) The standard bundling tools will create a git tag for each of the releases, so in our next installment we will start looking at the code differences between versions, both the commit messages as well as overall activity in these projects. This should help us understand how risky the upgrades are, and if we believe in semver.

1. https://stackoverflow.com/questions/38800129/parsing-a-gemfile-lock-with-bundl
2. https://rdoc.info/github/bundler/bundler/Bundler/LazySpecification
3. https://rdoc.info/github/rubygems/rubygems/Gem/Specification