Looking at Gemfiles

making sense of Gemfile.lock

Published September 4, 2020 #ruby, #bundler, #packagemanagers, #git

Bundler is the standard way for ruby projects to specifiy dependancies. Lets take a look at how that works, reimplement bundle outdated, and be able to see the changes that took place between the build you are using at the latest one.

Ruby ecosystem: Rubygems, Bundler, Gemfile, Gemfile.lock

rubygems is the overall ecosystem, which includes the main rubygems.org database of shared packages, and how they depend upon each other. At the heart of it is a gemspec file which describes the package, files included in it, dependancies and other metadata.

bundler the tool that helps you download and install specific gems for your application. It used a Gemfile to define what you want, and the resolves all the dependancies stores that into a Gemfile.lock file that contains a list of the exact versions that you are developing against.

Creating a working Gemfile

First lets create a project, so we are all working off of the same thing. We'll add bunder as normal, and then irb from git directly so we can see how that changes the Gemfile.lock. So, 3 gems, 2 from rubygems.org and one directly from github.

bundle init
bundle add bundler --version "~>2.1"
bundle add irb --git "https://github.com/ruby/irb"
bundle add gems --version "1.1.1"

Lets also add in a ruby "2.6.6" line that will tell bundler which version of ruby we want to use. This isn't always used but lets put it in to flesh out what is generated in the lock file. Feel free to replace this with whatever version is on your local machine.

Which gives us a Gemfile that looks like this:

# frozen_string_literal: true

source "https://rubygems.org"

git_source(:github) {|repo_name| "https://github.com/#{repo_name}" }

ruby "2.6.6"
# gem "rails"

gem "bundler", "~> 2.1"

gem "irb", git: "https://github.com/ruby/irb"

gem "gems", "1.1.1"

Doing a bundle add adds the specification to the Gemfile, downloads the gems and all of it's dependancies, resolving specific versions and downloading those as well. This is specified in the lock file, which has the main sections:

  1. GIT which is the first remote source on the list, here telling which commit that we are depending upon specifically.

  2. GEM which list out the the gems that are installed, with versions

  3. PLATFORMS which is for platform information (for native builds)

  4. DEPENDENCIES which is what comes out of the Gemfile

  5. RUBY VERSION the version of ruby, including the patch level, that this was bundled with.

  6. BUNDLED WITH version information about bundler.

GIT
  remote: https://github.com/ruby/irb
  revision: 9bb6562d0d841fd294e95bbb690e812cbd5090ce
  specs:
    irb (1.2.4)
      reline (>= 0.0.1)

GEM
  remote: https://rubygems.org/
  specs:
    gems (1.1.1)
      json
    io-console (0.5.6)
    json (2.3.1)
    reline (0.1.4)
      io-console (~> 0.5)

PLATFORMS
  ruby

DEPENDENCIES
  bundler (~> 2.1)
  gems (= 1.1.1)
  irb!

RUBY VERSION
   ruby 2.6.6p146

BUNDLED WITH
   2.1.4

Parsing Gemfile and Gemfile.lock together

In normal usage you would have both a Gemfile and a Gemfile.lock. You'd start up your program using bundle exec which would validate that the right things were installed, load the gems into the ruby environment, and you'd be good to go.

  #!/usr/bin/env ruby
  require 'bundler'

  context = Bundler.load

  puts "Dependencies"
  context.dependencies.each do |x|
    printf "%-20s %s\n", x.name, x.requirements_list.to_s
  end

  puts
  context.specs.each do |s|
    code = s.metadata['source_code_url'] || s.metadata['source_code_uri']
    puts "Gem from #{s.source.to_s}"
    printf "%-20s %-10s %-40s %s\n", s.name, s.version.to_s, code, s.homepage
  end

Which yields:

Dependencies
irb                  [">= 0"]
bundler              ["~> 2.1"]

Gem from the local ruby installation
bundler              2.1.4      https://github.com/bundler/bundler/      https://bundler.io
Gem from rubygems repository https://rubygems.org/ or installed locally
gems                 1.2.0                                               https://github.com/rubygems/gems
Gem from rubygems repository https://rubygems.org/ or installed locally
io-console           0.5.6      https://github.com/ruby/io-console       https://github.com/ruby/io-console
Gem from rubygems repository https://rubygems.org/ or installed locally
reline               0.1.4                                               https://github.com/ruby/reline
Gem from https://github.com/ruby/irb (at master@9bb6562)
irb                  1.2.4                                               https://github.com/ruby/irb

So some gems have source_code_uri or source_code_url set, and some don't have it at all though they point to a github page where presumably we can figure out where the code is loaded.

For the case of irb the repo is listed in the remote source itself.

The metadata, homepage, etc are from the gemspec files.

Parsing a Gemfile.lock directly

Let's now look at how to parse a Gemfile.lock of a different project, where we don't have the original Gemfile handy.

When we load in the lockfile using Bundler::LockfileParser we only have the gem name, version, and the source from which to get it. We may or may not have the gemspec on our local machine, and so additional metadata like the homepage, summary, codeurl, etc haven't been loaded yet. Lets first print out what we have and then figure out how to get gem information from the source, name, and version.

#!/usr/bin/env ruby
require 'bundler'

def describe_lockfile file = Bundler.default_lockfile
  context = Bundler::LockfileParser.new( Bundler.read_file( file ) )
  puts "Bundler version"
  puts context.bundler_version.to_s

  puts
  puts "Dependencies"
  context.dependencies.each do |name,x|
    printf "%-20s %s\n", x.name, x.requirements_list.to_s
  end

  puts
  puts "Gems"
  context.specs.each do |s|
    printf "%-20s %-10s\n", s.name, s.version.to_s
    printf "%-10s %s\n", s.source.class, s.source.to_s
  end
end

describe_lockfile
Bundler version
2.1.4

Dependencies
bundler              ["~> 2.1"]
gems                 ["~> 1.2"]
irb                  [">= 0"]

Gems
gems                 1.2.0     
Bundler::Source::Rubygems rubygems repository https://rubygems.org/ or installed locally
io-console           0.5.6     
Bundler::Source::Rubygems rubygems repository https://rubygems.org/ or installed locally
irb                  1.2.4     
Bundler::Source::Git https://github.com/ruby/irb (at master@9bb6562)
reline               0.1.4     
Bundler::Source::Rubygems rubygems repository https://rubygems.org/ or installed locally

Another thing that's interesting to node is that the version of bundler itself doesn't show up in the specified gems, though it is available from the bundler_version method.

Finding outdated gem versions

If you are working with a matching set of Gemfile and Gemfile.lock files, which most people are, there's a nifty command bundle outdated. This looks though the gems that you have to see if there's a later version released.

Lets recreate this using our Gemfile.lock only method so we can look at lock files and see which gems have updated code.

First we will pull the specs out of the lockfile. Then we will loop over them pulling in the infomation from rubygems or wherever the remote was specified. We aren't doing anything special with git sourced gems.

  #!/usr/bin/env ruby
  require 'bundler'
  require 'gems'

  def specs_from_lockfile file = Bundler.default_lockfile
    gems = {}
    context = Bundler::LockfileParser.new(Bundler.read_file(file))
    context.specs.each do |info|
      gems[info.name] = {version: info.version.to_s, source: info.source.class }
      if info.source.is_a? Bundler::Source::Rubygems
        gems[info.name][:remote] = info.source.remotes.first.to_s 
      elsif info.source.is_a? Bundler::Source::Git
        gems[info.name][:remote] = info.source.uri 
        gems[info.name][:ref] = info.source.ref
        gems[info.name][:revision] = info.source.revision
      else
        puts "Not sure how to process #{info.source.class}"
      end
    end
    gems
  end

  def add_rubygems_versions info
    info.each do |name, spec|
      if spec[:source] == Bundler::Source::Rubygems
        gems_client = Gems::Client.new( { host: spec[:remote] } )
        spec[:info] = gems_client.info( name )
      elsif spec[:source] == Bundler::Source::Git
        spec[:info] = Gems.info( name )
      else
        puts "Not sure of the source of #{name}"
      end
    end
  end

  specs = specs_from_lockfile ARGV[0] || Bundler.default_lockfile
  add_rubygems_versions( specs )

  printf "%15s  %-8s %-8s %3s %s\n", "Name", "Current", "Latest", "Old", "Info"

  specs.each do |name,info|
    info[:info] ||= {}
    current_version = info[:version]
    new_version = info[:info]["version"]
    printf "%15s  %-8s %-8s %-3s %s\n", 
           name,
           current_version,
           new_version,
           current_version == new_version ? "" : "Y",
           info[:info]["info"][0..50]
  end
           Name  Current  Latest   Old Info
           gems  1.1.1    1.2.0    Y   Ruby wrapper for the RubyGems.org API
     io-console  0.5.6    0.5.6        add console capabilities to IO instances.
            irb  1.2.4    1.2.4        Interactive Ruby command-line tool for REPL (Read E
           json  2.3.1    2.3.1        This is a JSON implementation as a Ruby extension i
         reline  0.1.4    0.1.4        Alternative GNU Readline or Editline implementation

This is a very simple project, but we can see that the one gem that we held back to 1.1.1 instead of the (current) latst of 1.2.0 is outdated.

Future thoughts

Looking at the Gemfile.lock file we're able to see which versions of the gems are installed, and we can pull down the git repos of most of them using the metadata. (Or homepage, for a lot of them.) The standard bundling tools will create a git tag for each of the releases, so in our next installment we will start looking seeing the code different between the versions, both the commit messages as well as overall activity in these projects. This should help us understand how risky the upgrades are, and if we believe in semvar.

References

  1. https://stackoverflow.com/questions/38800129/parsing-a-gemfile-lock-with-bundl

  2. https://rdoc.info/github/bundler/bundler/Bundler/LazySpecification

  3. https://rdoc.info/github/rubygems/rubygems/Gem/Specification

Read next

Previous Post: gitlog in sqlite

See also

Checking health of RSS feeds

Lets clean out the old stuff

I still use RSS feeds as my main way of consuming stuff on the internet. Old school. I was looking at the list of things that I've subscribed to which has been imported back from the Google Reader days, and figured I'd do some exploration of ye old OPML file. Setup the Gemfile We'll need to pull in a few gems here: source 'https://rubygems.org' gem 'httparty' # To pull stuff from the network gem 'feedjira' # To parse RSS feeds gem 'nokogiri' # To parse XML gem 'domainator' # To figured out the domain from the hostname gem 'whois' # To see how old the domain is gem 'whois-parser' # To actually make sense of the result And install everything bundle Look at domains First, lets see what we can find out about the machine that hosts the feed.

Read more

Splitting Git Repos and Work Directories

all the fun things git can do

I found a tutorial on how to manage your dotfiles, that works by splitting up the git repository (normally the .git directory) from the work directory. Since I have a lot of code that I put in my tutorials, I adapted the technique to have individual article directories mirrored in their own github repository. Repositories and Work Directories The normal usage of git is to type git clone <remote> to get a copy of the local directory, mess with stuff, and then add and commit your changes.

Read more