Looking at package.json

making sense of package-lock.json

Published October 13, 2020 #node, #npm, #packagemanagers, #git

Look at the dependencies

First lets create a simple project and add a single module, in this case npm-api which will we use to access the main repository.

npm init -y
npm add npm-api

And lets see what's been installed in node_modules:

ls -l node_modules | wc -l 
du -sh node_modules
68
12Mnode_modules

68 directories with 12M of code! Yowza! That's-a big package. Lets parse up the package-lock.js file to see if it agrees:

  'use strict';

  const fs = require('fs');

  let rawdata = fs.readFileSync('package-lock.json');
  let lockfile = JSON.parse(rawdata);

  Object.keys(lockfile.dependencies).forEach( (elem,index) => {
      console.log( `| ${index+1} | ${elem} | ${lockfile.dependencies[elem].version} |` );
  });
1JSONStream1.3.5
2ajv6.12.4
3asn10.2.4
4assert-plus1.0.0
5asynckit0.4.0
6aws-sign20.7.0
7aws41.10.1
8axios0.18.1
9bcrypt-pbkdf1.0.2
10caseless0.12.0
11clone-deep4.0.1
12combined-stream1.0.8
13core-util-is1.0.2
14dashdash1.14.1
15debug3.1.0
16delayed-stream1.0.0
17download-stats0.3.4
18ecc-jsbn0.1.2
19extend3.0.2
20extsprintf1.3.0
21fast-deep-equal3.1.3
22fast-json-stable-stringify2.1.0
23follow-redirects1.5.10
24forever-agent0.6.1
25form-data2.3.3
26getpass0.1.7
27har-schema2.0.0
28har-validator5.1.5
29http-signature1.2.0
30is-buffer1.1.6
31is-plain-object2.0.4
32is-typedarray1.0.0
33isobject3.0.1
34isstream0.1.2
35jsbn0.1.1
36json-schema0.2.3
37json-schema-traverse0.4.1
38json-stringify-safe5.0.1
39jsonparse1.3.1
40jsprim1.4.1
41kind-of6.0.3
42lazy-cache2.0.2
43mime-db1.44.0
44mime-types2.1.27
45moment2.27.0
46ms2.0.0
47npm-api1.0.0
48oauth-sign0.9.0
49paged-request2.0.1
50performance-now2.1.0
51psl1.8.0
52punycode2.1.1
53qs6.5.2
54request2.88.2
55safe-buffer5.2.1
56safer-buffer2.1.2
57set-getter0.1.0
58shallow-clone3.0.1
59sshpk1.16.1
60through2.3.8
61to-object-path0.3.0
62tough-cookie2.5.0
63tunnel-agent0.6.0
64tweetnacl0.14.5
65uri-js4.4.0
66uuid3.4.0
67verror1.10.0

There's one additional directory installed in node_modules called .bin which is where binary executables of installed packages live, so that's the difference.

We can see what commands are installed:

ls -l node_modules/.bin
total 20
lrwxrwxrwx 1 wschenk wschenk 20 Sep  9 14:26 JSONStream -> ../JSONStream/bin.js
lrwxrwxrwx 1 wschenk wschenk 23 Sep  9 14:26 sshpk-conv -> ../sshpk/bin/sshpk-conv
lrwxrwxrwx 1 wschenk wschenk 23 Sep  9 14:26 sshpk-sign -> ../sshpk/bin/sshpk-sign
lrwxrwxrwx 1 wschenk wschenk 25 Sep  9 14:26 sshpk-verify -> ../sshpk/bin/sshpk-verify
lrwxrwxrwx 1 wschenk wschenk 16 Sep  9 14:26 uuid -> ../uuid/bin/uuid

What are the specified deps

The structure of package-lock.json is much simpler than Gemfile.lock, and it doesn't show which modules are the ones that the developer specified and which are ones are derivatives. We can take a guess at this by looking at modules that aren't another's dependancy.

  'use strict';

  const fs = require('fs');

  let rawdata = fs.readFileSync('package-lock.json');
  let lockfile = JSON.parse(rawdata);

  let deps = Object.keys(lockfile.dependencies);
  let possible_mains = new Map();

  deps.forEach( elem => {
      possible_mains.set( elem,  true )
  } );

  deps.forEach( elem => {
      let requires = Object.keys(lockfile.dependencies[elem].requires || {})
      requires.forEach( dep => {
          possible_mains.delete( dep );
      } )
  } )

  possible_mains.forEach( (value, key) => console.log( key ) );

Which happily yields:

npm-api

So that looks correct for this project.

Loading project metadata from npm

The next thing we are looking for is the repository of the source code, so we can see what code there's out there, how well it's maintained, etc.

  var NpmApi = require('npm-api');

  (async () => {
      const repo = new NpmApi().repo( 'npm-api' );

      const pkg = await repo.package()

      console.log( "Name        ", pkg.name );
      console.log( "Version     ", pkg.version );
      console.log( "Description ", pkg.description );
      console.log( "License     ", pkg.license );
      console.log( "Homepage    ", pkg.homepage );
      console.log( "Repository  ", pkg.repository );
      console.log( "Clean repo  ", pkg.repository.url.replace( /git\+/g, '' ) );
      console.log( "Bug         ", pkg.bugs );
  })()
Name         npm-api
Version      1.0.0
Description  Base class for retrieving data from the npm registry.
License      MIT
Homepage     https://github.com/doowb/npm-api
Repository   { type: 'git', url: 'git+https://github.com/doowb/npm-api.git' }
Clean repo   https://github.com/doowb/npm-api.git
Bug          { url: 'https://github.com/doowb/npm-api/issues' }

Here we can see the repository is type git and the url has an unexplained git+ in front of it. Why? I'd love to know. But we can strip it out using the replace function to get something not pointlessly redundant from the type sibling attribute.

Finding out of date dependencies

npm has a similar function to bundle outdated called… npm outdated. Exciting! Lets recreate that now.

  'use strict';

  const NpmApi = require( 'npm-api' );
  const fs = require('fs');

  let rawdata = fs.readFileSync('package-lock.json');
  let lockfile = JSON.parse(rawdata);

  (async () => {    
      Object.keys(lockfile.dependencies).forEach( async (elem,index) => {
          const repo = new NpmApi().repo( elem );
          const pkg = await repo.package();

          const installed_version = lockfile.dependencies[elem].version;
          const current_version = pkg.version;

          console.log( `| ${elem} | ${installed_version} | ${current_version} | ${installed_version != current_version ? 'OUTDATED' : 'CURRENT'} |` );
      });
  })()

Which dumps out:

ModuleInstalledLatestStatus
core-util-is1.0.21.0.2CURRENT
caseless0.12.00.12.0CURRENT
performance-now2.1.02.1.0CURRENT
clone-deep4.0.14.0.1CURRENT
isstream0.1.20.1.2CURRENT
ecc-jsbn0.1.20.2.0OUTDATED
json-stringify-safe5.0.15.0.1CURRENT
isobject3.0.14.0.0OUTDATED
asn10.2.40.2.4CURRENT
combined-stream1.0.81.0.8CURRENT
ms2.0.02.1.2OUTDATED
paged-request2.0.12.0.1CURRENT
fast-json-stable-stringify2.1.02.1.0CURRENT
extend3.0.23.0.2CURRENT
download-stats0.3.40.3.4CURRENT
asynckit0.4.00.4.0CURRENT
bcrypt-pbkdf1.0.21.0.2CURRENT
extsprintf1.3.01.4.0OUTDATED
tunnel-agent0.6.00.6.0CURRENT
lazy-cache2.0.22.0.2CURRENT
is-buffer1.1.62.0.4OUTDATED
aws-sign20.7.00.7.0CURRENT
jsbn0.1.11.1.0OUTDATED
har-schema2.0.02.0.0CURRENT
delayed-stream1.0.01.0.0CURRENT
dashdash1.14.12.0.0OUTDATED
forever-agent0.6.10.6.1CURRENT
safer-buffer2.1.22.1.2CURRENT
is-plain-object2.0.45.0.0OUTDATED
getpass0.1.70.1.7CURRENT
json-schema-traverse0.4.10.5.0OUTDATED
json-schema0.2.30.2.5OUTDATED
is-typedarray1.0.01.0.0CURRENT
punycode2.1.12.1.1CURRENT
http-signature1.2.01.3.5OUTDATED
to-object-path0.3.00.3.0CURRENT
verror1.10.01.10.0CURRENT
fast-deep-equal3.1.33.1.3CURRENT
through2.3.82.3.8CURRENT
jsonparse1.3.11.3.1CURRENT
shallow-clone3.0.13.0.1CURRENT
safe-buffer5.2.15.2.1CURRENT
jsprim1.4.12.0.0OUTDATED
npm-api1.0.01.0.0CURRENT
set-getter0.1.00.1.0CURRENT
oauth-sign0.9.00.9.0CURRENT
uri-js4.4.04.4.0CURRENT
follow-redirects1.5.101.13.0OUTDATED
kind-of6.0.36.0.3CURRENT
aws41.10.11.10.1CURRENT
mime-db1.44.01.45.0OUTDATED
sshpk1.16.11.16.1CURRENT
psl1.8.01.8.0CURRENT
mime-types2.1.272.1.27CURRENT
har-validator5.1.55.1.5CURRENT
form-data2.3.33.0.0OUTDATED
tweetnacl0.14.51.0.3OUTDATED
debug3.1.04.2.0OUTDATED
tough-cookie2.5.04.0.0OUTDATED
qs6.5.26.9.4OUTDATED
JSONStream1.3.51.3.5CURRENT
axios0.18.10.20.0OUTDATED
moment2.27.02.29.1OUTDATED
uuid3.4.08.3.1OUTDATED
request2.88.22.88.2CURRENT
assert-plus1.0.01.0.0CURRENT
ajv6.12.46.12.6OUTDATED

Thoughts

As with our Gemfile exploration, we can

  1. Identify which modules are specified only from the lock file.

  2. Look at all of the dependancies of the project to see which is out of date

  3. Find the git repo that the original code is packaged from.

The next step will be to start looking into the repos themselves to ask a few questions:

  1. Is the project maintained?

  2. What is the project activity?

  3. Is it a semver project?

  4. What patch/minor/major code has changed?

  5. How is the project connected to other projects?

Stay tuned!

References

  1. https://docs.npmjs.com/configuring-npm/package-lock-json.html

  2. https://docs.npmjs.com/about-packages-and-modules#npm-package-git-url-formats

  3. https://github.com/doowb/npm-api

Read next

Previous Post: Looking at Gemfiles

See also

Looking at Gemfiles

making sense of Gemfile.lock

Bundler is the standard way for Ruby projects to specifiy dependencies. Let's take a look at how that works, reimplement bundle outdated, and be able to see the changes that took place between the build you are using and the latest one. Ruby ecosystem: Rubygems, Bundler, Gemfile, Gemfile.lock rubygems is the overall ecosystem, which includes the main rubygems.org database of shared packages, and how they depend upon each other.

Read more

Splitting Git Repos and Work Directories

all the fun things git can do

I found a tutorial on how to manage your dotfiles, that works by splitting up the git repository (normally the .git directory) from the work directory. Since I have a lot of code that I put in my tutorials, I adapted the technique to have individual article directories mirrored in their own github repository. Repositories and Work Directories The normal usage of git is to type git clone <remote> to get a copy of the local directory, mess with stuff, and then add and commit your changes.

Read more