jq, xq and yq - Handy tools for the command line

The jq tool for JSON seems to be already well known and indeed is a very handy tool as many software packages use JSON for serialized data. Here is a short example for parsing metadata of the npm package manager:

Note the usage of the "-r" option in the last example for "raw output". This allows us to easily use the result in the shell by command substitution:

The GNU/Linux distributions I use daily, e.g. Debian and Ubuntu, include the package so it is easy to install and keep up to date.

Note that jq will do syntax coloring and properly indent the output whatever you do. So if you are looking at a valid but difficult to read JSON file, the trivial filter . will give you a nice rendering of the data:

It seems tools learn to output complex data in JSON in which case jq should probably advance to the standard Linux command line toolkit. See my other post for an example of that.

While working on the scripts for my GPX collection I wondered if there is an equivalent for the XML format used here. It would be a significant advantage over the improvised grep and sed commands which I know to be not very robust.

It turns out that there are converters for YAML and XML to JSON and thus can be easily combined into an command-line YAML and XML processor.

Having installed yq with the Python packager pip, we get the yp binary for working with YAML files and the xq binary for XML. Using the jq manual, it is straightforward to apply it to GPX files in my collection. Here are a few examples intended to convey an idea of the possibilities. Extracting the name of the track is easy, also finding the number of track points in it:

Array indices make it easy to extract specific waypoints and for example the start and end time of the whole track:

In the comments to this post, Tom Brown contributed a very nice one liner to convert a GPX track to a CSV file. I’ll try to break the script down into smaller chunks and retrace the steps to arrive at the clever result.

The aim of the jq script is to transform the waypoints into single line records starting with the time attribute. Searching the jq manual, we find that the function @csv will render an array to a single CSV line. So one important task is to transform the JSON objects of the GPX file into arrays. To show the process, here is an example of using only the first two track points. The array for @csv is built by the map operator transforming the array of waypoints (i.e. JSON objects) into an array of string arrays with the values of the attributes:

Note that map works on an array, so for this (contrived) example we had to assemble the two waypoints explicitly into an array before passing it to map. The final invocation will not need this as .gpx.trk.trkseg.trkpt without any index is an array.

Now for @csv to work we again have to remove this (top level) extra array:

And finally we can call @csv on the result:

According to the GPX 1.1 Schema, the time field in wptType is of type xsd:dateTime with the letter T between the date and the time and an optional offset of the local timezone against UTC at the end. Usually this is just a plain Z specifying an offset of 0 or in other words to denote the time to be in UTC. Tom writes that the letters T and Z are not parsed correctly by Google Sheets so he uses string slicing to cut out only the wanted characters:

Putting all this together yields this clever one liner to convert the whole track at once:

So jq and its relatives are definitely a worthwhile addition to every command toolbox, but for my GPX files I would love to see tools capable of doing real processing of the track points. Although I did look for such tools, I was as yet not successful. So if you know about command line tools to calculate the usual statistics like distance, average speed, max speed, etc. I would love to see pointers in the comment section.