More OpenStreetMap Fixes for Road Bikers
Already in a previous post I wrote about fixing the OpenStreetMap data for better routing results for road bikers.
Recently another one of my cycling excursions led me through terrain where OSM was not reflecting the current situation anymore. Where it was previously possible to use the asphalt track next to the small forest strip on the north west corner of the Wilhelmsburg in Ulm, the military installation now extended the private area through a large fence. New gravel tracks are provided for the public as a replacement. As OSM still contained the asphalt track as passable, Komoot of course used it in its road bike routing setting.
Read more… (1 min remaining to read)
More Colors For My Blog
As I wrote in a previous post I am now very pleased with the syntax coloring for console transcripts in my blog posts. As a reminder, the resulting HTML color is not derived from the color information in the original output but rather added after the fact by the Pygments Python library to plain text transcripts.
Although the solution looks nice and allows me to keep the terminal transcripts right within the source files for the posts, it is not a full replacement for including terminal output with ANSI color escape sequences directly in the blog posts. This solution is required to reproduce terminal transcripts for programs that have individual colored output just like the script described in the checksec post.
In lack of another method I took screenshots of the terminal output and included the graphics rather than text with attributes. Again this is unfortunate as it renders the text in the graphics inaccessible for example to web search engines - or at least I currently believe that most of them don’t do OCR on arbitrary images.
Read more… (3 min remaining to read)
Syntax Colors For My Blog
A lot of my blog posts contain console logs from GNU/Linux systems. For the first few posts I was content having found an efficient way to copy real output into blog posts.
For ReST based posts it is as simple as using a double colon at the end of the previous paragraph, copy and paste the output with the mouse and finally indent the pasted text by spaces. This is an easy task in Emacs by selecting the beginning of the pasted output with "C-<space>", move to the end and call "C-x-r-t" (string-rectangle) with 4 spaces as a parameter to make the indentation. Below is an example fragment followed by its rendering:
<ReST Post>
This yields already acceptable output::
dzu@krikkit:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 10 (buster)
Release: 10
Codename: buster
dzu@krikkit:~$
This yields already acceptable output:
dzu@krikkit:~$ lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 10 (buster) Release: 10 Codename: buster dzu@krikkit:~$
Because the workflow was so easy and the result was usable I put aside my passing thoughts whether syntax coloring would increase the ease of reading the technical heavy blogs.
More than 5 years have passed since then and motivated by different reflections on documentation I suddenly remembered this itching for syntax coloring the console output.
Read more… (4 min remaining to read)
jq, xq and yq - Handy tools for the command line
The classic Unix command line tools like grep, sed and awk do not cope well with structured data formats like JSON, YAML or even XML. Although I did use them with some acrobatics for such formats, a proper solution requires tools that natively understand those formats.
JSON
The jq tool for JSON seems to be already well known and indeed is a very handy tool as many software packages use JSON for serialized data. Here is a short example for parsing metadata of the npm package manager:
dzu@krikkit:~$ jq .name ~/node_modules/bash-language-server/package.json
"bash-language-server"
dzu@krikkit:~$ jq .repository ~/node_modules/bash-language-server/package.json
{
"type": "git",
"url": "https://github.com/bash-lsp/bash-language-server"
}
dzu@krikkit:~$ jq .repository.url ~/node_modules/bash-language-server/package.json
"https://github.com/bash-lsp/bash-language-server"
dzu@krikkit:~$ jq -r .repository.url ~/node_modules/bash-language-server/package.json
https://github.com/bash-lsp/bash-language-server
dzu@krikkit:~$
Note the usage of the "-r" option in the last example for "raw output". This allows us to easily use the result in the shell by command substitution:
dzu@krikkit:~$ firefox $(jq -r .repository.url ~/node_modules/bash-language-server/package.json) dzu@krikkit:~$
The GNU/Linux distributions I use daily, e.g. Debian and Ubuntu, include the package so it is easy to install and keep up to date.
Note that jq will do syntax coloring and properly indent the output whatever you do. So if you are looking at a valid but difficult to read JSON file, the trivial filter . will give you a nice rendering of the data:
dzu@krikkit:~$ cat json-stringify
{"orderID":12345,"shopperName":"John Smith","shopperEmail":"johnsmith@example.com","contents":[{"productID":34,"productName":"SuperWidget","quantity":1},{"productID":56,"productName":"WonderWidget","quantity":3}],"orderCompleted":true}
dzu@krikkit:~$ jq . json-stringify
{
"orderID": 12345,
"shopperName": "John Smith",
"shopperEmail": "johnsmith@example.com",
"contents": [
{
"productID": 34,
"productName": "SuperWidget",
"quantity": 1
},
{
"productID": 56,
"productName": "WonderWidget",
"quantity": 3
}
],
"orderCompleted": true
}
dzu@krikkit:~$
JSON in the pipeline
It seems tools learn to output complex data in JSON in which case jq should probably advance to the standard Linux command line toolkit. See my other post for an example of that.
XML
While working on the scripts for my GPX collection I wondered if there is an equivalent for the XML format used here. It would be a significant advantage over the improvised grep and sed commands which I know to be not very robust.
Read more… (3 min remaining to read)
OpenStreetMap Fixes for Road Bikers
As I have been doing a lot of road biking recently, I found a cool example on how Open Data allows me to fix problems in otherwise proprietary services and so I decided to share this happy moment in this blog post.
As I already wrote in a previous post about trip planning for road bikes all of my road bike excursions are planned ahead of time at the PC and during the trip I use my Garmin eTrex only to superimpose the prepared GPX on a bike friendly rendering of the OpenStreetMap data. As a side note - at the beginning of my experiments with cycling and GPS I tried the interactive and audible turn indications of the Garmin device but found the simple map display to be better suited for myself.
However contrary to the previous post I switched from using GraphHopper to the paid service of Komoot for this planning. As far as I can tell it also uses GraphHopper under the hood for the routing, but it allows me to come up with much more pleasant routes to cycle thanks to the "highlights". This feature allows the Komoot community to mark especially nice locations or, even better, whole track segments especially suited for some activities. So there are "Hiking highlights", "MTB highlights" and of course "Road bike highlights" that can be integrated into your own planning by a simple mouse click. Through this feature I already got to know some very nice quiet back roads in the vicinity that I was not previously aware of.
Even Good Things Can Be Improved
Having selected "Road Bike" in the Komoot planner, the suggested route will usually be only on asphalt, but on one of my recent trips I encountered 500 meters of gravel. Of course, this is only a small inconvenience but as this was the only glitch on this 101 km trip, I decided to find out if I can fix this for the future and for other cyclists as well.
Back home it was only a matter of seconds to find that the segment was indeed the preferred connection of points A and B from my trip for road bikes. The screenshot shows this route and the problematic segment in green:
Read more… (3 min remaining to read)
Repairing My ChipTAN Generator
Many German banks adopted the ChipTAN scheme for online-banking over the last few years. Although many of them nowadays also offer TAN systems based on SMS or smartphone apps, I still trust the smart card based ChipTAN scheme most. Unfortunately my Kobil "TAN Optimus comfort V1.4" started to develop defective pixels as can be seen in the following figure.
With only a few pixels missing it was still reasonable to use it, but lately it was not possible to correctly read the generated TANs anymore. Of course I tried changing the batteries when the symptoms began, but this did not help. A cross check with the multimeter also showed the batteries were still fine. As the dead pixels sometimes worked spuriously again for short moments I began to suspect that the display itself was probably alright and the fault is somewhere else.
Searching the Internet did not yield many results but with different search terms I finally found an older post from the iwenzo portal put. It contained useful hints to eventually fix the problem myself.
Read more… (4 min remaining to read)
Reclaiming VM Hard Disk Space from VirtualBox
Creating a virtual machine requires specifying the amount of space to allocate for the virtual hard disk as one of the first steps. Dreading the notorious "not enough disk space" message, one usually aims to be on the safe side and reserves a generous amount at this stage. If the assignment is done in a static fashion then the allocated space is immediately claimed from the underlying file system and one thus has to come up with a pretty good guess in order not to waste space. Thankfully most VM solutions allow dynamic allocations which can grow on demand limited only by an upper bound making this a very attractive choice.
A newly created VM with such an dynamic disk image indeed starts out nice and space saving but over time the image will eventually take up all the committed underlying disk space even though the VM reports to use only a subset of it. Reflecting on why this is the case leads us to an elegant setup reclaiming unused storage space on the host system.
Read more… (6 min remaining to read)
Undertime - Matching Up Time Zones
Occasionally I need to setup teleconferences with people in different time zones. For two timezones I am still able to do the calculations and assessment of the possibilities in my head. However, once three or more timezones are involved, I find myself requiring some assistance.
Already a while ago I found the neat little Python script undertime which is very easy to use.
Read more… (2 min remaining to read)
DEM data for QMapShack
As noted in my previous post GPX Downloads from geocaching.com I still need to rework my workflows from QLandkarteGT to the new QMapShack tool as only the latter is available for a Debian Buster system.
For my use cases, knowing the height of a map location is very important and so providing QMapShack with digital elevation data (DEM for short) is an important step. I cannot remember exactly how I did this for QLandkarteGT in the past, but I remember a lot of web browsing to find and download the correct Shuttle Radar Topology Mission (SRTM) followed by some magic script invocations to package it up for QLandkarteGT. Switching to QMapShack seems like a good time to reevaluate how this can be done most easily.
Read more… (3 min remaining to read)
BPF Superpowers for Linux
A powerful set of tools is helpful not only for developers. On GNU/Linux strace is one such tool and helped me already with a wide variety of problems. However it is always confined to one process or a process tree and cannot help with system wide problems / questions. Although there have been many attempts in the past, the Linux developers seem to have found a common underlying infrastructure in the form of BPF for such tools, allowing flexible and high-performance probing.
Already back in 2016, the Netflix developer Brendan Gregg showed what is possible in his talk Linux BPF Superpowers.
Read more… (3 min remaining to read)