Thursday, March 7, 2019

Exclude files from being exported into the zip/tar source archives on github.com

GitHub.com (and probably GitLab too) provides various ways to export the Git branch contents or tags and releases as Zip- or Tar-archives. When creating a release, these tar-/zipballs are automatically created and added to the release. I often find archives, which contain a lot of files not useful to the end user, like .github directories, Git (.gitignore, .gitattributes) or CI related files (.travis.yml, .appveyor.yml). Sometimes they also contain directories (e.g. for test files), upstream hosts in Git, but does not need for the source distribution. But there is an easy way to keep these files out of the automatically created source archives and keep the latter clean by using the export-ignore attribute in the .gitattributes files:

# don't export the github-pages source
/docs export-ignore
# export some other irrelevant directories
/foo export-ignore
# don't export the files necessary for CI
Gemfile export-ignore
.appveyor.yml export-ignore
.travis.yml export-ignore
# ignore Git related files
.gitattributes export-ignore
.gitignore export-ignore

Sunday, February 24, 2019

Jekyll and GitHub pages: access the download URL (aka browser_download_url) for an asset of your latest release via site.github

Add the download URL of an asset of your latest release

Often there is the question to get the download URL for an asset (e.g. a setup-file) of the latest release of a project. In my case I provide an executable, which includes the version number in its name together with the source as ZIP- and Tarball-archive. Others provide versioned source tarballs or executables, which are different from the Git repository source tarballs.

project-X.Y.Z-setup.exe
project-X.Y.Z-src.tar.gz

Now to get the download URL(s) for the asset(s) using the GitHub API one can get and process this URL (replacing USER and PROJECT/var> with the GitHub user account and projectname accordingly):

https://api.github.com/repos/USER/PROJECT/releases/latest

Note, that the assets download URL is provided by the browser_download_url object in the assets objects list:

{
  ...
  "assets": [
    {
      ...
      "browser_download_url": "...",
      ...
    }
  ]
}

The content provided by the API is also available to Jekyll sites hosted on GitHub pages via the site.github namespace. You can easily check all the content of this namespace using this approach (somewhere in your code):

{{ site.github | inspect }}

You'll find, that you can even access detailed author and project information. Now to get the download URL of my asset, I just access the first list entry using this:

{{ site.github.latest_release.assets[0].browser_download_url }}

or this approach (less typing):

{% assign release = site.github.latest_release %}
{{ release.assets[0].browser_download_url }}

I use this to create structured data in JSON-LD for a software application. I can even access the file size, the creation and publication date of my asset. The following shows the JSON-LD snippet I add to one of my GitHub project pages (I replaced fixed content with dots):

{% assign release = site.github.latest_release %}
{
  "@context": "http://schema.org/",
  "@type": "SoftwareApplication",
  "name": "...",
  "softwareVersion": "{{ release.tag_name | strip | remove: 'v' }}",
  "alternateName": [
    "...",
    "{{ release.name }}"
  ],
  "description": "...",
  "applicationCategory": "...",
  "inLanguage": ["..", ".."],
  "operatingSystem": [
    "...",
    "..."
  ],
  "downloadUrl": "{{ release.assets[0].browser_download_url }}",
  "fileSize": "{{ release.assets[0].size | divided_by: 1024 }}",
  "releaseNotes": "{{ release.html_url }}",
  "license": "...",
  "url": "{{ site.github.repository_url }}",
  "datePublished": "{{ release.published_at }}",
  "dateCreated": "{{ release.created_at }}",
  "author":    {%- include json/person.json -%},
  "publisher": {%- include json/publisher.json -%}
}

If there is more than one asset (the GitHub repository source tarball and zipball are not assets) one probably has to use a more flexibale approach then accessing the first list entry via asset[0] as shown above. If there are several assets and the asset file name is created the same way for every release but includes the version number (see the file name examples from the beginning of this post), there is another approach, that might be to used. One can process:

{{ site.github.latest_release.tag_name }}

and create the download URL like this

{{ site.github.releases_url }}/download/latest/foo-{{ site.github.latest_release.tag_name | strip | remove 'v' }}-setup.exe
{{ site.github.releases_url }}/download/latest/foo-{{ site.github.latest_release.tag_name | strip | remove 'v' }}-src.tar.gz

Because it is common to tag the version as vX.Y.Z the leading v is removed from the version tag in the examples above.

Using the approach above one can even loop over site.github.releaes and create a changelog/news page automatically for all releases! Maybe you can share your ideas about the suggested approaches on my GIST page.

Monday, April 30, 2018

Re-enabling right click functionality for my Thinkpad touchpad

I have a Lenovo Thinkpad Yoga 11e running Debian Sid. The touchpad has a left and a right click area at the bottem. For some reason, the right click ability recently stopped working. I have not yet found the reason, but I was able to fix it by adding the emphasized lines in /usr/share/X11/xorg.conf.d/70-synaptics.conf.


Section "InputClass"
        Identifier "Default clickpad buttons"
        MatchDriver "synaptics"
        Option "ClickPad" "true"
        Option "EmulateMidButtonTime" "0"
        Option "SoftButtonAreas" "50% 0 82% 0 0 0 0 0"
        Option "SecondarySoftButtonAreas" "58% 0 0 15% 42% 58% 0 15%"
EndSection
Edit

Nope. Stopped working again and both bottom areas act as left click. As it is working as a left click, I guess, the touchpad is physically ok. So I have no idea, what's going on :(

Edit 2

Thanks to Cole Robinson and the link he provided I found the reason and a fix. GNOME 3.28 uses the clickfinger behaviour by default now. By setting the click method from the 'fingers' (clickfinger) method back to 'areas', either using gsettings or gnome-tweaks, the right click ability is back after rebooting the system.

PS: With the new default clickfinger method, the right- and middle click are emulated using a two- and three-finger tap.

Monday, February 5, 2018

Get your namespace_id(s) for salsa.debian.org

In addition to Christoph Bergs script, to import packages to salsa.debian.org, I'd like to provide some more ways to determine the namesapce_id parameter you'll need.

Say that GROUP is the name of the group or a string of the groups name and that TOKEN is a personal access token you created. So this will work much faster:

curl --request GET -d "search=GROUP" -s https://salsa.debian.org/api/v4/groups | jq '.[].id'

If the group is not public, you need to add your token:

curl [..] --header "PRIVATE-TOKEN: TOKEN" [..]

The command might provide several IDs, if the search term matches several groups. In this case, you might want to have a look at the raw output without piping it to jq or look at the result of ...

https://salsa.debian.org/api/v4/groups/?search=GROUP

... in a browser. Interestingly, the latter doesn't provide any output in the browser, if you are not part of the group. But it provides the necessary information using curl. Bug or feature?

Another way that works is, to look at the output of...

https://salsa.debian.org/api/v4/namespaces

... when you are logged in or ...

curl --request GET --header "PRIVATE-TOKEN: TOKEN" https://salsa.debian.org/api/v4/namespaces

This has the advantage of also getting the namespace_id for your personal repository, say for e.g. importing projects hosted on https://people.debian.org/~USER/ or https://anonscm.debian.org/cgit/users/USER/.

Wednesday, January 31, 2018

Migrating the debichem group subversion repository to Git - Part 1: svn-all-fast-export basics

With the deprecation of alioth.debian.org the subversion service hosted there will be shut down too. According to lintian the estimated date is May 1st 2018 and there are currently more then 1500 source packages affected. In the debichem group we've used the subversion service since 2006. Our repository contains around 7500 commits done by around 20 different alioth user accounts and the packaging history of around 70 to 80 packages, including packaging attempts. I've spent the last days to prepare the Git migration, comparing different tools, controlling the created repositories and testing possibilities to automate the process as much as possible. The resulting scripts can currently be found here.

Of course I began as described at the Debian Wiki. But following this guide, using git-svn and converting the tags with the script supplied under rubric Convert remote tags and branches to local one gave me really weird results. The tags were pointing to the wrong commit-IDs. I thought, that git-svn was to blame and reported this as bug #887881. In the following mail exchange Andreas Kaesorg explained to me, that the issue is caused by so-called mixed-revision-tags in our repository as shown in the following example:

$ svn log -v -r7405
------------------------------------------------------------------------
r7405 | dleidert | 2018-01-17 18:14:57 +0100 (Mi, 17. Jan 2018) | 1 Zeile
Geänderte Pfade:
   A /tags/shelxle/1.0.888-1 (von /unstable/shelxle:7396)
   R /tags/shelxle/1.0.888-1/debian/changelog (von /unstable/shelxle/debian/changelog:7404)
   R /tags/shelxle/1.0.888-1/debian/control (von /unstable/shelxle/debian/control:7403)
   D /tags/shelxle/1.0.888-1/debian/patches/qt5.patch
   R /tags/shelxle/1.0.888-1/debian/patches/series (von /unstable/shelxle/debian/patches/series:7402)
   R /tags/shelxle/1.0.888-1/debian/rules (von /unstable/shelxle/debian/rules:7403)

[svn-buildpackage] Tagging shelxle 1.0.888-1
------------------------------------------------------------------------

Looking into the git log, the tags deteremined by git-svn are really not in their right place in the history line, even before running the script to convert the branches into real Git tags. So IMHO git-svn is not able to cope with this kind of situation. Because it also cannot handle our branch model, where we use /branch/package/, I began to look for different tools and found svn-all-fast-export, a tool created (by KDE?) to convert even large subversion repositories based on a ruleset. My attempt using this tool was so successful (not to speak of, how fast it is), that I want to describe it more. Maybe it will prove to be useful for others as well and it won't hurt to give some more information about this poorly documented tool :)

Step 1: Setting up a local subversion mirror

First I suggest setting up a local copy of the subversion repository to migrate, that is kept in sync with the remote repository. This can be achieved using the svnsync command. There are several howtos for this, so I won't describe this step here. Please check out this guide. In my case I have such a copy in /srv/svn/debichem.

Step 2: Creating the identity map

svn-all-fast-export needs at least two files to work. One is the so called identity map. This file contains the mapping between subversion user IDs (login names) and the (Git) committer info, like real name and mail address. The format is the same as used by git-svn:

loginname = author name <mail address>

e.g.

dleidert = Daniel Leidert <dleidert@debian.org>

The list of subversion user IDs can be obtained the same way as described in the Wiki:

svn log SVN_URL | awk -F'|' '/^r[0-9]+/ { print $2 }' | sort -u

Just replace the placeholder SVN_URL with your subversion URL. Here is the complete file for the debichem group.

Step 3: Creating the rules

The most important thing is the second file, which contains the processing rules. There is really not much documentation out there. So when in doubt, one has to read the source file src/ruleparser.cpp. I'll describe, what I already found out. If you are impatient, here is my result so far.

The basic rules are:

create repository REPOSITORY
...
end repository

and

match PATTERN
...
end match

The first rule creates a bare git repository with the name you've chosen (above represented by REPOSITORY). It can have one child, that is the repository description to be put into the repositories description file. There are AFAIK no other elements allowed here. So in case of e.g. ShelXle the rule might look like this:

create repository shelxle
description packaging of ShelXle, a graphical user interface for SHELXL
end repository

You'll have to create every repository, before you can put something into it. Else svn-all-fast-export will exit with an error. JFTR: It won't complain, if you create a repository, but don't put anything into it. You will just end up with an empty Git repository.

Now the second type of rule is the most important one. Based on regular expression match patterns (above represented by PATTERN), one can define actions, including the possibility to limit these actions to repositories, branches and revisions. The patterns are applied in their order of appearance. Thus if a matching pattern is found, other patterns matching but appearing later in the rules file, won't apply! So a special rule should always be put above a general rule. The patterns, that can be used, seem to be of type QRegExp and seem like basic Perl regular expressions including e.g. capturing, backreferences and lookahead capabilities. For a multi-package subversion repository with standard layout (that is /PACKAGE/{trunk,tags,branches}/), clean naming and subversion history, the rules could be:

match /([^/]+)/trunk/
  repository \1
  branch master
end match

match /([^/]+)/tags/([^/]+)/
  repository \1
  branch refs/tags/debian/\2
  annotated true
end match

match /([^/]+)/branches/([^/]+)/
  repository \1
  branch \2
end match

The first rule captures the (source) package name from the path and puts it into the backreference \1. It applies to the trunk directory history and will put everything it finds there into the repository named after the directory - here we simply use the backreference \1 to that name - and there into the master branch. Note, that svn-all-fast-export will error out, if it tries to access a repository, which has not been created. So make sure, all repositories are created as shown with the create repository rule. The second rule captures the (source) package name from the path too and puts it into the backreference \1. But in backreference \2 it further captures (and applies to) all the tag directories under the /tags/ directory. Usually these have a Debian package version as name. With the branch statement as shown in this rule, the tags, which are really just branches in subversion, are automatically converted to annotated Git tags (another advantage of svn-all-fast-export over git-svn). Without enabling the annotated statement, the tags created will be lightweight tags. So the tag name (here: debian/VERSION) is determined via backreference \2. The third rule is almost the same, except that everything found in the matching path will be pushed into a Git branch named after the top-level directory captured from the subversion path.

Now in an ideal world, this might be enough and the actual conversion can be done. The command should only be executed in an empty directory. I'll assume, that the identity map is called authors and the rules file is called rules and that both are in the parent directory. I'll also assume, that the local subversion mirror of the packaging repository is at /srv/svn/mymirror. So ...

svn-all-fast-export --stats --identity-map=../authors.txt --rules=../debichem.rules --stats /srv/svn/mymirror

... will create one or more bare Git repositories (depending on your rules file) in the current directory. After the command succeeded, you can test the results ...

git -C REPOSITORY/ --bare show-ref
git -C REPOSITORY/ --bare log --all --graph

... and you will find your repositories description (if you added one to the rules file) in REPOSITORY/description:

cat REPOSITORY/description

Please note, that not all the debian version strings are well formed Git reference names and therefor need fixing. There might also be gaps shown in the Git history log. Or maybe the command didn't even suceed or complained (without you noticing it) or you ended up with an empty repository, although the matching rules applied. I encountered all of these issues and I'll describe the cause and fixes in the next blog article.

But if everything went well (you have no history gaps, the tags are in their right place within the linearized history and the repository looks fine) and you can and want to proceed, you might want to skip to the next step.

In the debichem group we used a different layout. The packaging directories were under /{unstable,experimental,wheezy,lenny,non-free}/PACKAGE/. This translates to /unstable/PACKAGE/ and /non-free/PACKAGE/ being the trunk directories and the others being the branches. The tags are in /tags/PACKAGE/. And packages, that are yet to upload are located in /wnpp/PACKAGE/. With this layout, the basic rules are:

# trunk handling
# e.g. /unstable/espresso/
# e.g. /non-free/molden/
match /(?:unstable|non-free)/([^/]+)/
  repository \1
  branch master
end match

# handling wnpp
# e.g. /wnpp/osra/
match /(wnpp)/([^/]+)/
  repository \2
  branch \1
end match

# branch handling
# e.g. /wheezy/espresso/
match /(lenny|wheezy|experimental)/([^/]+)/
  repository \2
  branch \1
end match

# tags handling
# e.g. /tags/espresso/VERSION/
match /tags/([^/]+)/([^/]+)/
  repository \1
  annotated true
  branch refs/tags/debian/\2
  substitute branch s/~/_/
  substitute branch s/:/_/
end match

In the first rule, there is a non-capturing expression (?: ... ), which simply means, that the rule applies to /unstable/ and /non-free/. Thus the backreference \1 refers to second part of the path, the package directory name. The contents found are pushed to the master branch. In the second rule, the contents from the wnpp directory are not pushed to master, but instead to a branch called wnpp. This was necessary because of overlaps between /unstable/ and /wnpp/ history and already shows, that the repositories history makes things complicated. In the third rule, the first backreference \1 determines the branch (note the capturing expression in contrast to the first rule) and the second backreference \2 the package repository to act on. The last rule is similar, but now \1 determines the package repository and \2 the tag name (debian package version) based on the matching path. The example also shows another issue, which I'd like to explain more in the next article: some characters we use in debian package versions, e.g. the tilde sign and the colon, are not allowed within Git tag names and must therefor be substituted, which is done by the substitute branch EXPRESSION instructions.

Step 4: Cleaning the bare repository

The tool documentation suggests to run ...

git -C REPOSITORY/ repack -a -d -f

... before you upload this bare repository to another location. But Stuart Prescott told me on the debichem list, that this might not be enough and still leave some garbage behind. I'm not experienved enough to judge here, but his suggestion is, to clone the repository, either a bare clone or clone and init a new bare. I used the first approach:

git -C REPOSITORY/ --bare clone --bare REPOSITORY.git
git -C REPOSITORY.git/ repack -a -d -f

Please note, that this won't copy the repositories description file. You'll have to copy it manually, if you wanna keep it. The resulting bare repository can be uploaded (e.g. to git.debian.org as personal repository:

cp REPOSITORY/description REPOSITORY.git/description 
touch REPOSITORY.git/git-daemon-export-ok
rsync -avz REPOSITORY.git git.debian.org:~/public_git/

Or you clone the repository, add a remote origin and push everything there. It is even possible to use the gitlab API at salsa.debian.org to create a project and push there. I'll save the latter for another post. If you are hasty, you'll find a script here.

Sunday, January 14, 2018

Make 'bts' (devscripts) accept TLS connection to mail server with self signed certificate

My mail server runs with a self signed certificate. So bts, configured like this ...

BTS_SMTP_HOST=mail.wgdd.de:587
BTS_SMTP_AUTH_USERNAME='user'
BTS_SMTP_AUTH_PASSWORD='pass'

...lately refused to send mails with this error:

bts: failed to open SMTP connection to mail.wgdd.de:587
(SSL connect attempt failed error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed)

After searching a bit, I found a way to fix this locally without turning off the server certificate verification. The fix belongs into the send_mail() function. When calling the Net::SMTPS->new() constructor, it is possible to add the fingerprint of my self signed certificate like this (bold):

if (have_smtps) {
    $smtp = Net::SMTPS->new($host, Port => $port,
        Hello => $smtphelo, doSSL => 'starttls',
        SSL_fingerprint => 'sha1$hex-fingerprint')
        or die "$progname: failed to open SMTP connection to $smtphost\n($@)\n";
} else {
    $smtp = Net::SMTP->new($host, Port => $port, Hello => $smtphelo)
        or die "$progname: failed to open SMTP connection to $smtphost\n($@)\n";
}

Pretty happy to being able to use the bts command again.

Tuesday, December 19, 2017

Silikonreste/Silikonklebereste im Aquarium entfernen

Dies ist ein Tip, um Silikonreste von ehemals verklebten Innenfiltern und Strukturwänden sauber und rückstandslos zu entfernen. Die fragliche Stelle muss trocken sein, also mindestens über der Wasseroberfläche liegen.

Zunächst nehme ich einen Ceranfeld- oder Glasschaber. Im Prinzip ist dies nicht anderes als ein Gerät, mit einer scharfen Rasierklinge im Kopf. Mit dem Schaber wird zunächst das Silikon soweit wie möglich vorsichtig abgetragen. Bei korrekter Anwendung ist das für das Glas selbst auch ungefährlich. In der Regel verbleibt danach nur noch eine sehr dünne Schicht Silikon auf dem Glas. Um diese nun ebenfalls zu entfernen, trage ich vorsichtig mit einem Pinsel Silikonentferner (fertiges Lösemittelgemisch aus dem Baumarkt) auf die fragliche Stelle auf und lasse es einwirken. Hierbei sollte man grundsätzlich sehr vorsichtig sein, da ja alle Verklebungen am Aquarium aus Silikon sind. Nach der vorgeschriebenen Einwirkzeit kann nun auch die dünne Silikonschicht mit dem Schaber recht einfach entfernt werden. Evtl. ist an wenigen Stellen die Behandlung zu wiederholen. Die Stelle kann und sollte im Anschluss mit Aceton o. ä. vorsichtig gereinigt werden, um die Resten des Silikonentferners ebenfalls zu entfernen. Wer möchte, wischt feucht nach. Das Glas ist danach rückstandsfrei sauber.

Das Ganze funktioniert im Übrigen auch auf dem Innenfilter selbst. Allerdings sind hier bei der mechanischen Bearbeitung mit dem Schaber Kratzer möglich!