Activity Workshop

Murmeli - Development

Yes, development is ongoing. The ideas have been bouncing around for half a decade now, and I was beginning to wonder whether it would ever reach the coding stage, because the problems seemed too enormous and complex to solve. Yet here we are, and work has now started on the foundations of this new platform.

As you've seen, the working title is now Murmeli rather than Murmur. But many of the ideas remain the same. It's just become a little clearer how to actually implement some of these ideas now.

Building blocks

For a long time, the thinking was to either contribute to, or extend or adapt, or to build on top of, one of the existing platforms such as OneSwarm or RetroShare. But these codebases are huge, and fluid, and so the direction has changed. Work has started using a combination of building blocks, which should be available on all the platforms we need to support.

For Debian, the required packages are python3, python3-pyqt4, tor, python3-pysocks, gnupg, python3-gnupg, python3-tempita and python3-pil (all from Debian stable).

Update: python3-pysocks might be called python3-socks instead now, also for the Qt5 branch you'll need python3-pyqt5 and python3-pyqt5.qtwebengine.

For Windows, you'll have to find, download and install all of these dependencies yourself - they come in a mix of .exe and .msi installers, .tar.gz python tarballs and .egg python eggs. In particular, the tor installer is quite difficult to find, as for Murmeli you only need the tor exe and not the whole browser bundle.


It's still at a relatively early stage, but the fundamentals are all working. We have a Qt GUI, able to show html pages with css, javascript and jquery, and we have connection to the database, GPG and tor. We also have a separate startup wizard, which can guide the user to set everything up, checking dependencies and specifying paths for the local database and the various executables such as tor and gpg.

For an idea of the current development status, see the new (silent) demonstration videos on youtube:

These screenshots show the flow through the startup wizard, where the language is selectable on the first panel. Subsequent panels lead through the steps in turn, if necessary setting up the database, generating an asymmetric keypair for the encryption, and starting one's own profile.

startup wizard

Introduction page

startup wizard

Checking dependencies

startup wizard

Defining paths

startup wizard

Starting services

startup wizard

Key generation

startup wizard

Key selection

startup wizard


The required services are now started, and the main application can be launched. Then we come to the main window shown in the screnshots below. The main bit of the Murmeli window is a QWebView from Qt (now called QWebEngineView), giving the flexibility of HTML display for showing messages, profiles and so on. In the Settings panel you can change the language, currently just English and German are supported.

main window

Basic GUI with main menu items

window icon


profile page

Profile page from the database
with avatar icon

edit own details

Options to edit
one's own details

add new contact

Adding a new contact
using their id

show incoming requests

Listing of incoming
contact request messages

check key fingerprints

Instructions for checking
key fingerprints

incoming message

Incoming regular message

settings page

Settings page
for changing language

new message

Composing a
new message

message tree

A tree of
message replies

friend graph

Drawing of friend graph

led graph

Send and receive statistics

These screenshots show a selection of the currently working functionality, including showing a contact list, requesting a new contact, checking fingerprints and showing messages. These are the functions which Murmeli is now able to do:

Excitingly, Murmeli has now not only sent its first ever message successfully through tor, but also sent its first ever encrypted message successfully through tor! Both databases have now been updated with each other's details, and the public keys have been added to each other's keyrings, so the basic connections have been established. We've also now got the exchange of profile information, and for the first time also blind relaying between trusted friends. And the best thing about this is that apart from the initial key generation during the startup wizard, there has been no mention of encryption in the Murmeli GUI. That means, no need to enter passwords (there aren't any), no need to choose encryption schemes or strengths (there is none of the confusion about S/MIME or GnuPG or inline PGP you get with encrypted email), no need to choose whether you want things to be sent encrypted or not, it just does it.

Ok, since the addition of the key confirmation then there is at least some extra complexity and some mention of the care required to verify the secure connection, but hopefully the codeword scheme is a little bit friendlier than reading out long alphanumeric key fingerprints to each other.

As shown in one of these screenshots, the code from Brainstorm can be used to display graphically your network of friends. If your friends allow it, you can see who your friends' friends are, and connect those up too. And of course the graph wobbles in a pleasing way. It would be nice to be able to refer contacts to each other from this graph too.

The last photo shows (faked) send and receive statistics displayed on a Scrollbot. The idea is that one trace is shown with bright dots rising from the bottom of the display, and the other trace has half-brightness with the zero value at the top of the screen. So one could be number of messages sent per minute and one the number received, with the screen showing each of the last fourteen minutes. This is just one suggestion so far.

Don't take too much notice of the exact presentation of these pages just yet - work is concentrating on functionality rather than visual design. All the pages are built with tempita templates, html and css (and a little jquery) so the layout can easily be changed once everything is working properly. Eventually it may be possible to customise the display with different themes, for example.

Not all of the current development code is ready for publishing yet, but the basics can now be found on Gitlab. At the time of writing, this includes a frozen main branch with a functional prototype, and a separate "redesign" branch being updated on a semi-regular basis. This side-branch has taken several steps backwards in terms of what works and what doesn't, but functionality is steadily returning.

Can Johnny encrypt now?

There have been many reports over the last two decades about how difficult PGP is, how unfriendly encrypted email is, and how much difficulty even experienced computer users have with the concepts of keys, keyrings and encryption schemes. All too often, users get so frustrated trying to set things up that they either fail or give up. And even then, it is all too easy to accidentally send something unencrypted or wrongly encrypted. Hopefully Murmeli's process will prove to be much simpler, much more user-friendly, and much more difficult to incorrectly use.

With Murmeli, you don't choose how to import public keys, you don't choose which keys to encrypt with, and you never choose whether to encrypt things or not. There still remains the question about how to satisfy the user that things really are being sent encrypted, but the aim is to keep that as hidden as possible. Perhaps an additional log window (which only the really curious would open!) could show the actual bytes being sent and received - which would of course make no sense to anybody apart from the unencrypted headers.

Murmeli ids

As already explained, Murmeli uses Tor's so-called "hidden services" (now called "onion services") to communicate between clients. In order to do this, tor generates an identifier for the service to publish, and this becomes your Murmeli id which you can give to your friends. Currently these ids are always 16-character, alphanumeric strings (like "06zyyt0xf0mt7gw8"), as determined by the tor protocol. Therefore, as it stands, if I wanted to give you my Murmeli id over the phone, the id is short enough that I could read it out and you could type it in, without too much inconvenience.

However, tor development doesn't stand still, and there are currently proposals in the works to change this to make all the ids longer (presumably to expand the id space and make it possible to publish more such services without collisions, and make deliberate collisions more difficult). Makes sense. Except that means the ids may be in future 52 characters long instead of 16, and that would make it really awkward to exchange ids except by copy-pasting. It would also mean changes to Murmeli's assumptions about id length, of course, and raises the questions about whether the two id formats will operate concurrently or whether all the current 16-character ids will become invalid at some point. We'll keep an eye on this.

Replacing Mongodb

This is certainly something which wasn't planned, but it became apparent that Mongodb isn't going to be a good long-term choice for our database storage. It was with significant reluctance that the Mongodb was removed and a much simpler, file-based solution was introduced instead.

So why change, and waste the spent effort? A combination of things. Firstly, although Mongodb does work, there's a sizeable amount of complexity in starting the database server (including setting up the necessary internal authentication), and this proved to be unreliable. On Windows, we couldn't tell whether it was running or not and the behaviour is different from that on linux. On linux, the service was often started for us by the system at boot time, so we needed 'sudo' rights to stop this service to allow us to start our own. Sometimes the startup time of this service was longer than expected, so Murmeli thought that the database hadn't been setup yet, and launched the startup wizard. All of these problems grew to be small but annoying distractions.

Plus, and it happened a few times, the Murmeli database can become "corrupted", requiring it to be "repaired", which requires searching on the internet for the right command to repair the database.

In addition, requiring Windows users to find, download and install the mongodb installer is a burden - it's over a hundred megabytes and is confusingly listed only for "Windows Server 2008 R2 or later". It doesn't work on 32-bit platforms and requires a "hotfix" for Windows 7. It became a barrier to entry.

Furthermore, it's a real load on the file system. Even if the database itself is tiny, the binary files it generates easily grow to over 100 MB. And as if that wasn't enough, it also reserves for itself over 700 MB of "pre-allocated" space in /var/lib/mongodb/journal, presumably for some kind of journalling. If you've got a terabyte harddrive this probably won't be a problem, but if you've got a raspberry pi using one of the partitions on an SD card, then you can risk filling the card and making your system unbootable. Not ideal.

So what's the alternative? Well, Murmeli now uses a local, python-based solution instead of Mongo. It allows multi-threading but doesn't require the launching of a separate server process, so it's more robust, and it's much more testable now too. Murmeli should work as before, but with the advantages of reduced external dependencies, faster, simpler and more reliable startup, lower resources usage, greater robustness and easier debugging.

Replacing PyQt4

So the (temporary) status was that Murmeli was working on Debian Stable, Raspbian Stable and Linux Mint. Except for the intermittent problem of SegmentationFaults with the WebView of Qt4. Given that Debian Stable (the new stable, "Stretch") now includes Qt5, and Qt5 recommends to use the new WebEngine components rather than the old WebView components, it seems that a migration to Qt5 and PyQt5 is in order. The WebEngine is based on Chromium, rather than Webkit, and should hopefully prove more resilient to crashes as well as being more future-proof.

There are just a few little problems to solve though - firstly the syntax of the signals and slots and connects has changed, so all those had to be modified, and many of the classes moved packages.

Secondly, the WebEngine does allow the possibility of intercepting form POSTs, but doesn't appear to allow access to the POSTed form data (as the QtNetwork.QNetworkAccessManager for the WebView used to). So Murmeli can tell that a form was submitted, but can't read the contents. I would assume this is just an oversight, but it seems like it's been like this for several releases of Qt5 so it seems deliberate. Neither the QWebEnginePage nor the QWebEngineUrlRequestInterceptor let you access the form data, just the URL. A real backwards step from Qt4.

One possible solution to this could be to send all forms with "GET" requests instead, as long as this doesn't overstep some URL length boundaries. Of course this forces URL encoding on all the parameter values, and here comes a very strange feature of Qt. With the URL encoding, a simple string like "om pom+" becomes "om+pom%2B", so spaces become "+" and plusses become percent encoded. And does Qt provide a way to get back to the decoded string? No, you can only choose between "om+pom+" and "om+pom%2B". There appears to be no way using either QUrl or QUrlQuery to get back to the actual parameter value. Fortunately, at least under python, urllib provides some sanity to save the day.

At the moment it is still hoped that form gets will be sufficient, but if not then maybe Qt5's QWebChannel could be an alternative, or in the worst case it may be necessary to look at alternatives to Qt (which would be a much bigger prospect).

And finally the WebEngine modules currently aren't available in Raspbian Stable, so it looks like Murmeli's Qt GUI will stop working on the Raspberry Pi, at least for now (and possibly for ever). Instead, the 'redesign' branch on GitHub is providing a gui-less option without Qt for robot relays, so with that at least headless robots (and LED robots!) can run on the Pi. Tests are ongoing with a Raspberry Pi 1 and a Raspberry Pi Zero W (in the form of a Scrollbot).