Planet

Subscribe to the RSS feed of this planet! RSS
Aaron Seigo's picture
Wed, 2015-04-01 08:58

Working with Kolab has kept me busy on numerous fronts since I joined near the end of last year. There is the upcoming Kolab Summit, refreshing Kolab Systems' messaging, helping with progress around Kolab Now, collaborating on development process improvement, working on the design and implementation of Akonadi Next, the occassional sales engineering call ... so I've been kept busy, and been able to work with a number of excellent people in the process both in Kolab Systems and the Kolab community at large.

While much of that list of topics doesn't immediately bring "writing code" to mind, I have had the opportunity to work on a few "hands on keyboard, writing code" projects. Thankfully. ;)

One of the more interesting ones, at least to me, has been work on an emerging data loss prevention and audit trail system for Kolab called Bonnie. It's one of those things that companies and governmental users tend to really want, but which is fairly non-trivial to achieve.

There are, in broad strokes, three main steps in such a system:

  1. Capturing and recording events
  2. Storing data payloads associated with those events
  3. Recreating histories which can be reviewed and even restored from

I've been primarily working on the first two items, while a colleague has been focusing on the third point. Since each of these points is a relatively large topic on their own, I'll be covering each individually in subsequent blog entries.

We'll start in the next blog entry by looking at event capture and storage, why it is necessary (as opposed to simply combing through system logs, e.g.) and what we gain from it. I'll also introduce one of the Bonnie components, Egara, which is responsible for this set of functionality.


Aaron Seigo's picture
Fri, 2015-03-27 18:47

Today "everyone" is online in one form or another, and it has transformed how many people connect, communicate, share and collaborate with others. To think that the Internet really only hit the mainstream some 20 years ago. It has been an amazingly swift and far-reaching shift that has touched people's personal and professional lives.

So it is no surprise that the concept of eGovernment is a hot one and much talked about. However, the reality on the ground is that governments tend not to be the swiftest sort of organizations when it comes to adopting change. (Which is not a bad thing; but that's a topic for another blog perhaps.) Figuring out how to modernize the communication and interaction of government with their constituencies seems to largely still be in the future. Even in countries where everyone is posting pictures taken on their smartphones of their lunch to all their friends (or the world ...), governments seem to still be trying to figure out how to use the Internet as an effective tool for democratic discourse.

The Netherlands is a few steps ahead of most, however. They have an active social media presence which is used by numerous government offices to collaborate with each other as well as to interact with the populace. Best of all, they aren't using a proprietary, lock-in platform hosted by a private company oversees somewhere. No, they use a free software social media framework that was designed specifically for this: Pleio.

They have somewhere around 100,000 users of the system and it is both actively used and developed to further the aims of the eGovernment initiative. It is, in fact, an initiative of the Programme Office 2.0 with the Treasury department, making it a purposeful program rather than simply a happy accident.

In their own words:

The complexity of society and the need for citizens to ask for an integrated service platform where officials can easily collaborate with each other and engage citizens.

In addition, hundreds of government organizations all have the same sort of functionality needed in their operations and services. At this time, each organization is still largely trying to reinvent the wheel and independently purchase technical solutions.

That could be done better. And cheaper. Gladly nowadays new resources are available to work together government-wide in a smart way and to exchange knowledge. Pleio is the platform for this.

Just a few days ago it was anounced publicly that not only is the Pleio community is hard at work on improving the platform to raise the bar yet again, but that Kolab will be a part of that. A joint development project has been agreed to and is now underway as part of a new Pleio pilot project. You can read more about the collaboration here.


Fri, 2015-03-27 17:44

Kolab has joined Pleio to aid in the production of a new version of the Pleio social media platform. Pleio is currently used by some 100,000 people within the Dutch government to interact with each other and their constituencies as part of the nation's eGovernment initiative. The new pilot project was announced by Kolab Systems CEO Georg Greve at the Netfutures conference on March 25 in Brussels, and underscores the participating organizations' shared fundamental vision with regards to open source software, open standards and data security.

The goal of the pilot is to to provide Pleio with a broader, more modular base architecture by integrating components from the Kolab collaboration suite into its core. Both products will remain independent, continuing to be developed within their respective communities, but efforts will be bundled wherever possible with a focus on strengthening their ecosystems together.

The Kolab 4 Pleio pilot project was announced along with a call for participation on the Pleio website and development planning has begun.

About Pleio

Today's citizens are asking for an integrated service platform where officials can easily collaborate with each other and engage citizens, and hundreds of government organizations share this exact same need. Historically, each organization has largely tried to reinvent the wheel and independently purchase technical solutions. That could be done better and more economically. Pleio is the open platform that provides a social network with integrated blogging, groups and meetings, wikis, and file sharing designed for use in eGovernment.

Pleio is an initiative of the Programme Officer 2.0 with the Treasury in the Netherlands in coordination with other Dutch governmental agencies.

About Kolab

Kolab Groupware is a Free Software collaboration system providing email, events and appointments, contacts, notes, file cloud and more that scales from the home office to enterprise deployments. All major desktop and mobile systems are supported, thanks to broad usage of open standards and formats.


roundcube's picture
Wed, 2015-03-18 01:00

This is the first service release to update the stable version 1.1. It contains
some important bug fixes and improvements in error handling as well as a few
new features and configuration options. See the full changelog here.

It’s considered stable and we recommend to update all productive installations
of Roundcube with this version. Download it from roundcube.net/download,

Please do backup your data before updating!


Aaron Seigo's picture
Mon, 2015-03-16 11:52

We just announced that registration and presentation proposal submission is now open for the Kolab Summit 2015 which is being held in The Hague on May 2-3.

Just as Kolab itself is made up of many technologies, many technologies will be present at the summit. In addition to topics on Kolab, there will be presentations covering Roundcube, KDE Kontact and Akonadi, cyrus imap, and OpenChange among others. We have some pretty nifty announcements and reveals already lined up for the event, which will be keynoted by George Greve (CEO of Kolab Systems AG) and Jeroen van Meeuwen (lead Kolab architect). Along with the usual BoFs and hacking rooms, this should be quite an enjoyable event.

As an additional and fun twist, the Kolab Summit will be co-located with the openSUSE conference which is going on at the same time. So we'll have lots of opportunity for "hallway talks" with Geekos as well. In fact, I'll be giving a keynote presentation at the openSUSE conference about freedom as innovation. A sort of "get the engines started" presentation that I hope provokes some thought and gets some energy flowing.


Aaron Seigo's picture
Mon, 2015-03-16 11:52

Kolab Summit 2015

We just announced that registration and presentation proposal submission is now open for the Kolab Summit 2015 which is being held in The Hague on May 2-3.

Just as Kolab itself is made up of many technologies, many technologies will be present at the summit. In addition to topics on Kolab, there will be presentations covering Roundcube, KDE Kontact and Akonadi, cyrus imap, and OpenChange among others. We have some pretty nifty announcements and reveals already lined up for the event, which will be keynoted by George Greve (CEO of Kolab Systems AG) and Jeroen van Meeuwen (lead Kolab architect). Along with the usual BoFs and hacking rooms, this should be quite an enjoyable event.

As an additional and fun twist, the Kolab Summit will be co-located with the openSUSE conference which is going on at the same time. So we'll have lots of opportunity for "hallway talks" with Geekos as well. In fact, I'll be giving a keynote presentation at the openSUSE conference about freedom as innovation. A sort of "get the engines started" presentation that I hope provokes some thought and gets some energy flowing.


Mon, 2015-03-16 10:38

The Kolab Collaboration Suite has been adopted by companies and governments around the world, making it one of most successful "poster children" for Free Software and Open Standards. In order to chart the next steps forward, the Kolab community is excited to announce the inaugural Kolab Summit to be held in The Hague on May 2-3, 2015.

Come and join lead developers, deployment experts and Kolab partners for two days packed with talks, code sprints and social events! The Kolab Summit is your best opportunity to meet and network with the community which has grown around Kolab since its first release in 2003. Registration is free but space is limited, so be sure to register soon.

Keynotes from Georg Greve, CEO of Kolab Systems AG, and Jeroen van Meeuwen, lead Kolab architect, will open the speaking schedule, providing a look at what is coming for Kolab in 2015 and beyond. Lead developers from Roundcube, the worlds most popular webmail application, along with key participants from KDE Kontact, cyrus-imap, Seafile, OpenChange and more will in attendance and presenting talks.

In addition to development topics, deployment and support of Kolab will be another hot topic at the event. Leading experts in Kolab deployment will be in attendance, and talks covering deployment strategies such Docker are already scheduled.

There are still several slots open in the speaking schedule, however. The call for presentations also begins today and is open to everyone interested in presenting a talk at the Summit. Submissions should be made before April 1st for consideration, with confirmations of accepted proposals happening directly afterwards. The submission form is available once you are registered.

There will also be workshop and development rooms where attendees can gather to discuss topics of interest and work on the Kolab code base in person. These two intense days focused on the future of free, secure and open collaboration will be co-located alongside the openSUSE Conference 2015, expanding the possibilities even further.

The entire Kolab community hopes to see you there!


Andreas Cordes's picture
Fri, 2015-03-13 22:20

Hello,

Seafile is an open source cloud software which is free for private use.

There is also a professional edition which is not necessary for my needs. I just want to sync files across more than one device. In the past I used ownCloud which was pretty good for my needs.

First I decided to have ownCloud integrated in Kolab as a backend but I had a bit lack of time so developing a new driver for chwala was not so easy.

After a while I noticed that Seafile is integrated in chwala 0.3.0 and with Kolab 3.4 it is quite stable to install.

Last week I managed all the stuff and here is my Step-By-Step Guide:

  1. Install Kolab 3.4 and test it
    It's obvious that you need Kolab for this. Please refer to the kolab.org web page for an installation guide for your distro
  2. Install seafile and test it.
    The homepage of seafile will guide you through all the stuff you need for that. So just download, extract and run it.
  3. Connect Seafile to Kolab-LDAP
    Following Seafile->Using LDAP is exactly what I did. My installation was spreaded over /opt/seafile, /opt/seafile/seafile-server-latest and /mnt/seafile
    But you'll find your ccnet.conf and add the LDAP part to it (please change the values according to your installation):
    [LDAP]
    HOST = ldap://127.0.0.1
    BASE = ou=People,dc=example,dc=com
    USER_DN = cn=directory manager
    PASSWORD = youdon'tknowjack
    LOGIN_ATTR = mail
  4. Now test seafile again if you can now login with your kolab main mailaddress.
    Please keep in mind that each user has to login for the first time to seafile in order to get the right folders.
  5. If everything is ok, now you have to use Apache as a proxy for Seafile
    Following Seafile -> Deploy with Apache was ok for me for the first time.
    Well, this will break your Kolab if you follow the steps directly.
  6. If this is ok as well, now fine tune a bit the Apache configuration for Seafile.
    These lines should be changed in your apache.conf (or vhost):
    #
    # seahub
    #
    RewriteRule ^/(media.*)$ /$1 [QSA,L,PT]
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteRule ^(.*)$ /seahub.fcgi$1 [QSA,L,E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]

    This will redirect all request to Seafile, but if you only have one SSL certificate and one domain you have to ignore this for all the Kolab modules.
    So please add the following RewriteCond to the config:

    RewriteRule ^/(media.*)$ /$1 [QSA,L,PT]
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_URI} !^/roundcubemail.*
    RewriteCond %{REQUEST_URI} !^/Microsoft.*
    RewriteCond %{REQUEST_URI} !^/iRony.*
    RewriteCond %{REQUEST_URI} !^/chwala.*
    RewriteCond %{REQUEST_URI} !^/kolab.*
    RewriteRule ^(.*)$ /seahub.fcgi$1 [QSA,L,E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]

    The first RwriteCond tells Apache to rewrite the URL if the file does not exist. For the Kolab installation there are some URL more not to check against an existing file because they are an Alias in the Apache way. So we ignore this rewrite when the URL starts with "roundcube", "Microsoft" (for active sync) and so on.
    Reading the conf above should be:

    Rewrite all URL starting with /media
    ReWrite the URL when the requested file does not exist (! -f)
    AND (is implicitly) the request does not start with roundcube (!/^/roundcubemail.*)
    AND the request does not start with Microsoft
    AND the request does not start with iRony
    AND the erquest does not start with chwala
    AND the request does not start with kolab
    TO /seahub.fcgi PLUS the request string including the trailing slash (^(.*)$ /seahub.fcgi$1)

  7. Now test your Kolab and Seafile again, everything should now work on the same server.
  8. It's time to combine Chwala with Seafile. The most interesting part. :-)
    Please edit your /etc/roundcubemail/config.inc.php file and all the Seafile stuff:
    // seafile
    $config['fileapi_backend'] = 'seafile';
    $config['fileapi_seafile_host'] = 'www.example.com';
    $config['fileapi_seafile_ssl_verify_peer'] = false;
    $config['fileapi_seafile_ssl_verify_host'] = false;
    $config['fileapi_seafile_cache_ttl'] = '14d';
    $config['fileapi_seafile_debug'] = true;
  9. That's it.

Now you can use your Seafile server in Kolab as a file storage. But keep in mind, password protected folders are not accessible through chwala this way.

Feel free to leave any comments and
Greetz

Timotheus Pokorra's picture
Thu, 2015-03-05 08:20

Previously, my nightly builds of Kolab have been built and hosted here:

https://obs.kolabsys.com/project/show/home:tpokorra:branches:Kolab:Development

There were 2 problems recently:

  • I was not able to add builds for CentOS7. The error message is:
    Failed to add project or repository: unable to walk on path 'CentOS:7/standard'
  • Each time there was a new patch or release on Kolab:Development, I needed to manually remove the patch from the nightly package because the patch was already part of git master. I also needed to resolve conflicts in the debian.changelog and package dsc file because the version number changed.

UPDATE: since there was an easy fix for Point 1 (just edit the Meta file of the project to add different Operating System), I also realised I could update the OBS packaging instructions the same way as I do for LBS, avoiding conflicts. So we will see if that works over the coming weeks…

After some improvements to my LightBuildServer, I am now able to build the nightly packages on my own LBS server.

This is done in two steps:

  1. Updating the package instructions: This is done by an LBS job, that basically is running this script: The package instructions are downloaded from https://obs.kolabsys.com/project/show/Kolab:Development, and the source tarballs are downloaded from https://git.kolab.org/. Then the changelog is updated, patches removed, some other patches applied… Also some conversion is done on the files to make Debian happy to build them. The result is uploaded to https://github.com/TBits/lbs-kolab-nightly
  2. Then I run different jobs on LBS, for each Operating System that I currently support: CentOS6, CentOS7, and Debian, to rebuild the selected packages with nightly tarballs.
    The result can be viewed here: https://lbs.solidcharity.com/project/tbits.net/kolab-nightly

These steps are executed each night, by a cronjob that initiates the builds on the LBS, by calling for example

tbitscredentials=tbits.net/secret
wget -O /dev/null https://lbs.solidcharity.com/triggerbuildproject/tbits.net/kolab-nightly/centos/6/amd64/$tbitscredentials

To test the nightly builds, you can install the nightly repository like this, additionally to the Kolab 3.4 and Kolab Development repo:

For CentOS6:

yum install yum-utils
yum-config-manager --add-repo https://download.solidcharity.com/repos/tbits.net/kolab-nightly/centos/6/lbs-tbits.net-kolab-nightly.repo

For CentOS7:

yum install yum-utils
yum-config-manager --add-repo https://download.solidcharity.com/repos/tbits.net/kolab-nightly/centos/7/lbs-tbits.net-kolab-nightly.repo

For Debian Wheezy:

apt-get install apt-transport-https
echo 'deb https://download.solidcharity.com/repos/tbits.net/kolab-nightly/debian/wheezy/ /' >> /etc/apt/sources.list
apt-get update

With the LightBuildServer I have full control over the builds, can just modify a config file to add a new target OS (if my LXC scripts support it), and can review the history of the package instructions on Github.

Of course, one disadvantage of LBS compared to OBS is: the LightBuildServer is not intended to directly support the work in a team. Team-work happens via Github (or your self hosted Gitlab), and every team member could install his own LightBuildServer.

A feature that LBS is still missing is that multiple build containers are just assigned with any job, without checking if one job should wait for another job to finish. A first step would be to distribute the jobs per Operating System to different build containers. Well, still lots of room for improvement…


Andreas Cordes's picture
Thu, 2015-03-05 00:24

Hello @all,

+Kolab.org Community released Version 3.4 and it was time to get the new updates up and running.

This time there is a new driver for "chwala" integrated for +Seafile.VN so I gave it a try and tried to mix it all together.

After lots pf solving dependencies and so on I managed to get the new version up and running with Seafile attached.

That's quite amazing, for this you have to configure Seafile through Apache and connect to the LDAP.

When you were able to manage all these things, you can easily save attachments in your cloud and syncing them to your smartphone. :-)

There are still some issues on my Cubietruck which I have to solve but my Idea is to rebuild all the stuff from scratch in a chroot environment and track down all steps to get it working.

Two Screenshots showing the new skin for +Roundcube and the integration of Seafile into Chwala.

That's the new Chameleon skin for Roundcube

The same folders in seafile and chwala except the password protected one.

So the next couple of days I'm trying to set up a new build environment for the ARM-HF packages for Kolab and a Step-By-Step description.

Greetz


Daniel Hoffend's picture
Wed, 2015-03-04 23:06

With the recently announced Kolab Groupware 3.4 Version a new default skin got introduced: "Chameleon". The new skin (which is still based on larry) provides a very clean layout which is very easy to customize.

Thanks to the LESS CSS Compilier you basically only need to define a few main colors (I changed 2), compile the style sheets and exchange the skinlogo. Most of the the other colors used within roundcube are based on your main-color (lighten by 30%, etc.).

If you don't like some of the colors in the icons, change them too by either changeing the psd files or just edit the png files (which sufficent) and within 10-120 minutes you got your personal kolab skin running.

The file structure

The roundcube theme is split into 2 componentes. The internal skin parts which consists of templates, configuration files, etc. and the asset part like CSS and images.

/usr/share/roundcubemail/skins/<name>/
/usr/share/roundcubemail/public_html/assets/skins/<name>/

Let's start

  1. Duplicate the existing chameleon skin
    cd /usr/share/roundcubemail
    cp -a skins/chameleon skins/mytheme
    ​cp -a public_html/assets/skins/chameleon public_html/assets/skins/mytheme

     

  2. Give your theme a name

    skins/mytheme/meta.json

    "name": "My Theme", 
  3. Set the 2 most important colors (main color + link color)

    public_html/assets/skins/colors.less

    @main-color:  #C1D72E;
    @link-color:  #ADAA38;

    ​ 

  4. To compile and minifiy your stylesheets you need the node-less compiler. You can either install nodejs and node-less (version >= 1.8.5) from your favorite repo or just use the version from the kolab repository.
    lessc public_html/assets/skins/styles.less > public_html/assets/skins/styles.css

     

  5. Replace the kolab logo (which is used in the upper left corner) with your logo.
    cp ~/mytheme_logo.png public_html/assets/skins/images/kolab_logo.png

     

  6. Change the skin in roundcube's configuration

    /etc/roundcubemail/config.inc.php

    $config['skin'] = 'mytheme';
    ​$config['dont_override'] = array('skin');
  7. Be Happy :-)

The optional part (icons)

For those who think that the color of the icons doesn't match their new color scheme: Here's a quick walk through on how to quickly change the colors using Gimp. And even me (usually using photoshop and yelling why my keyboard shortcuts don't work) managed to get things done fairly quickly (if you know the basics about layers and blending modes).

The majority of icons or images can be broken down into 2 categories:

  • Single-Color Icons with transparent background (example: buttons.png)
  • Single-Color Icons with white background (example: contactpic.png)

Example 1 (buttons.png)

  1. Open the Image
    File > Open > buttons.png

     

  2. For images that have indexed colors switch to RGP mode
    Image > Mode > RGP

     

  3. In the "Layers" Panel click on the [Lock alpha channel] to preserve the transparent background.

     
  4. Grab a pencil with a brushsize of around 10 pixel
     
  5. Pick your favorite color and start painting over the icons you like to change. Pay attention to not draw over the neighbour icons or parts you don't want to change.

    For icons like the thread open/close icons we use a different way (see below).
     

  6. Save the image
    File > Overwrite buttons.png

Example 2 (contactpic.png)

  1. Open the Image
    File > Open > buttons.png

     

  2. For images that have indexed colors switch to RGP mode
    Image > Mode > RGP

     

  3. (Optional) Make a selection on what you want to change (if you only want to change selected icons)
     
  4. Now change everything black/white
    Colors > Hue-Saturation > Saturation = -100
    Colors > Levels > Click [Auto]

    The contactpic should have have black borders and a white background.
     

  5. Create a new transparent layer and change the blending mode to "Lighten Only"
     
  6. Draw on the empty layer with your favorite color and everything which was black before will be filled with your color.

     
  7. Flatten the image again
    Layers > Merge Down

     

  8. Save the image
    ​File > Overwrite contactpic.png

vanmeeuwen's picture
Fri, 2015-02-27 16:22

What is the most original birthday present one can give their spouse? Well, how about a release of your favorite collaboration software suite!

I'm pleased to announce that Kolab 3.4 is available via our OBS repositories immediately!

Please see our Installation Guide for instructions on getting you some Kolab on your favorite platform, and if you already have Kolab running, check out the Upgrade Note from Kolab 3.3 to 3.4, kindly contributed for your benefit by Daniel Hoffend.

Kolab 3.4 comes with a new skin, called chameleon, that is a nice and clean-cut, modern skin for the web client -- courtesy of Kolab Systems, patron of the Kolab Groupware solution.

Two weeks ago, we have had our first release planning meeting on IRC, which has resulted in very fruitful feedback, contributions and most importantly a significant chunk of quality assurance from various participants in the community. A special thanks goes out to Daniel Hoffend and Timotheus Pokorra, who've spent plenty of their spare time on ensuring that Kolab 3.4 is the best it can be right out of the box. One slice of the pie on your right is theirs.

We're definitely going to continue to open up more processes, such as, for example, the Kolab 3.5 Roadmap.

The Kolab 3.4 release also marks the first release with an actual release party - though naturally many people are not able to attend. We're celebrating the Kolab 3.4 release is probably the best quality assured stable release Kolab.org has yet performed


vanmeeuwen's picture
Fri, 2015-02-27 16:22

What is the most original birthday present one can give their spouse? Well, how about a release of your favorite collaboration software suite!

I'm pleased to announce that Kolab 3.4 is available via our OBS repositories immediately!

Please see our Installation Guide for instructions on getting you some Kolab on your favorite platform, and if you already have Kolab running, check out the Upgrade Note from Kolab 3.3 to 3.4, kindly contributed for your benefit by Daniel Hoffend.

Kolab 3.4 comes with a new skin, called chameleon, that is a nice and clean-cut, modern skin for the web client -- courtesy of Kolab Systems, patron of the Kolab Groupware solution.

Two weeks ago, we have had our first release planning meeting on IRC, which has resulted in very fruitful feedback, contributions and most importantly a significant chunk of quality assurance from various participants in the community. A special thanks goes out to Daniel Hoffend and Timotheus Pokorra, who've spent plenty of their spare time on ensuring that Kolab 3.4 is the best it can be right out of the box. One slice of the pie on your right is theirs.

We're definitely going to continue to open up more processes, such as, for example, the Kolab 3.5 Roadmap.

The Kolab 3.4 release also marks the first release with an actual release party - though naturally many people are not able to attend. We're celebrating the Kolab 3.4 release is probably the best quality assured stable release Kolab.org has yet performed


Aaron Seigo's picture
Mon, 2015-02-23 12:08
Protocol Plugest - http://www.protocolsplugfest.com/europe/

The "world wide web" has been such an amazing success in large part because it was based on open protocols and formats that anyone can implement and use on a level playing field. This opened the way for interoperability on a grand and global scale, and is why http and HTML succeeded where many others failed previously.

Unfortunately, not all areas of computing are as blessed with open protocols and formats. Some are quite significant, too, with hundreds of millions of people using and relying on them every single day. Thankfully, some brave souls have stepped up to implement these proprietary protocols and formats using open technology, specifically as free software. The poster child for this is Samba and the ubiquitous file and print server protocols from Microsoft.

Such formats abound and are a key component in every day business (and personal) computer usage, and since the protocols are often not as open as we'd like it can be tricky to provide free, open and interoperable implementations of them. Again, just ask the Samba team. Yet bringing the option of freedom in technologies used by business and government around the world relies on these efforts.

The free software community is moving rapidly on all fronts of this landscape, and to help ensure that our efforts actually do work as expected and that we are sharing useful information with each other between projects, a fantastic conference has been set up: the Protocol Plugfest which will be held in Zaragoza, Spain on May 12-14. This is much like the ODF Plugfest which focuses on office file formats, but with a stronger focus on protocols found in products such as Active Directory, Exchange, Sharepoint, CIFS and LDAP.

The call for papers is now open and several speakers have already been confirmed. This include Kolab System's own Georg Greve who will be speaking on the topic of "Achieving All-Platform Interoperability", reflecting on Kolab's path towards supporting the wide world of clients, data formats and wire protocols such as ActiveSync.

He will also have some exciting announcements to make in this area during the presentation, so I hope everyone interested in free software collaboration suites will keep an eye on the event!

The other speakers include people from LibreOffice, Samba, Zentyal and ... Microsoft! Yes, I count no fewer than six speakers from Microsoft who are there to speak about the various protocols they've inflicted upon the world. This kind of engagement, while not perfect compared to having proper open standards, will certainly help push forward the state of support for all the protocols of the world in free software products.

I hope to see many of you there!


Aaron Seigo's picture
Mon, 2015-02-23 12:08

The "world wide web" has been such an amazing success in large part because it was based on open protocols and formats that anyone can implement and use on a level playing field. This opened the way for interoperability on a grand and global scale, and is why http and HTML succeeded where many others failed previously.

Unfortunately, not all areas of computing are as blessed with open protocols and formats. Some are quite significant, too, with hundreds of millions of people using and relying on them every single day. Thankfully, some brave souls have stepped up to implement these proprietary protocols and formats using open technology, specifically as free software. The poster child for this is Samba and the ubiquitous file and print server protocols from Microsoft.

Such formats abound and are a key component in every day business (and personal) computer usage, and since the protocols are often not as open as we'd like it can be tricky to provide free, open and interoperable implementations of them. Again, just ask the Samba team. Yet bringing the option of freedom in technologies used by business and government around the world relies on these efforts.

The free software community is moving rapidly on all fronts of this landscape, and to help ensure that our efforts actually do work as expected and that we are sharing useful information with each other between projects, a fantastic conference has been set up: the Protocol Plugfest which will be held in Zaragoza, Spain on May 12-14. This is much like the ODF Plugfest which focuses on office file formats, but with a stronger focus on protocols found in products such as Active Directory, Exchange, Sharepoint, CIFS and LDAP.

The call for papers is now open and several speakers have already been confirmed. This include Kolab System's own Georg Greve who will be speaking on the topic of "Achieving All-Platform Interoperability", reflecting on Kolab's path towards supporting the wide world of clients, data formats and wire protocols such as ActiveSync.

He will also have some exciting announcements to make in this area during the presentation, so I hope everyone interested in free software collaboration suites will keep an eye on the event!

The other speakers include people from LibreOffice, Samba, Zentyal and ... Microsoft! Yes, I count no fewer than six speakers from Microsoft who are there to speak about the various protocols they've inflicted upon the world. This kind of engagement, while not perfect compared to having proper open standards, will certainly help push forward the state of support for all the protocols of the world in free software products.

I hope to see many of you there!


vanmeeuwen's picture
Thu, 2015-02-19 21:24

Welcome back to the ongoing series of blog posts on benchmarking storage pods! Today is another beautiful Thursday and we have some extra information for you.

In my first blog post in this series, I had just barely gotten my hands on a Storage Pod -- and I was out to set a baseline for the storage performance. I mentioned that our intention had been to use SSDs for really fast access, and bulk SATA drives for the massive amounts of storage. I may also have mentioned that the controllers were seemingly unfairly balanced. More details of course are in part I.

First of all, I have to send a big thank you to our vendor, who almost immediately responded with some quick tips, clearly showing that the Storage Pod crowd of 45drives.com is paying attention, and wants you to get the loudest bang for your buck. Much appreciated, and nothing but kudos!

Now, admittedly, I don't know all that much about hardware -- plenty of people are much better designated experts. I do appreciate the effects of (a bit of, pun intended) electrical interference on a 4-layer mainboard PCB under high frequency throughput though, as well as the accuracy of x86 CPU architectures vs. let's say, s390(x). That is to say, I'm completely out of my comfort zone when I ask for advice in a computer shop, because I have to dumb it down (a lot), not even expecting an answer that helps me.

That said, loud the bang will be, because 45 readily available SATA drives of 4 TB each give you a raw storage volume of 180 TB, regardless of what you use it for. At the cost of the chassis and the cost of the individual drives, all put together, you will elect to put both performance and data redundancy in the hands of replicating and balancing entire pods rather than using multiple 3U storage arrays from the usual vendors.

However, naturally you will still want a single storage pod to perform well. You will also want each Storage Pod to be some level of redundant in and by itself, to stop you from having to go in to the datacenter with a few drives too often.

I talked about the initial synchronization of the two RAID-5 arrays I created taking a little while at a sustained throughput of about ~75 - ~90 MBps. I'm fairly certain you do already appreciate, but I have to mention anyway, that this is an implied throughput per disk, rather than for the array as a whole. It is therefore not a baseline, but it does give us information; If each disk can do ~85 MBps, then 19 such disks should (theoretically) be able to accumatively achieve a total throughput of just under 1.6 GBps, right? Right, but also wrong. The I/O pattern of a RAID resync is completely different from the I/O pattern of regular use. Luckily though, each SATA backplane is capable of sustaining 5 GBps, so we're also not maxing out the backplane.

This gives us two interesting paths to explore;

  1. Does "md126" (with 20 partipant, active drives) outperform "md128" (with 17 participant, active drives)?
  2. Does substituting the SATA drives for SSDs mean anything (on Highpoint Rocket 370 controllers)?

In this blog post, I will likely only get around to answering question #1 -- sorry to dissappoint.

We first seek a baseline comparison on the current situation -- remember 20 disks on one array, 17 on the other, RAID 5, each has two hot spares (not included in the count).

Q: What shall be your benchmark?

The answer is simple: Bonnie++.

Q: How shall you run it?

This is a simple one too, but the answer is a little longer. One opts for a bunch of defaults without tweaking at first. This shall be your baseline.

In our particular scenario, running Red Hat Enterprise Linux 7 on these Storage Pods, the default filesystem is XFS. I come from an era in which this was not the default, however, and we want to compare "now" vs. "then" -- meaning we'll start out with EXT4.

The choice of filesystem, I think, has logical implications on the performance, even in a Bonnie++ scenario (with a default chunk size of just under 16GB). I'm taking in to account things like journalling, and there may also be points where one filesystem driver is inclined to hook in to kernel-level storage interfaces slightly different from another filesystem driver.

Hence we have set the scenario: We'll want a genuine comparison of different filesystems for Bonnie++ to benchmark using the two different RAID 5 arrays.

Q: Proper scientists formulate hypothesis before they start running in circles, so what's yours?

The hypothesis is that md126 (20 disks) will outperform md128 (17 disks), and XFS will outperform EXT4 but not by as much as the md126 array will outperform md128.

I feel inclined to acknowledge the following assumptions with regards to this hypothesis, since you started to go all scientist on me;

  1. XFS surely isn't equal to or more than 118% more efficient than EXT4 at the pattern we're about to throw at it, and
  2. whatever pattern you throw at it when benchmarking very rarely represents the actual patterns thrown at it in a production environment.

First Things First

  1. Reading the output from Bonnie++, and secondly interpreting what it means, is a skill in and by itself -- I know because I've learned the hard way. More on this later.
  2. We're continuously running Munin on the "localhost" -- with its default 5 minute interval, and a non-CGI HTML and graph strategy. This means as much as that every 5 minutes, Munin is eating CPU and I/O (though not on the same disks), and therefore we have to repeat runs of Bonnie++ 10 times, in order to get results that better represent actual as opposed to ficticious througput.
  3. Bonnie++ is run with only a -d /path/to/mountpoint command-line, with /path/to/mountpoint being a given Logical Volume we use for the specific test. That is to say that each RAID array has been made a Physical Volume, each PV has been added to a Volume Group unique to that PV, and test has a Logical Volume (of a set, constant size) in the appropriate VG.

Recognizing and acknowledging the I/O pattern likely to be thrown at the Storage Pod helps in determining the type of benchmark you will want to run. After all, a virtualization guest's node disk image -- despite the contents inside that disk -- does establish a pattern slightly different from an IMAP spool or even a number of package build roots / docker images.

To obtain this information, let's see what a guest's disk image tends to do; internally to the guest, a disk is partitioned and filesystems be mounted. Data is read from and written to places in this filesystem, but underneath it all may be something like a qcow2 thin-provisioned image file. This basically means that the I/O pattern be random, yet -- for most tech running inside your vm -- of a block-stream level.

An RPM installation however uses cpio, and a build root tends to install many RPMs -- much like a yum update does, or the yum installs for a Docker container. This particular I/O pattern tends to be supremely expensive on the disk / array controller -- which is why most build roots are in (the in-memory) tmpfs.

Long story short, whether a Storage Pods' individual controllers need to maintain mirrors of entire RAID arrays, whether RAID arrays are themselves (a level of) robust, what your expectations are of the individual disks and how many Storage Pods you planned on including in your environment, as well as the particular technology you're thinking of using to communicate with your Storage Pods -- software iSCSI (tgtd)? NFS? GlusterFS (Over NFS? Replicated? Distributed? Both)? -- all of it matters. This is subject to requirement engineering, backed by large amounts of expertise, experience, information, skill and proper judgement.

How About that 20-Disk vs. 17-Disk Array?

Well, luckily one part of the hypothesis turns out to be true: The "md126" array (20 disks) outperforms the "md128" array (17 disks). Or does it?

However, it only does so using a particular I/O pattern that Bonnie++ tests:

Bonnie++ per-character
  K putc() per second CPU Usage
md126 894.4 97.2%
md128 898.1 93.9%

 

Say what?

When you run Bonnie++, it will default to doing 16 GB worth of putc() calls, or otherwise individual bits and bytes, or 16 billion in total or so -- give or take a dozen. What Bonnie++ has reported here is an average of 10 individual runs, where md126 averages out putting through 894.4K putc() calls a second, and md128 more -- 898.1K per second to be precise.

Let us back up, and see if these numbers somewhat represent what we find in real life;

Some ~900k calls per second, with ~16 billion or so in total, would mean that the test takes ~5 minutes to complete, right? Check, it does take that approximately that long.

So md128 is the clear winner, with more putc() calls per second, and lower CPU usage. Huh?

Bonnie++ "efficient block write"
  K per second CPU Usage
md126 678483.9 46.1%
md128 582650.7 38.8%

 

Here, what Bonnie++ calls "efficient block writes" achieve a significantly higher rate for the md126 array than for the md128 array. However, when you calculate back the amount of CPU usage involved, md128 again outperforms md126 in efficiency (at 15k per %1 vs. 14k per 1%).

What are "efficient block writes"? I'm more than happy to admit I cannot say for sure. Deriving from the context I imagine efficient block writes have to do with the I/O scheduler in the kernel and subsequent subsystems. This tells me I will want to perform the same tests using different kernel I/O schedulers for the set of individual block devices in each array. Noted.

Note however these are ext4 based tests. We have the XFS tests to go still:

Bonnie++ per-character
  K putc() per second CPU Usage
md126 1987.3 96.4%
md128 1794.0 98.0%

 

This brings us to the second part of the hypothesis -- XFS outperforming EXT4, but by no more than 117%. Well, as far as this benchmark goes, it is busted. Let's look at the "efficient block write" stats:

Bonnie++ "efficient block writes"
  K per second CPU Usage
md126 699922.7 31.0%
md128 588464.0 29.0%

 

Not that much gain in throughput, although apparently XFS is slightly faster than EXT4, but what a decrease in CPU usage -- XFS seems much more efficient at around and about a rate of some 30%!

Finally ...

I hope you enjoy reading about Storage Pods and the novice level approach I'm taking to try and comprehensively benchmark them, using exclusively Free and Open Source Software. That said, your feedback and ideas on things to also try is much appreciated! Please do not hesitate and call out vanmeeuwen@kolabsys.com, or hit me on Twitter (@kanarip).

While it is not part of this series of blog posts about my attempts to get the loudest bang for my buck, we do appreciate collaboration. I've consistently collected the raw statistics and reports, and I'm fully aware that the aforementioned numbers to not mean anything without them. Please do not hesitate and contact Kolab Systems if you are interested in reviewing the raw data yourself.

I also appreciate your feedback on how you think multiple Storage Pods would fit in your infrastructure. Are you considering NFS servers for a virtualization environment, perhaps replicated through DRBD, or take a Ceph/GlusterFS approach? How do you think the concept of shared storage would fit in with future technologies such as Docker/Atomic? Hit me at vanmeeuwen@kolabsys.com or Twitter.

 


vanmeeuwen's picture
Thu, 2015-02-12 03:41

I have recently obtained access to a so-called Storage Pod -- an open hardware design for stuffing up to 45 SATA III drives in to a chassis no higher than 4U -- a.k.a. "Storinator".

How does one benchmark such goodness? Well, most importantly you first need to set your baseline. This I will do in this Part I of what hopefully becomes a series of posts on the subject.

The pod comes with 2 Highpoint Rocket 750 controller cards connected to 3 SATA back-planes, each of which capable of transferring up to 5 GBps. This seems unfairly balanced, since the slots these controller cards are stuffed in each have their own maximum transfer rate. Let's keep this in mind while we dive in deeper.

It's stuffed with 39 times 4 TB Western Digital "Red" drives, striking a balance between overall power consumption, capacity and Mean Time Between Failure (MTBF). This leaves 6 slots open, in which are 1 TB Samsung SSDs.

Your first mistake will be to order 39 drives (or actually, 78+) from the same vendor in one fell swoop. Don't do this, unless you have the necessary appreciation of what is likely to happen to your other drives once one drive fails -- the likelihood of all other drives with a too much similar serial number may suffer from the same manufacturing error. Suggested solution: Be close to or on-site, and have plenty of spare drives. This storage solution is not intended to make you care about each individual drive of CHF 200,- as much as you would when it is your only drive.

Second order of business is to appreciate the mainboard, CPU and available RAM do not necessarily comprise your ideal server system -- mine has only 8 GB of RAM, and an Intel Core i3 quad-core CPU. Nothing to write home about, yet presumably sufficient for some decent performance -- we'll find out.

So how about the baseline? Well, let's take this on a controller-by-controller basis. Two primary entry-points are available when you fire up the system:

  • /dev/disk/by-id/

    This directory contains symbolic links to block devices /dev/sd*, where the link is named such that it represents a humanly readible name for the device, namely "type", "make", "model" and "serial". This is your first assistence for inventory.
     

  • /dev/disk/by-path/

    This directory contains the "path" you would think the disk is at from the CPU's perspective. You'll find something like "pci-0000:01:" and "pci-0000:02:" types of names in here. You are not mistaken in thinking these numbers may have something to do with the controller number -- as indeed there are more drives connected to the first controller than there are to the second controller, remember the unfair balancing I mentioned earlier.
     

So What Is Your Baseline?

A not too unfair baseline is to have as many disks as possible, all on one controller, all of one type, be in a type of RAID array where even continuous blocks are distributed over disks. So not a JBOD type of array, because there continous blocks end up on the same disk unless you cross a disk's boundary. I have chosen RAID 5 for the job, because that is likely a large part of what we'll be using when we start using these systems for real.

Using information from the disk devices, write down the following numbers:

# cat /sys/block/sdd/queue/optimal_io_size
0
# cat /sys/block/sdd/queue/minimum_io_size
512
# cat /sys/block/sdd/alignment_offset
0
# cat /sys/block/sdd/queue/physical_block_size
512

If your alignment offset is non-zero, use it as the offset.

If your optimal I/O size is zero, use the literal string "1MiB" as the start for your "Linux RAID auto-detect" partition.

If your optimal I/O size is non-zero, some calculations are involved to align the partition properly.

For the end sector of the partitions, I favored the "flat on one's face" approach:

# parted /dev/sdd mklabel gpt
# parted -a optimal /dev/sdd -- mkpart primary 1MiB -1s
Warning: You requested a partition from 1000kB to 4001GB (sectors 1953..7814037167).
The closest location we can manage is 1000kB to 4001GB (sectors 1953..7814037134).
Is this still acceptable to you?
Yes/No? No

So rather than "-1s" we are to use "-34s".

Since all my drives are exactly the same (up to the last few characters of the serial number), I can start cracking -- note that all my disks are completely empty and have no pre-existing medical conditions such as partitions and whatnot:

# cd /dev/disk/by-id/
# for disk in $(readlink ata-WDC* | grep -vE -- "-part[0-9]+$"); do
    parted -a optimal $disk -- mklabel gpt
    parted -a optimal $disk -- mkpart primary 1MiB -34s
    parted -a optimal $disk -- set 1 raid on
done

Before you actually start, it might be worthwhile to get some statistics gathered and graphed:

# yum -y install munin
# systemctl start munin-node.service

Now, create a RAID 5 array from the disks on controller 1 (21 SATA drives):

# cd /dev/disk/by-path/
# unset disks
# declare -a disks
# for dbp in $(ls -1 pci-0000:01* | grep -E -- "-part[0-9]+$"); do
    disk=$(basename $(readlink $dbp))
    dbi=$(ls -l /dev/disk/by-id/ | grep $disk | grep -vE "(md|wwn)" | awk '{print $9}')
    if [ ! -z "$(echo $dbi | grep ata-WDC)" ]; then
        disks[${#disks[@]}]="/dev/$disk"
    fi
done
# mdadmin --create /dev/md126 --level=5 --raid-devices=${#disks[@]} ${disks[@]}

The initial process of syncing can easily take up to 15 hours. You could assume the array is clean of course, but that is not putting your drives through their paces -- remember you will want to find drive errors early.

I'm at a sustained throughput of ~75 - ~90 MBps for the resync, without any tweaking whatsoever. This is by no means the baseline, but we're going to have to wait for the resync to complete to start establishing the actual.

Stay "tuned" for part II -- pun intended.


vanmeeuwen's picture
Thu, 2015-02-12 02:27

So far, Kolab.org releases have largely been planned by me, Jeroen van Meeuwen. That is to say, I may have previously pulled the trigger at your proverbial 04:00 military time and subsequently pulled power-nap assisted binges of between 48 and 72 hours.

This can clearly be orchestrated better, and so we are further opening up this process. This time, we call on you in advance to participate in making sure the Kolab 3.4 release is as smooth as silk.

On Friday, February 13th, at 15:00 CET, we shall meet in the IRC channel #kolab on the FreeNode network. We have a little bit of an agenda to discuss with regards to this release, and in the future, we'll have more to discuss in terms of roadmap and development.

For more details, read the announcement to the development mailing list.


Aaron Seigo's picture
Wed, 2015-02-11 16:03

An interesting problem came up during a recent bit of Kolab server-side work: tracking the lifetime of objects in the IMAP store. How hard can it be, right? In my experience, it is the deep, dark holes which start with that question. ;)

In our case, the IMAP server keeps objects (e.g. emails, events, whatever) in directories that appear as folders to the client. It keeps each object in a separate file, so each object gets a unique filename in that folder. That file name plus the folder path is unique at that point in time, something enforced by the filesystem. Let's call this the "object id" or OID for short. Of course, when you move an object from one folder to another that id changes; in fact, the file name may also change and so the OID is not constant over time. We can track those changes, however.
Inside the file is the message content of the object. There is an ID there as well, and generally those are unique to a given user. Let's call this the "message id", or MID for short. However, it is possible to share messages with others, such as when sending a calendar invitation around between users. This can lead to situations quite easily where multiple users have a message with the same MID. They of course have different OIDs .. right?
Well, yes, but only at a given point in time. It is possible that over time, with "clever" moving of objects between folders, particularly once we add in the concept of shared folders (which is the same as saying shared calendars, notes, etc.), it is possible that at different points in time there can be objects with the same OID and the same MID but which are actually physically different things.
Now if we want to get the history for a given object in the IMAP storage and list all the things relevant to it in an entirely deterministic way, how can we guarantee proper results? In a simple logging model one might want to simply key change events with the OID, MID or some combination, but since those are not guaranteed to be unique over time in any combination this leaves a small challenge on the table to make it deterministic.
We didn't want to change the data in the messages or the way storage worked. Changing less means fewer opportunities for problems (read: bugs), and besides it's always fun to work out these kinds of problems. After an hour or so of pen and paper doodling, we were able to devise a deterministic algorithm for tracking object history without changing anything in the storage system. 
The key to the puzzle? Message move events are generated when objects move around on the server, allowing a UID+MID change timeline to be logged in parallel which can be compared with the general modification history and, with a bit of effort, reliably differentiate objects even if they had the same OID and MID at different points in time. Given that history queries always have a known starting state in time (now) and that OID+MID is guaranteed unique at any given moment in time (e.g. now) it is possible to track history working backwards from now (or forwards towards it if one must). There is very little overhead to do this, so we elected for this route.
So even though OID + MID uniqueness is not guaranteed over time, it is possible to accurately track object history through time without resorting to injecting some sort of true UUID data in each and every message on top of all the other data that is in there ... not to mention ensuring that it continues to work as expected with all client software and other parts of the Kolab stack.

Aaron Seigo's picture
Wed, 2015-02-11 16:03

An interesting problem came up during a recent bit of Kolab server-side work: tracking the lifetime of objects in the IMAP store. How hard can it be, right? In my experience, it is the deep, dark holes which start with that question. ;)In our case, the IMAP server keeps objects (e.g. emails, events, whatever) in directories that appear as folders to the client. It keeps each object in a separate file, so each object gets a unique filename in that folder. That file name plus the folder path is unique at that point in time, something enforced by the filesystem. Let's call this the "object id" or OID for short. Of course, when you move an object from one folder to another that id changes; in fact, the file name may also change and so the OID is not constant over time. We can track those changes, however.Inside the file is the message content of the object. There is an ID there as well, and generally those are unique to a given user. Let's call this the "message id", or MID for short. However, it is possible to share messages with others, such as when sending a calendar invitation around between users. This can lead to situations quite easily where multiple users have a message with the same MID. They of course have different OIDs .. right?Well, yes, but only at a given point in time. It is possible that over time, with "clever" moving of objects between folders, particularly once we add in the concept of shared folders (which is the same as saying shared calendars, notes, etc.), it is possible that at different points in time there can be objects with the same OID and the same MID but which are actually physically different things.Now if we want to get the history for a given object in the IMAP storage and list all the things relevant to it in an entirely deterministic way, how can we guarantee proper results? In a simple logging model one might want to simply key change events with the OID, MID or some combination, but since those are not guaranteed to be unique over time in any combination this leaves a small challenge on the table to make it deterministic.We didn't want to change the data in the messages or the way storage worked. Changing less means fewer opportunities for problems (read: bugs), and besides it's always fun to work out these kinds of problems. After an hour or so of pen and paper doodling, we were able to devise a deterministic algorithm for tracking object history without changing anything in the storage system. The key to the puzzle? Message move events are generated when objects move around on the server, allowing a UID+MID change timeline to be logged in parallel which can be compared with the general modification history and, with a bit of effort, reliably differentiate objects even if they had the same OID and MID at different points in time. Given that history queries always have a known starting state in time (now) and that OID+MID is guaranteed unique at any given moment in time (e.g. now) it is possible to track history working backwards from now (or forwards towards it if one must). There is very little overhead to do this, so we elected for this route.So even though OID + MID uniqueness is not guaranteed over time, it is possible to accurately track object history through time without resorting to injecting some sort of true UUID data in each and every message on top of all the other data that is in there ... not to mention ensuring that it continues to work as expected with all client software and other parts of the Kolab stack.


roundcube's picture
Sun, 2015-02-08 15:12

We’re proud to announce the arrival of the next major version 1.1.0 of
Roundcube webmail which is now available for download. With this
milestone we introduce new features since version 1.0 as well as
some clean-up with the 3rd party libraries:

  • Allow searching across multiple folders
  • Improved support for screen readers and assistive technology using
    WCAG 2.0 and WAI ARIA standards
  • Update to TinyMCE 4.1 to support images in HTML signatures (copy & paste)
  • Added namespace filter and folder searching in folder manager
  • New config option to disable UI elements/actions
  • Stronger password encryption using OpenSSL
  • Support for the IMAP SPECIAL-USE extension
  • Support for Oracle as database backend
  • Manage 3rd party libs with Composer

In addition to that, we added some new features to improve protection
against possible but yet unknown CSRF attacks - thanks to the help of
Kolab Systems who supplied the concept
and development resources for this.

Although the new security features are yet experimental and disabled by default,
our wiki describes how to enable the Secure URLs
and give it a try.

And of course, this new version also includes all patches for reported
CSRF and XSS vulnerabilities previously released in the 1.0.x series.

IMPORTANT: with the 1.1.x series, we drop support for PHP < 5.3.7
and Internet Explorer < 9. IE7/IE8 support can be restored by
enabling the ‘legacy_browser’ plugin.

See the complete Changelog at trac.roundcube.net/wiki/Changelog
and download the new packages from roundcube.net/download.


mollekopf's picture
Sun, 2015-02-08 01:36

Ever since we introduced our ideas the next version of akonadi, we’ve been working on a proof of concept implementation, but we haven’t talked a lot about it. I’d therefore like to give a short progress report.

By choosing decentralized storage and a key-value store as the underlying technology, we first need to prove that this approach can deliver the desired performance with all pieces of the infrastructure in place. I think we have mostly reached that milestone by now. The new architecture is very flexible and looks promising so far. We managed IMO quite well to keep the levels of abstraction to a necessary minimum, which results in a system that is easily adjusted as new problems need to be solved and feels very controllable from a developer perspective.

We’ve started off with implementing the full stack for a single resource and a single domain type. For this we developed a simple dummy-resource that currently has an in-memory hash map as backend, and can only store events. This is a sufficient first step, as turning that into the full solution is a matter of adding further flatbuffer schemas for other types and defining the relevant indexes necessary to query what we want to query. By only working on a single type we can first carve out the necessary interfaces and make sure that we make the effort required to add new types minimal and thus maximize code reuse.

The design we’re pursuing, as presented during the pim sprint, consists of:

  • A set of resource processes
  • A store per resource, maintained by the individual resources (there is no central store)
  • A set of indexes maintained by the individual resources
  • A clientapi that knows how to access the store and how to talk to the resources through a plugin provided by the resource implementation.

By now we can write to the dummyresource through the client api, the resource internally queues the new entity, updates it’s indexes and writes the entity to storage. On the reading part we can execute simple queries against the indexes and retrieve the found entities. The synchronizer process can meanwhile generate also new entities, so client and synchronizer can write concurrently to the store. We therefore can do the full write/read roundtrip meaning we have most fundamental requirements covered. Missing are other operations than creating new entities (removal and modifications), and the writeback to the source by the synchronizer. But that’s just a matter of completing the implementation (we have the design).

To the numbers: Writing from the client is currently implemented in a very inefficient way and it’s trivial to drastically improve this, but in my latest test I could already write ~240 (small) entities per second. Reading works around 40k entities per second (in a single query) including the lookup on the secondary index. The upper limit of what the storage itself can achieve (on my laptop) is at 30k entities per second to write, and 250k entities per second to read, so there is room for improvement =)

Given that design and performance look promising so far, the next milestone will be to refactor the codebase sufficiently to ensure new resources can be added with sufficient ease, and making sure all the necessary facilities (such as a proper logging system), or at least stubs thereof, are available.

I’m writing this on a plane to Singapore which we’re using as gateway to Indonesia to chase after waves and volcanoes for the next few weeks, but after that I’m  looking forward to go full steam ahead with what we started here. I think it’s going to be something cool =)


Timotheus Pokorra's picture
Thu, 2015-01-29 09:22

TBits.net is glad to announce that nightly tests now run every night against the nightly development packages of Kolab, for both CentOS 6 and Debian Wheezy.

At the moment, the nightly tests also run for the Kolab 3.3 Updates for CentOS 6.

Through these tests, we can spot problems and obvious bugs in the code early, when they are still easy to fix. This is TBits.net’s contribution to making Kolab more stable and reliable.

As soon as Kolab 3.4 is out (expected for the middle of February 2015), we will enable nightly tests for CentOS 6 and Debian Wheezy for Kolab 3.4.

You can see the tests and the results here: https://lbs.solidcharity.com/package/tbits.net/kolab-test/kolab-test

I use the TBits KolabScripts to install the server unattended.

The TBits KolabScripts also contain some Selenium tests written in Python. These tests check if the user can change his own password, and tests creating new domains and new users, sending emails, catchall addresses, etc.

Also the TBits scripts for ISPs are tested: These scripts add domain admins functionality to give customers the option to manage their own domains, limited by a domain quota and maximum accounts number.

We use the LightBuildServer for running the tests. This software is written in Python, by Timotheus Pokorra. It is very light-weight, does not come with much code, and is easy to extend and configure. It uses LXC to create virtual machines for each test or build run. You can see it live in action for Kolab

We now also initiate the nightly development packages with the LightBuildServer: https://lbs.solidcharity.com/package/tbits.net/kolab-nightly-sync/updatecode. This job is run every night, fetches the source from git.kolab.org, modifies the package version number, and uploads the new package instructions to obs.kolabsys.com so that they can be built on the OBS server sponsored by Kolab Systems.

If you want to test the nightly packages yourself, please read the chapter in the Kolab Developer Guide: http://docs.kolab.org/developer-guide/nightly-builds/install.html


Andreas Cordes's picture
Mon, 2015-01-26 20:35
Hi,

due to a new job since the beginning of the year and moving to a new country I was a bit stressed. In addition to this, my provider changed from IPv4 to IPv6 at a DSLite line.

So my port forwarding is no longer working and I had to organize all the stuff to get access to my mails again :-). Well, that's a disadvantage of my solution but I hope this will be solved somehow.

So my script to install Kolab on my truck:

export RELEASE=3.3
export RELEASEDIR=kolab_3.3_src
mkdir -p /mnt/kolab/$RELEASEDIR
mkdir -p /mnt/kolab/kolab_${RELEASE}_deb
echo "deb-src http://obs.kolabsys.com:82/Kolab:/$RELEASE/Debian_7.0/ ./" > /etc/apt/sources.list.d/kolab.list
echo "deb-src http://obs.kolabsys.com:82/Kolab:/$RELEASE:/Updates/Debian_7.0/ ./" >> /etc/apt/sources.list.d/kolab.list
#echo "deb http://kolab-deb.zion-control.org /" >> /etc/apt/sources.list.d/kolab.list
echo "Package: *" >  /etc/apt/preferences.d/kolab
echo "Pin: origin obs.kolabsys.com" >>  /etc/apt/preferences.d/kolab
echo "Pin-Priority: 501" >>  /etc/apt/preferences.d/kolab
wget -qO - http://obs.kolabsys.com:82/Kolab:/$RELEASE/Debian_7.0/Release.key | apt-key add -
wget -qO - http://obs.kolabsys.com:82/Kolab:/$RELEASE:/Updates/Debian_7.0/Release.key | apt-key add -
aptitude update
cd /mnt/kolab/$RELEASEDIR
echo Get debian sources

wget http://obs.kolabsys.com:82/Kolab:/$RELEASE/Debian_7.0/Packages -O packages.txt
wget http://obs.kolabsys.com:82/Kolab:/$RELEASE:/Updates/Debian_7.0/Packages -O packages_updates.txt
apt-get -ym source `grep Package packages.txt | awk '{print $2}' | grep -v "kolab-ucs" | sort -u`
apt-get -ym source `grep Package packages_updates.txt | awk '{print $2}' | grep -v "kolab-ucs" | sort -u`
apt-get -y build-dep `grep Package packages.txt | awk '{print $2}' | grep -v kolab-ucs | sort -u`
dpkg -i 389-ds-base-libs_1.2.11.29-0_armhf.deb
dpkg -i 389-ds-base-dev_1.2.11.29-0_armhf.deb
cd python-icalendar-3.4/
debuild -us -uc -b
cd ..
apt-get install python-dateutil python-tz
dpkg -i python-icalendar_3.4-1_all.deb
cd libcalendaring_4.9.0
debuild -us -uc -b
cd ..
dpkg -i libcalendaring_4.9.0-3_armhf.deb
dpkg -i libcalendaring-dev_4.9.0-3_armhf.deb
apt-get install libboost-program-options-dev
cd libkolabxml-1.1~dev20140624/
debuild -us -uc -b
cd ..
dpkg -i libkolabxml1_1.1~dev20140624-0~kolab1_armhf.deb
dpkg -i libkolabxml-dev_1.1~dev20140624-0~kolab1_armhf.deb
cd libkolab-0.6~dev20140624/
debuild -us -uc -b
cd ..
dpkg -i libkolab0_0.6~dev20140624-0~kolab1_armhf.deb
dpkg -i libkolab-dev_0.6~dev20140624-0~kolab1_armhf.deb
apt-get -y build-dep `grep Package packages.txt | awk '{print $2}' | grep -v kolab-ucs | sort -u`
apt-get -y build-dep `grep Package packages_updates.txt | awk '{print $2}' | grep -v kolab-ucs | sort -u`
ls -ld * | grep '^d' | awk '{print $9}' | while read verz
do
        cd $verz
        debuild -us -uc -b
        cd ..
done

cp *deb ../kolab_${RELEASE}_deb/
cd ../kolab_${RELEASE}_deb
dpkg-scanpackages -m .  | gzip -c9 > Packages.gz

After a while it was installed and I had to move all mails to my truck. I decided to reorganize all the mails according to the recipient in a separate folder (because of my C@tch@ll). I used "imapfilter" for that and for this I wrote a little LUA script:

function processMailbox(raspberryPi,cubietruck, mb,subfolders)
        -- Status
        total, recent,unseen,nextid = raspberryPi[mb]:check_status()
        -- Alle Nachrichten auswaehlen
        results = raspberryPi[mb]:select_all()
        number = 1
        for _, message in ipairs(results) do
                print ("")
                mailbox, uid = table.unpack(message)
                header = mailbox[uid]:fetch_header()
                recipient = string.gsub(header, ".*for <([A-Za-z0-9-]+@example.com)>.*","%1")
                if (recipient:find(":")) then
                        headerTo = mailbox[uid]:fetch_field('To')
                        if (headerTo == nil) then
                                recipient = "invalid_to@example.com"
                        else
                                headerTo = headerTo:gsub("\r","")
                                headerTo = headerTo:gsub("\n","")
                                headerTo = headerTo:gsub(" ","")
                                headerTo = headerTo:gsub("To: ","")
                                if (headerTo == "") then
                                        recipient = "invalid_to@example.com"
                                else
                                        recipient = headerTo:gsub("To: ","")
                                end
                        end
                end
                if (recipient:find("<")) then
                        recipient = recipient:gsub(".*<(.*)>","%1")
                end
                toFolder = string.gsub(recipient,"(.-)(@.*)","%1")
                messageId = mailbox[uid]:fetch_field("Message-id")
                messageId = string.gsub(messageId,".-<(.-)>","%1")
                subject   = mailbox[uid]:fetch_field('Subject')
                subject   = subject:gsub("\r"," ")
                subject   = subject:gsub("\n"," ")
                print('Processing : <' .. mb .. '>')
                print('Subject    : <' .. subject .. '>')
                print('Recipient  : <' .. recipient .. '>')
                print('Folder     : <' .. toFolder .. '>')
                print('ID         : <' .. messageId .. '>')
                prozent = number / total * 100
                print('Number     : <' .. number .. '/' .. total .. '> in ' .. mb .. ' (' .. prozent .. ' %)')
                cubietruck:create_mailbox('Archive/' .. toFolder)
                toMove = raspberryPi[mb]:contain_field("Message-id",messageId)
                for _, messageM in ipairs(toMove) do
                        tmMB,tmUID = table.unpack(message)
                end
                toMove:copy_messages(cubietruck['Archive/' .. toFolder])
                number = number + 1
        end
        -- process all subfolder
        if (subfolders) then
                rPi_Mailboxes, rPi_Folders = raspberryPi:list_all(mb)
                for _, mbSub in ipairs(rPi_Mailboxes) do
                        processMailbox(raspberryPi,cubietruck,mbSub,true)
                end
        end
end
---------------
--  Options  --
---------------
options.timeout = 120
options.subscribe = true
options.create = true
----------------
--  Accounts  --
----------------
-- Quelle ist Pi und Ziel ist Truck
raspberryPi = IMAP {
    server = 'raspberrypi.example.com',
    username = 'username@example.com',
    password = ':-)',
}
cubietruck = IMAP {
    server = 'cubietruck.example.com',
    username = 'username@example.com',
    password = ';-)',
}
processMailbox(raspberryPi,cubietruck,'Subfolder',false)

Don't hesitate to ask any questions
Greetz

roundcube's picture
Sat, 2015-01-24 17:52

We just published a security update to the stable version 1.0.
Beside a recently reported Cross-Site-Scripting vulnerability,
this release also contains some bug fixes and improvements we
found important for the long term support branch of Roundcube.

It’s considered stable and we recommend to update all productive installations
of Roundcube with this version. Download it from roundcube.net/download,
see the full changelog here.

Please do backup before updating!