Planet

Subscribe to the RSS feed of this planet! RSS
Aaron Seigo's picture
Wed, 2015-05-27 15:28

transactional b-trees and what-not

Over the last few months I've been reading more than the usual number of papers on a selection of software development topics that are of recent interest to me. The topics have been fairly far flung as there are a few projects I have been poking at in my free time.

By way of example, I took a couple weeks reading about transitory trust algorithms that are resistant to manipulation, which is a pretty interesting problem with some rather elegant (partial) solutions which are actually implementable at the individual agent level, though computationally impractical if you wish to simulate a whole network which thankfully was not what I was interested in. (So reasonable for implementing real-world systems with, though not simulations or finding definitive solutions to specific problems.)

This past week I've been reading up on a variety of B-tree algorithms. These have been around since the early 1970s and are extremely common in all sorts of software, so one might expect that after 40+ years of continuous use of such a simple concept that there'd be very little to talk about, but it's quite a vast territory. In fact, each year for the last two decades Donald Knuth has held a public lecture around Christmas-time about trees. (Yes, they are Christmas Tree Lectures. ;) Some of the papers I've been reading were published in just the last few years, with quite a bit of interesting research having gone on in this area over the last decade.

The motivation for reading up on the topic is I've been looking for a tree that is well suited to storing the sorts of indexes that Akonadi Next is calling for. They need to be representable in a form that multiple processes can access simultaneously without problems with multiple readers and (at least) one writer; they also need to be able to support transactions, and in particular read transactions so that once a query is started the data being queried will remain consistent at least until the query is complete even if an update is happening concurrently. Preferably without blocking, or at least as little blocking as possible. Bonus points for being able to roll-back transactions and keeping representations of multiple historic versions of the data in certain cases.

In the few dozen papers I downloaded onto the tablet for evening reading, I came across Transactions on the Multiversion B+-Tree which looks like it should do the trick nicely and is also (thankfully) nice and elegant. Worth a read if you're into such things.

As those who have been following Akonadi Next development know, we are using LMDB for storage and it does a very nice job of that but, unfortunately, does not provide "secondary" indexes on data which Akonadi Next needs. Of course one can "fake" this by inserting the values to be indexed (say, the dates associated with an email or calendar event) as keys with the value being they key of the actual entry, but this is not particularly beautiful for various reasons, including:

  • this requires manually cleaning up all indexes rather than having a way to efficiently note that a given indexed key/value pair has been removed and have the indexes cleaned up for you
  • some data sets have a rather low cardinality which would be better represented with approaches such as bitmap indexes that point to buckets (themselves perhaps trees) of matching values
  • being able to index multiple boolean flags simultaneously (and efficiently) is desirable for our use cases (think: "unread mails with attachments")
  • date range queries of the sort common in calendars ("show this month", "show this week", e.g.) could also benefit from specialized indexes

I could go on. It's true that these are the sorts of features that your typical SQL database server provides "for free", but in our case it ends up being anything but "free" due to overhead and constraints on design due to schema enforcement. So I have been looking at what we might be able to use to augment LMDB with the desired features, and so the hunt for a nice B+-tree design was on. :) I have no idea what this will all lead to, if anything at all even, as it is purely an evening research project for me at the moment.

They application-facing query system itself in Akonadi Next is slowly making its way to something nice, but that's another topic for another day.


bruederli's picture
Wed, 2015-05-20 11:22

While Roundcube One originates from a private fun project with email – and only email – in mind, we have learned our lessons and are committed to do Roundcube Next right from the ground up. In the year 2015, communication combines a variety of tools we need to connect to each others. And that’s exactly what we aim to cover with the architectural design of Roundcube Next. It shall become a solid and open foundation for building communication apps on top of it. Email will certainly remain a key component as it still is the most important means of communication today. But there’s more and therefore we want to make Roundcube Next the WordPress of communication if you will.

After we opened Roundcube up for plugins in version 0.3, we witnessed an amazing creativity in what people start building around an open source email application. From a car dealer system to mailing list archives, many custom solutions were built on top of Roundcube. This definitely inspired us to support and facilitate this aspect in the very core of the new system.

The plugin infrastructure of Roundcube Next will be your new best friend for building web apps for your specific communication needs. The new core will provide an easy-to-use framework with lots of reusable components for both building the UI of your application as well as for synchronizing the data to the server and the underlying storage backend of your choice.

So if you’re a developer who got annoyed with the limitations of closed systems from the big vendors and you don’t want to build a complex web application from scratch, Roundcube Next deserves your attention and support. Go to https://roundcu.be/next and get yourself a backstage pass for the Roundcube Next forums or even a seat in the advisory committee. And don’t forget to spread the word about this new opportunity for the free software world.


roundcube's picture
Wed, 2015-05-20 11:22

While Roundcube One originates from a private fun project with email – and only email – in mind, we have learned our lessons and are committed to do Roundcube Next right from the ground up. In the year 2015, communication combines a variety of tools we need to connect to each others. And that’s exactly what we aim to cover with the architectural design of Roundcube Next. It shall become a solid and open foundation for building communication apps on top of it. Email will certainly remain a key component as it still is the most important means of communication today. But there’s more and therefore we want to make Roundcube Next the WordPress of communication if you will.

After we opened Roundcube up for plugins in version 0.3, we witnessed an amazing creativity in what people start building around an open source email application. From a car dealer system to mailing list archives, many custom solutions were built on top of Roundcube. This definitely inspired us to support and facilitate this aspect in the very core of the new system.

The plugin infrastructure of Roundcube Next will be your new best friend for building web apps for your specific communication needs. The new core will provide an easy-to-use framework with lots of reusable components for both building the UI of your application as well as for synchronizing the data to the server and the underlying storage backend of your choice.

So if you’re a developer who got annoyed with the limitations of closed systems from the big vendors and you don’t want to build a complex web application from scratch, Roundcube Next deserves your attention and support. Go to https://roundcu.be/next and get yourself a backstage pass for the Roundcube Next forums or even a seat in the advisory committee. And don’t forget to spread the word about this new opportunity for the free software world.


greve's picture
Tue, 2015-05-19 09:02

If you are a user of Roundcube, you want to contribute to roundcu.be/next. If you are a provider of services, you definitely want to get engaged and join the advisory group. Here is why.

Free Software has won. Or has it? Linux is certainly dominant on the internet. Every activated Android device is another Linux kernel running. At the same time we see a shift towards “dumber” devices which are in many ways more like thin clients of the past. Only they are not connected to your own infrastructure.

Alerted by the success of Google Apps, Microsoft has launched Office 365 to drive its own transformation from a software vendor into a cloud provider. Amazon and others have also joined the race to provide your collaboration platform. The pull of these providers is already enormous. Thanks to networking effects, economies of scale, and ability to leverage deliberate technical incompatibilities to their advantage, the drawing power of these providers is only going to increase.

Open Source has managed to catch up to the large providers in most functions, bypassing them in some, being slightly behind in others. Kolab has been essential in providing this alternative especially where cloud based services are concerned. Its web application is on par with Office 365 and Google Apps in usability, attractiveness and most functions. Its web application is the only fully Open Source alternative that offers scalability to millions of users and allows sharing of all data types in ways that are superior to what the proprietary competition has to offer.

Collaborative editing, chat, voice, video – all the forms of synchronous collaboration – are next and will be added incrementally. Just as Kolab Systems will keep driving the commercial ecosystem around the solution, allowing application service providers (ASP), institutions and users to run their own services with full professional support. And all parts of Kolab will remain Free and Open, as well as committed to the upstream, according to best Free Software principles. If you want to know what that means, please take a look at Thomas Brüderlis account of how Kolab Systems contributes to Roundcube.

TL;DR: Around 2009, Roundcube founder Thomas Brüderli got contacted by Kolab at a time when his day job left him so little time to work on Roundcube that he had played with the thought of just stepping back. Kolab Systems hired the primary developers of Roundcube to finish the project, contributing in the area of 95% of all code in all releases since 0.6, driving it its 1.0 release and beyond. At the same time, Kolab Systems carefully avoided to impose itself on the Roundcube project itself.

From a Kolab perspective, Roundcube is the web mail component of its web application.

The way we pursued its development made sure that it could be used by any other service provider or ISV. And it was. Roundcube has an enormous adoption rate with millions of downloads, hundreds of thousands of sites and an uncounted number beyond the tens of millions. According to cPanel, 62% of their users choose Roundcube as their web mail application. It’s been used in a wide number of other applications, including several service providers that offer mail services that are more robust against commercial and governmental spying. Everyone at Kolab considers this a great success, and finds it rewarding to see our technology contribute essential value to society in so many different ways.

But while adoption sky-rocketed, contribution did not grow in the same way. It’s still Kolab Systems driving the vast majority of all code development in Roundcube along with a small number of occasional contributors. And as a direct result of the Snowden revelations the development of web collaboration solutions fragmented further. There are a number of proprietary approaches, which should be self-evidently disqualified from being taken serious based on what we have learned about how solutions get compromised. But there are also Open Source solutions.

The Free Software community has largely responded in one of two ways. Many people felt re-enforced in their opinion that people just “should not use the cloud.” Many others declared self-hosting the universal answer to everything, and started to focus on developing solutions for the crypto-hermit.

The problem with that is that it takes an all or nothing approach to privacy and security. It also requires users to become more technical than most of them ever wanted to be, and give up features, convenience and ease of use as a price for privacy and security. In my view that ignores the most fundamental lesson we have learned about security throughout the past decades. People will work around security when they consider it necessary in order to get the job done. So the adoption rate of such technologies will necessarily remain limited to a very small group of users whose concerns are unusually strong.

These groups are often more exposed, more endangered, and more in need of protection and contribute to society in an unusually large way. So developing technology they can use is clearly a good thing.

It just won’t solve the problem at scale.

To do that we would need a generic web application geared towards all of tomorrow’s form factors and devices. It should be collaboration centric and allow deployment in environments from a single to hundreds of millions of users. It should enable meshed collaboration between sites, be fun to use, elegant, beautiful and provide security in a way that does not get into the users face.

Fully Free Software, that solution should be the generic collaboration application that could become in parts or as a whole the basis for solutions such as mailpile, which focus on local machine installations using extensive cryptography, intermediate solutions such as Mail-in-a-Box, all the way to generic cloud services by providers such as cPanel or Tucows. It should integrate all forms of on-line collaboration, make use of all the advances in usability for encryption, and be able to grow as technology advances further.

That, in short, is the goal Kolab Systems has set out to achieve with its plans for Roundcube Next.

While we can and of course will pursue that goal independently in incremental steps we believe that would be missing two rather major opportunities. Such as the opportunity to tackle this together, as a community. We have a lot of experience, a great UI/UX designer excited about the project, and many good ideas.

But we are not omniscient and we also want to use this opportunity to achieve what Roundcube 1.0 has not quite managed to accomplish: To build an active, multi-vendor community around a base technology that will be fully Open Source/Free Software and will address the collaborative web application need so well that it puts Google Apps and Office 365 to shame and provides that solution to everyone. And secondly, while incremental improvements are immensely powerful, sometimes leapfrogging innovation is what you really want.

All of that is what Roundcube Next really represents: The invitation to leapfrog all existing applications, as a community.

So if you are a user that has appreciated Roundcube in the past, or a user who would like to be able to choose fully featured services that leave nothing to be desired but do not compromise your privacy and security, please contribute to pushing the fast forward button on Roundcube Next.

And if you are an Application Service Provider, but your name is not Google, Microsoft, Amazon or Apple, Roundcube Next represents the small, strategic investment that might just put you in a position to remain competitive in the future. Become part of the advisory group and join the ongoing discussion about where to take that application, and how to make it reality, together.

 


Aaron Seigo's picture
Wed, 2015-05-06 11:37

Today at 13:00 UTC I will be hosting a Google+ Hangout with Roundcube founder and lead developer, Thomas Brüderli. I will link the video below once we are done, but everyone is welcome to join us live and provide feedback and questions in IRC while we're chatting.

So, what are we going to talk about? Well, Roundcube, of course! :) I'll be asking Thomas why he decided that now was the appropriate time for a refactor of Roundcube, what it means for Roundcube 1.x (the current stable release), and if we have time we'll start tucking into the current feature and design thinking.

So come join us on the Roundcube G+ page / Youtube channel as well as the #roundcube channel on irc.freenode.net today at 13:00 UTC!

Hope to see you all there!

Update: The video is up on Youtube, with some blank airtime (and a fun moment of feedback) edited out .. you can watch it below:


bruederli's picture
Mon, 2015-05-04 13:38

It all started with this hypothetical question: how would we implement Roundcube if we could start over again? And now this idea has already grown into a concrete plan how to create the responsive, fast and beautiful successor of Roundcube.

The architectural changes necessary for this are clearly too big to be applied to the current Roundcube codebase without breaking the compatibility for most plugins and extensions. So we won’t take that risky path but rather define Roundcube One as feature complete and focus on a new core engine for the future Roundcube webmail application. This will enable everybody to participate in the process of reshaping the architecture and to adapt the existing plugins to the new API as we go along.

There’s no doubt that such a major refactoring is a huge endeavor and requires a substantial effort in concepts, development and testing. Nothing to be done over the weekend but we also don’t want to spend another 10 years to make this become reality. Luckily we have strong partners and supporters to push this forward. Kolab Systems has offered to drive this project by contributing their well established software development capabilities, from project management, developer power to QA and testing. In addition to that, the folks at Kolab Digital can’t wait to share their expertise on the UX and design part. However, such a level of professionalism also comes with a price.

Getting help from the crowd to back this

In order to enable both Kolab Systems and Kolab Digital to actually assign the necessary resources to the “Roundcube Next” project, we sat together and decided that it would make sense to reach out to the entire Roundcube community to help make this happen. Yesterday, we proudly announced the crowd funding campaign at the end of the Kolab Summit in The Hague.

The Funding StepsTogether, we can make this a great success! Please help spread the word, back the campaign with a pledge, and join us for what is going to be a fantastic journey. Regular updates will be posted to the crowd funding page, and we are excited to make the run to our initial goal and beyond with you!


roundcube's picture
Mon, 2015-05-04 13:38

It all started with this hypothetical question: how would we implement Roundcube if we could start over again? And now this idea has already grown into a concrete plan how to create the responsive, fast and beautiful successor of Roundcube.

The architectural changes necessary for this are clearly too big to be applied to the current Roundcube codebase without breaking the compatibility for most plugins and extensions. So we won’t take that risky path but rather define Roundcube One as feature complete and focus on a new core engine for the future Roundcube webmail application. This will enable everybody to participate in the process of reshaping the architecture and to adapt the existing plugins to the new API as we go along.

There’s no doubt that such a major refactoring is a huge endeavor and requires a substantial effort in concepts, development and testing. Nothing to be done over the weekend but we also don’t want to spend another 10 years to make this become reality. Luckily we have strong partners and supporters to push this forward. Kolab Systems has offered to drive this project by contributing their well established software development capabilities, from project management, developer power to QA and testing. In addition to that, the folks at Kolab Digital can’t wait to share their expertise on the UX and design part. However, such a level of professionalism also comes with a price.

Getting help from the crowd to back this

In order to enable both Kolab Systems and Kolab Digital to actually assign the necessary resources to the “Roundcube Next” project, we sat together and decided that it would make sense to reach out to the entire Roundcube community to help make this happen. Yesterday, we proudly announced the crowd funding campaign at the end of the Kolab Summit in The Hague.

The Funding StepsTogether, we can make this a great success! Please help spread the word, back the campaign with a pledge, and join us for what is going to be a fantastic journey. Regular updates will be posted to the crowd funding page, and we are excited to make the run to our initial goal and beyond with you!


roundcube's picture
Sun, 2015-05-03 19:00

Roundcube prouldy announces the crowd funding campaign to
bring our vision of a better email experience to reality.

The web has evolved a lot in the last decade, and we want Roundcube
to take full advantage of the best web technologies available today.
Therefore it’s time for a dramatic change to the Roundcube architecture
and to also to rethink email in general, how it’s used today and how we
could use the new technologies to give the best user experience to
everyday communication.

Applying what we’ve learned from our first 10 years of experience developing
Roundcube, we have been working on a development plan for how to achieve our
new goals. And in order to finally make this happen, we also need your support
to drive the professional software development process behind this plan.

Please join the fun at roundcu.be/next and support our
crowd funding campaign either directly or by simply spreading the word about it.

Roundcube Next Campaign Video


Aaron Seigo's picture
Sun, 2015-05-03 17:36

Today we closed out the first (and quite successful) Kolab Summit in front of both the Kolab and openSUSE attendees with some really big news: the Roundcube team has launched a significant new development project to give Roundcube, the world's most popular free software webmail system, a modern fluid "single-page" user interface. The UI will be rendered entirely in the browser, and the server will only do minimal business logic in support of that.

The focus is on modularity (to make it easier to extend Roundcube's core features), scalability, and deployability. At the same time, the Roundcube team needs to maintain the current version (we have commitments to clients and users that stretch years into the future) as well as build a migration strategy to the new version when it becomes available. Thomas, the founder and project lead for Roundcube, gave a great presentation explaining the whole thing.

As you might imagine, achieving these goals will involve refactoring nearly the entire codebase. We plan to commit three developers along with a UI designer to the project with support of the Kolab Systems project management infrastructure and staff.

So this is a pretty big project, but quite achievable. While discussing how best to make this all happen, the Roundcube team decided that it would make sense to reach out to the entire Roundcube user community to help make this happen, and therefore launched a crowd funding campaign today at Indiegogo.

Quite a way to close out the conference!

http://igg.me/at/roundcubenext

Together, we can make this a great success! Please help spread the word, back the campaign with a pledge, and join us for what is going to be a fantastic journey. Regular updates will be posted to the crowdfunding page, and we are excited to make the run to our initial goal of $80,000 with you!


Aaron Seigo's picture
Sun, 2015-05-03 14:17

On the first day of the Kolab Summit we announced that Kolab is getting full extended MAPI support. That was in itself a pretty fantastic announcement, but it was accompanied by announcements of instant messaging, WebRTC and collaborative editing.

Here is a picture which I think captures what the LibreOffice and WebODF people think about this direction, captured over lunch today:


Aaron Seigo's picture
Sat, 2015-05-02 11:04

Kolab Summit - Day 1

Yesterday I delivered a keynote at the openSUSE conference about the best feature of Free software: freedom. This is a message that is easy to lose sight of in the maker/creator community around free software given the understandable focus on business goal metrics such as market penetration, developer adoption, innovation rates, etc. You can see my slides here, and the video of the presentation will be uploaded later by the conference team. (I'll link to it when it appears.) The questions after the presentation were excellent as well and the conversations continued out into the hallways afterwards.

Kolab Summit - Day 1

That was yesterday. Today, the Kolab Summit began. Georg Greve kicked things off by sharing the vision for Kolab this year (slides here).

He covered three areas of focus for Kolab this year:

  1. Real-time collaboration: IM, WebRTC, document editing. This will allow us to compliment the existing asynchronous communication Kolab excels at (email, calendaring, notes, files, etc.) with synchronous, collaborative editing.
  2. User experience refactor: major work is being done with the Kolab clients, in particular the Roundcube client. The goal here is to surpass what is available elsewhere in the market to keep free software as a leader in this area.
  3. Full extended MAPI support. Yes, Kolab will be able to support Outlook out of the box. Fully. The lead OpenChange developer is here to discuss this further later in the summit.

There are many other projects we are digging into significantly, and Kolab's system architect, Jeroen van Meeuwen, followed Georg with a technical roadmap overview. He not only filled in the details behind the three focus areas Georg highlighted, but shared our road map for data loss prevention, multi-factor authentication, in-web-browser encryption, Akonadi Next for the desktop client ... in short we're very, very busy.

Everything we are doing has a very clear use case that needs these tools so that they can choose to also use free software for their collaboration needs.


Aaron Seigo's picture
Wed, 2015-04-22 14:16

In my last blog entry, I mentioned that we have been working on a comprehensive data loss prevention (DLP) and audit trail system for use with Kolab, with the end goal being not only DLP but also a platform for business intelligence. In that entry I listed the three parts of the system, noting that I'd be writing about one at a time. I had hoped to jump on the first of those a day or two after writing the entry, but life and work intervened and then I was off on a short family vacation ... but now I'm back. So let's talk about the capture side of the system.

Kolab can be viewed as a set of cooperative microservices: smtp, imap, LDAP, spam/virus protection, invitation auto-processing, web UI, etc. etc. There are a couple dozen of these and up until now they have all done the traditional, and correct, thing of logging events to a system log.

This has numerous drawbacks, however. First, on a distributed system where different services are running on different hosts (physical or VMs), the result is data spread over many systems. Not great for subsequent reporting. At the time of logging, the events are in a "raw" state: each service likely does not know about the rest of the Kolab services and thus how their events relate to the whole system. With logs going through the host systems it makes it difficult to ensure that they are not easily tampered with; this can be somewhat alleviated by setting up remote logging but this also only goes so far. Finally, logging tends to be a firehose of data and for our specific interests here we want a very specific sub-stream of that total flow.

So we have written yet another service whose entire job is to collect events as they are generated. This service is itself distributed, allowing collection agents to be run across a cluster running a Kolab instance, and it stores its data in a dedicated key-value store which can be housed on an isolated (and specially secured, if desire) system. The program running this service is called Egara, which is Sumerian for "storehouse", and it is written in Erlang due to its robustness (this service must simply never go down), scalability and distributed communication features. The source repository can be found here. Egara itself is part of the overall DLP/auditing system we have named Bonnie.

The high-level purpose of Egera is to create a consistent and complete history of what happens to objects within the groupware system over time. An "object" might be an email, a user account, a calendar event, a tag, a note, a todo item, etc. An event (or "what happens") including things such as new objects, deletions, setting of flags or tags, changing the state (e.g. from unread to read), starting or tearing down an authenticated session, etc. In other words, its job is to create, in real-time, a complete history of who did what when. As such I've come to view it as an automated historian for your world of groupware.

Egara itself is divided into three core parts:

  • incoming handlers: these components implement a standard behavior and are responsible for collecting events from a specific service (e.g. cyrus-imap) and relaying them to the core application once received
  • event normalizers: these workers process events from the new event queue and are tasked with normalizing and augmenting the data within them, creating complete point-in-time additions to the history. Many events come in with simple references to other objects, such as a mail folder; the event normalization workers need to turn those implicit bits of information into explicit links that can be reliably followed over time
  • middleware: these are mainly the bits that provide process supervision, populate and manage the shared queues of events as information arrives from incoming handlers and is processed by normalizers.

This all happen asynchronously and provides guarantees at each step of correct handling (inasumuch as each reporting service allows for that). This means that individual normalizers can fail in even spectacular fashion and not disrupt the system, that an admin can halt and restart the system at will without fear of loss of events (save those that are generated during downtime periods, assuming a full Egara take-down), etc.

Final storage is done in a Riak database, with queues managed by the Mnesia database built into Erlang's OTP system itself. Mnesia can best be thought of as a built-in Redis: entirely in-memory (fast) with disk backing (robust); just add built-in clustering and native, first-class API for storage and retrieval (e.g. we are able to use Erlang functions to do perform updates and filtering over all or part of a queue's dataset). Data in Mnesia is stored as native Erlang records, while data in Riak is stored as JSON documents.

Incoming events may be any format and any delivery mechanism. They can be parallelized, spread across a cluster of machines ... it doesn't matter. The incoming handler is tasked with translating the stream of events into an Erlang term that can be passed on to the normalizer for processing. This allows us to extend Egara in a very easy way with new service-specific handlers to virtually any dataset we wish to keep track of within Kolab or its surroundings.

Normalizers will eventually also join this level of abstraction, though right now the sole worker implementation is specific to groupware data objects. Future releases of Egara will add support for different workers for different classes of events, giving a nice symmetry with the incoming event handlers.

The middleware is designed to be used without modification as the system grows in capability while being scalable. Multiple instances can be run across different systems and the results should (eventually) be the same. I say "eventually" since in such a system one can not guarantee the exact order of events, only the exact results after some period of time. Or, in more familiar terms, it is eventually consistent.

The whole system is quite flexible at runtime, as well. One can configure which kinds of events one cares to track; which data payloads (if any) to archive; which incoming handlers to run on a given node, etc. This will expand over time as well to allow normalizers and their helpers to be quarantined to specific systems within a cluster.

Egara works nicely with Kolab 3.4 and Kolab Enterprise 14, though Bonnie is not officially a part of either. I expect the entire system will be folded into a future Kolab release to ease usage. It will almost certainly remain an optional component, however: not everyone needs these features, and if you don't then there's no reason to pay the price of the runtime overhead and maintenance.

That's a "50,000 foot" view of the historian component of Bonnie. The next installments in this blog series will look a bit closer at the storage model, history querying and replayability and, finally, what this means for end-users and organizations running Kolab with the Bonnie suite.


Aaron Seigo's picture
Wed, 2015-04-01 08:58

Working with Kolab has kept me busy on numerous fronts since I joined near the end of last year. There is the upcoming Kolab Summit, refreshing Kolab Systems' messaging, helping with progress around Kolab Now, collaborating on development process improvement, working on the design and implementation of Akonadi Next, the occassional sales engineering call ... so I've been kept busy, and been able to work with a number of excellent people in the process both in Kolab Systems and the Kolab community at large.

While much of that list of topics doesn't immediately bring "writing code" to mind, I have had the opportunity to work on a few "hands on keyboard, writing code" projects. Thankfully. ;)

One of the more interesting ones, at least to me, has been work on an emerging data loss prevention and audit trail system for Kolab called Bonnie. It's one of those things that companies and governmental users tend to really want, but which is fairly non-trivial to achieve.

There are, in broad strokes, three main steps in such a system:

  1. Capturing and recording events
  2. Storing data payloads associated with those events
  3. Recreating histories which can be reviewed and even restored from

I've been primarily working on the first two items, while a colleague has been focusing on the third point. Since each of these points is a relatively large topic on their own, I'll be covering each individually in subsequent blog entries.

We'll start in the next blog entry by looking at event capture and storage, why it is necessary (as opposed to simply combing through system logs, e.g.) and what we gain from it. I'll also introduce one of the Bonnie components, Egara, which is responsible for this set of functionality.


Aaron Seigo's picture
Fri, 2015-03-27 18:47

Today "everyone" is online in one form or another, and it has transformed how many people connect, communicate, share and collaborate with others. To think that the Internet really only hit the mainstream some 20 years ago. It has been an amazingly swift and far-reaching shift that has touched people's personal and professional lives.

So it is no surprise that the concept of eGovernment is a hot one and much talked about. However, the reality on the ground is that governments tend not to be the swiftest sort of organizations when it comes to adopting change. (Which is not a bad thing; but that's a topic for another blog perhaps.) Figuring out how to modernize the communication and interaction of government with their constituencies seems to largely still be in the future. Even in countries where everyone is posting pictures taken on their smartphones of their lunch to all their friends (or the world ...), governments seem to still be trying to figure out how to use the Internet as an effective tool for democratic discourse.

The Netherlands is a few steps ahead of most, however. They have an active social media presence which is used by numerous government offices to collaborate with each other as well as to interact with the populace. Best of all, they aren't using a proprietary, lock-in platform hosted by a private company oversees somewhere. No, they use a free software social media framework that was designed specifically for this: Pleio.

They have somewhere around 100,000 users of the system and it is both actively used and developed to further the aims of the eGovernment initiative. It is, in fact, an initiative of the Programme Office 2.0 with the Treasury department, making it a purposeful program rather than simply a happy accident.

In their own words:

The complexity of society and the need for citizens to ask for an integrated service platform where officials can easily collaborate with each other and engage citizens.

In addition, hundreds of government organizations all have the same sort of functionality needed in their operations and services. At this time, each organization is still largely trying to reinvent the wheel and independently purchase technical solutions.

That could be done better. And cheaper. Gladly nowadays new resources are available to work together government-wide in a smart way and to exchange knowledge. Pleio is the platform for this.

Just a few days ago it was anounced publicly that not only is the Pleio community is hard at work on improving the platform to raise the bar yet again, but that Kolab will be a part of that. A joint development project has been agreed to and is now underway as part of a new Pleio pilot project. You can read more about the collaboration here.


Fri, 2015-03-27 17:44

Kolab has joined Pleio to aid in the production of a new version of the Pleio social media platform. Pleio is currently used by some 100,000 people within the Dutch government to interact with each other and their constituencies as part of the nation's eGovernment initiative. The new pilot project was announced by Kolab Systems CEO Georg Greve at the Netfutures conference on March 25 in Brussels, and underscores the participating organizations' shared fundamental vision with regards to open source software, open standards and data security.

The goal of the pilot is to to provide Pleio with a broader, more modular base architecture by integrating components from the Kolab collaboration suite into its core. Both products will remain independent, continuing to be developed within their respective communities, but efforts will be bundled wherever possible with a focus on strengthening their ecosystems together.

The Kolab 4 Pleio pilot project was announced along with a call for participation on the Pleio website and development planning has begun.

About Pleio

Today's citizens are asking for an integrated service platform where officials can easily collaborate with each other and engage citizens, and hundreds of government organizations share this exact same need. Historically, each organization has largely tried to reinvent the wheel and independently purchase technical solutions. That could be done better and more economically. Pleio is the open platform that provides a social network with integrated blogging, groups and meetings, wikis, and file sharing designed for use in eGovernment.

Pleio is an initiative of the Programme Officer 2.0 with the Treasury in the Netherlands in coordination with other Dutch governmental agencies.

About Kolab

Kolab Groupware is a Free Software collaboration system providing email, events and appointments, contacts, notes, file cloud and more that scales from the home office to enterprise deployments. All major desktop and mobile systems are supported, thanks to broad usage of open standards and formats.


roundcube's picture
Wed, 2015-03-18 01:00

This is the first service release to update the stable version 1.1. It contains
some important bug fixes and improvements in error handling as well as a few
new features and configuration options. See the full changelog here.

It’s considered stable and we recommend to update all productive installations
of Roundcube with this version. Download it from roundcube.net/download,

Please do backup your data before updating!


Aaron Seigo's picture
Mon, 2015-03-16 11:52

We just announced that registration and presentation proposal submission is now open for the Kolab Summit 2015 which is being held in The Hague on May 2-3.

Just as Kolab itself is made up of many technologies, many technologies will be present at the summit. In addition to topics on Kolab, there will be presentations covering Roundcube, KDE Kontact and Akonadi, cyrus imap, and OpenChange among others. We have some pretty nifty announcements and reveals already lined up for the event, which will be keynoted by George Greve (CEO of Kolab Systems AG) and Jeroen van Meeuwen (lead Kolab architect). Along with the usual BoFs and hacking rooms, this should be quite an enjoyable event.

As an additional and fun twist, the Kolab Summit will be co-located with the openSUSE conference which is going on at the same time. So we'll have lots of opportunity for "hallway talks" with Geekos as well. In fact, I'll be giving a keynote presentation at the openSUSE conference about freedom as innovation. A sort of "get the engines started" presentation that I hope provokes some thought and gets some energy flowing.


Aaron Seigo's picture
Mon, 2015-03-16 11:52

Kolab Summit 2015

We just announced that registration and presentation proposal submission is now open for the Kolab Summit 2015 which is being held in The Hague on May 2-3.

Just as Kolab itself is made up of many technologies, many technologies will be present at the summit. In addition to topics on Kolab, there will be presentations covering Roundcube, KDE Kontact and Akonadi, cyrus imap, and OpenChange among others. We have some pretty nifty announcements and reveals already lined up for the event, which will be keynoted by George Greve (CEO of Kolab Systems AG) and Jeroen van Meeuwen (lead Kolab architect). Along with the usual BoFs and hacking rooms, this should be quite an enjoyable event.

As an additional and fun twist, the Kolab Summit will be co-located with the openSUSE conference which is going on at the same time. So we'll have lots of opportunity for "hallway talks" with Geekos as well. In fact, I'll be giving a keynote presentation at the openSUSE conference about freedom as innovation. A sort of "get the engines started" presentation that I hope provokes some thought and gets some energy flowing.


Mon, 2015-03-16 10:38

The Kolab Collaboration Suite has been adopted by companies and governments around the world, making it one of most successful "poster children" for Free Software and Open Standards. In order to chart the next steps forward, the Kolab community is excited to announce the inaugural Kolab Summit to be held in The Hague on May 2-3, 2015.


Come and join lead developers, deployment experts and Kolab partners for two days packed with talks, code sprints and social events! The Kolab Summit is your best opportunity to meet and network with the community which has grown around Kolab since its first release in 2003. Registration is free but space is limited, so be sure to register soon.


Keynotes from Georg Greve, CEO of Kolab Systems AG, and Jeroen van Meeuwen, lead Kolab architect, will open the speaking schedule, providing a look at what is coming for Kolab in 2015 and beyond. Lead developers from Roundcube, the worlds most popular webmail application, along with key participants from KDE Kontact, cyrus-imap, Seafile, OpenChange and more will in attendance and presenting talks.


In addition to development topics, deployment and support of Kolab will be another hot topic at the event. Leading experts in Kolab deployment will be in attendance, and talks covering deployment strategies such Docker are already scheduled.


There are still several slots open in the speaking schedule, however. The call for presentations also begins today and is open to everyone interested in presenting a talk at the Summit. Submissions should be made before April 1st for consideration, with confirmations of accepted proposals happening directly afterwards. The submission form is available once you are registered.


There will also be workshop and development rooms where attendees can gather to discuss topics of interest and work on the Kolab code base in person. These two intense days focused on the future of free, secure and open collaboration will be co-located alongside the openSUSE Conference 2015, expanding the possibilities even further.


The entire Kolab community hopes to see you there!


Andreas Cordes's picture
Fri, 2015-03-13 22:20

Hello,

Seafile is an open source cloud software which is free for private use.

There is also a professional edition which is not necessary for my needs. I just want to sync files across more than one device. In the past I used ownCloud which was pretty good for my needs.

First I decided to have ownCloud integrated in Kolab as a backend but I had a bit lack of time so developing a new driver for chwala was not so easy.

After a while I noticed that Seafile is integrated in chwala 0.3.0 and with Kolab 3.4 it is quite stable to install.

Last week I managed all the stuff and here is my Step-By-Step Guide:

  1. Install Kolab 3.4 and test it
    It's obvious that you need Kolab for this. Please refer to the kolab.org web page for an installation guide for your distro
  2. Install seafile and test it.
    The homepage of seafile will guide you through all the stuff you need for that. So just download, extract and run it.
  3. Connect Seafile to Kolab-LDAP
    Following Seafile->Using LDAP is exactly what I did. My installation was spreaded over /opt/seafile, /opt/seafile/seafile-server-latest and /mnt/seafile
    But you'll find your ccnet.conf and add the LDAP part to it (please change the values according to your installation):
    [LDAP]
    HOST = ldap://127.0.0.1
    BASE = ou=People,dc=example,dc=com
    USER_DN = cn=directory manager
    PASSWORD = youdon'tknowjack
    LOGIN_ATTR = mail
  4. Now test seafile again if you can now login with your kolab main mailaddress.
    Please keep in mind that each user has to login for the first time to seafile in order to get the right folders.
  5. If everything is ok, now you have to use Apache as a proxy for Seafile
    Following Seafile -> Deploy with Apache was ok for me for the first time.
    Well, this will break your Kolab if you follow the steps directly.
  6. If this is ok as well, now fine tune a bit the Apache configuration for Seafile.
    These lines should be changed in your apache.conf (or vhost):
    #
    # seahub
    #
    RewriteRule ^/(media.*)$ /$1 [QSA,L,PT]
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteRule ^(.*)$ /seahub.fcgi$1 [QSA,L,E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]

    This will redirect all request to Seafile, but if you only have one SSL certificate and one domain you have to ignore this for all the Kolab modules.
    So please add the following RewriteCond to the config:

    RewriteRule ^/(media.*)$ /$1 [QSA,L,PT]
    RewriteCond %{REQUEST_FILENAME} !-f
    RewriteCond %{REQUEST_URI} !^/roundcubemail.*
    RewriteCond %{REQUEST_URI} !^/Microsoft.*
    RewriteCond %{REQUEST_URI} !^/iRony.*
    RewriteCond %{REQUEST_URI} !^/chwala.*
    RewriteCond %{REQUEST_URI} !^/kolab.*
    RewriteRule ^(.*)$ /seahub.fcgi$1 [QSA,L,E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]

    The first RwriteCond tells Apache to rewrite the URL if the file does not exist. For the Kolab installation there are some URL more not to check against an existing file because they are an Alias in the Apache way. So we ignore this rewrite when the URL starts with "roundcube", "Microsoft" (for active sync) and so on.
    Reading the conf above should be:

    Rewrite all URL starting with /media
    ReWrite the URL when the requested file does not exist (! -f)
    AND (is implicitly) the request does not start with roundcube (!/^/roundcubemail.*)
    AND the request does not start with Microsoft
    AND the request does not start with iRony
    AND the erquest does not start with chwala
    AND the request does not start with kolab
    TO /seahub.fcgi PLUS the request string including the trailing slash (^(.*)$ /seahub.fcgi$1)

  7. Now test your Kolab and Seafile again, everything should now work on the same server.
  8. It's time to combine Chwala with Seafile. The most interesting part. :-)
    Please edit your /etc/roundcubemail/config.inc.php file and all the Seafile stuff:
    // seafile
    $config['fileapi_backend'] = 'seafile';
    $config['fileapi_seafile_host'] = 'www.example.com';
    $config['fileapi_seafile_ssl_verify_peer'] = false;
    $config['fileapi_seafile_ssl_verify_host'] = false;
    $config['fileapi_seafile_cache_ttl'] = '14d';
    $config['fileapi_seafile_debug'] = true;
  9. That's it.

Now you can use your Seafile server in Kolab as a file storage. But keep in mind, password protected folders are not accessible through chwala this way.

Feel free to leave any comments and
Greetz

Timotheus Pokorra's picture
Thu, 2015-03-05 08:20

Previously, my nightly builds of Kolab have been built and hosted here:

https://obs.kolabsys.com/project/show/home:tpokorra:branches:Kolab:Development

There were 2 problems recently:

  • I was not able to add builds for CentOS7. The error message is:
    Failed to add project or repository: unable to walk on path 'CentOS:7/standard'
  • Each time there was a new patch or release on Kolab:Development, I needed to manually remove the patch from the nightly package because the patch was already part of git master. I also needed to resolve conflicts in the debian.changelog and package dsc file because the version number changed.

UPDATE: since there was an easy fix for Point 1 (just edit the Meta file of the project to add different Operating System), I also realised I could update the OBS packaging instructions the same way as I do for LBS, avoiding conflicts. So we will see if that works over the coming weeks…

After some improvements to my LightBuildServer, I am now able to build the nightly packages on my own LBS server.

This is done in two steps:

  1. Updating the package instructions: This is done by an LBS job, that basically is running this script: The package instructions are downloaded from https://obs.kolabsys.com/project/show/Kolab:Development, and the source tarballs are downloaded from https://git.kolab.org/. Then the changelog is updated, patches removed, some other patches applied… Also some conversion is done on the files to make Debian happy to build them. The result is uploaded to https://github.com/TBits/lbs-kolab-nightly
  2. Then I run different jobs on LBS, for each Operating System that I currently support: CentOS6, CentOS7, and Debian, to rebuild the selected packages with nightly tarballs.
    The result can be viewed here: https://lbs.solidcharity.com/project/tbits.net/kolab-nightly

These steps are executed each night, by a cronjob that initiates the builds on the LBS, by calling for example

tbitscredentials=tbits.net/secret
wget -O /dev/null https://lbs.solidcharity.com/triggerbuildproject/tbits.net/kolab-nightly/centos/6/amd64/$tbitscredentials

To test the nightly builds, you can install the nightly repository like this, additionally to the Kolab 3.4 and Kolab Development repo:

For CentOS6:

yum install yum-utils
yum-config-manager --add-repo https://download.solidcharity.com/repos/tbits.net/kolab-nightly/centos/6/lbs-tbits.net-kolab-nightly.repo

For CentOS7:

yum install yum-utils
yum-config-manager --add-repo https://download.solidcharity.com/repos/tbits.net/kolab-nightly/centos/7/lbs-tbits.net-kolab-nightly.repo

For Debian Wheezy:

apt-get install apt-transport-https
echo 'deb https://download.solidcharity.com/repos/tbits.net/kolab-nightly/debian/wheezy/ /' >> /etc/apt/sources.list
apt-get update

With the LightBuildServer I have full control over the builds, can just modify a config file to add a new target OS (if my LXC scripts support it), and can review the history of the package instructions on Github.

Of course, one disadvantage of LBS compared to OBS is: the LightBuildServer is not intended to directly support the work in a team. Team-work happens via Github (or your self hosted Gitlab), and every team member could install his own LightBuildServer.

A feature that LBS is still missing is that multiple build containers are just assigned with any job, without checking if one job should wait for another job to finish. A first step would be to distribute the jobs per Operating System to different build containers. Well, still lots of room for improvement…


Andreas Cordes's picture
Thu, 2015-03-05 00:24

Hello @all,

+Kolab.org Community released Version 3.4 and it was time to get the new updates up and running.

This time there is a new driver for "chwala" integrated for +Seafile.VN so I gave it a try and tried to mix it all together.

After lots pf solving dependencies and so on I managed to get the new version up and running with Seafile attached.

That's quite amazing, for this you have to configure Seafile through Apache and connect to the LDAP.

When you were able to manage all these things, you can easily save attachments in your cloud and syncing them to your smartphone. :-)

There are still some issues on my Cubietruck which I have to solve but my Idea is to rebuild all the stuff from scratch in a chroot environment and track down all steps to get it working.

Two Screenshots showing the new skin for +Roundcube and the integration of Seafile into Chwala.

That's the new Chameleon skin for Roundcube

The same folders in seafile and chwala except the password protected one.

So the next couple of days I'm trying to set up a new build environment for the ARM-HF packages for Kolab and a Step-By-Step description.

Greetz


Daniel Hoffend's picture
Wed, 2015-03-04 23:06

With the recently announced Kolab Groupware 3.4 Version a new default skin got introduced: "Chameleon". The new skin (which is still based on larry) provides a very clean layout which is very easy to customize.

Thanks to the LESS CSS Compilier you basically only need to define a few main colors (I changed 2), compile the style sheets and exchange the skinlogo. Most of the the other colors used within roundcube are based on your main-color (lighten by 30%, etc.).

If you don't like some of the colors in the icons, change them too by either changeing the psd files or just edit the png files (which sufficent) and within 10-120 minutes you got your personal kolab skin running.

The file structure

The roundcube theme is split into 2 componentes. The internal skin parts which consists of templates, configuration files, etc. and the asset part like CSS and images.

/usr/share/roundcubemail/skins/<name>/
/usr/share/roundcubemail/public_html/assets/skins/<name>/

Let's start

  1. Duplicate the existing chameleon skin
    cd /usr/share/roundcubemail
    cp -a skins/chameleon skins/mytheme
    ​cp -a public_html/assets/skins/chameleon public_html/assets/skins/mytheme

     

  2. Give your theme a name

    skins/mytheme/meta.json

    "name": "My Theme", 
  3. Set the 2 most important colors (main color + link color)

    public_html/assets/skins/colors.less

    @main-color:  #C1D72E;
    @link-color:  #ADAA38;

    ​ 

  4. To compile and minifiy your stylesheets you need the node-less compiler. You can either install nodejs and node-less (version >= 1.8.5) from your favorite repo or just use the version from the kolab repository.
    lessc public_html/assets/skins/styles.less > public_html/assets/skins/styles.css

     

  5. Replace the kolab logo (which is used in the upper left corner) with your logo.
    cp ~/mytheme_logo.png public_html/assets/skins/images/kolab_logo.png

     

  6. Change the skin in roundcube's configuration

    /etc/roundcubemail/config.inc.php

    $config['skin'] = 'mytheme';
    ​$config['dont_override'] = array('skin');
  7. Be Happy :-)

The optional part (icons)

For those who think that the color of the icons doesn't match their new color scheme: Here's a quick walk through on how to quickly change the colors using Gimp. And even me (usually using photoshop and yelling why my keyboard shortcuts don't work) managed to get things done fairly quickly (if you know the basics about layers and blending modes).

The majority of icons or images can be broken down into 2 categories:

  • Single-Color Icons with transparent background (example: buttons.png)
  • Single-Color Icons with white background (example: contactpic.png)

Example 1 (buttons.png)

  1. Open the Image
    File > Open > buttons.png

     

  2. For images that have indexed colors switch to RGP mode
    Image > Mode > RGP

     

  3. In the "Layers" Panel click on the [Lock alpha channel] to preserve the transparent background.

     
  4. Grab a pencil with a brushsize of around 10 pixel
     
  5. Pick your favorite color and start painting over the icons you like to change. Pay attention to not draw over the neighbour icons or parts you don't want to change.

    For icons like the thread open/close icons we use a different way (see below).
     

  6. Save the image
    File > Overwrite buttons.png

Example 2 (contactpic.png)

  1. Open the Image
    File > Open > buttons.png

     

  2. For images that have indexed colors switch to RGP mode
    Image > Mode > RGP

     

  3. (Optional) Make a selection on what you want to change (if you only want to change selected icons)
     
  4. Now change everything black/white
    Colors > Hue-Saturation > Saturation = -100
    Colors > Levels > Click [Auto]

    The contactpic should have have black borders and a white background.
     

  5. Create a new transparent layer and change the blending mode to "Lighten Only"
     
  6. Draw on the empty layer with your favorite color and everything which was black before will be filled with your color.

     
  7. Flatten the image again
    Layers > Merge Down

     

  8. Save the image
    ​File > Overwrite contactpic.png

vanmeeuwen's picture
Fri, 2015-02-27 16:22

What is the most original birthday present one can give their spouse? Well, how about a release of your favorite collaboration software suite!

I'm pleased to announce that Kolab 3.4 is available via our OBS repositories immediately!

Please see our Installation Guide for instructions on getting you some Kolab on your favorite platform, and if you already have Kolab running, check out the Upgrade Note from Kolab 3.3 to 3.4, kindly contributed for your benefit by Daniel Hoffend.

Kolab 3.4 comes with a new skin, called chameleon, that is a nice and clean-cut, modern skin for the web client -- courtesy of Kolab Systems, patron of the Kolab Groupware solution.

Two weeks ago, we have had our first release planning meeting on IRC, which has resulted in very fruitful feedback, contributions and most importantly a significant chunk of quality assurance from various participants in the community. A special thanks goes out to Daniel Hoffend and Timotheus Pokorra, who've spent plenty of their spare time on ensuring that Kolab 3.4 is the best it can be right out of the box. One slice of the pie on your right is theirs.

We're definitely going to continue to open up more processes, such as, for example, the Kolab 3.5 Roadmap.

The Kolab 3.4 release also marks the first release with an actual release party - though naturally many people are not able to attend. We're celebrating the Kolab 3.4 release is probably the best quality assured stable release Kolab.org has yet performed


vanmeeuwen's picture
Fri, 2015-02-27 16:22

What is the most original birthday present one can give their spouse? Well, how about a release of your favorite collaboration software suite!

I'm pleased to announce that Kolab 3.4 is available via our OBS repositories immediately!

Please see our Installation Guide for instructions on getting you some Kolab on your favorite platform, and if you already have Kolab running, check out the Upgrade Note from Kolab 3.3 to 3.4, kindly contributed for your benefit by Daniel Hoffend.

Kolab 3.4 comes with a new skin, called chameleon, that is a nice and clean-cut, modern skin for the web client -- courtesy of Kolab Systems, patron of the Kolab Groupware solution.

Two weeks ago, we have had our first release planning meeting on IRC, which has resulted in very fruitful feedback, contributions and most importantly a significant chunk of quality assurance from various participants in the community. A special thanks goes out to Daniel Hoffend and Timotheus Pokorra, who've spent plenty of their spare time on ensuring that Kolab 3.4 is the best it can be right out of the box. One slice of the pie on your right is theirs.

We're definitely going to continue to open up more processes, such as, for example, the Kolab 3.5 Roadmap.

The Kolab 3.4 release also marks the first release with an actual release party - though naturally many people are not able to attend. We're celebrating the Kolab 3.4 release is probably the best quality assured stable release Kolab.org has yet performed