Kolab 16 slow performance vs 3.4
I'm new to Kolab but familiar with Roundcube. Couple a weeks ago i installed fresh vmware CentOS7 and Kolab 3.4 , but when testing 3.4 i notice that Kolab 16 came out. So i installed that to another vmware VM. Both in same server with same memory/cpu/disks.
i configured multidomain for both and they are working, but Kolab 16 is having strange performance issues.
In morning i try to load login page, it takes like 1 minute to login, same when try to read inbox messages after that, takes 10sec to open message. Watching server load it tells same, just like missing I/O from disk, every processes are slow.
BUT in same time with kolab 3.4 no problems, little slower that our other plain Roundcube webmails, but thats understood because they don't have all plugins. Both show messages 1-2 second after clicking. But every click i do in Kolab 16 shows more load in "top" compare to 3.4 in same hardware, in same server.
Most strange thing is that after playing around sometime with kolab 16 , it starts to works as fast ?
Rebooting Kolab 16 machine also seems to help, not 100% sure for that, have to test it more.
Something to do with guam that 3.4 doesn't have ? Or some cache thing ?
Any ideas how to search problem, remove 1 plugin at time from Roundcube and every morning test if problem goes away ? What log's to follow ?
Just let everyone know it was plain memory problem, VM was swapping.
But Kolab 3.4 run fine with 1GB memory (1 test user), and it seems that Kolab 16 needs at least 4GB memory to run 1 user (i had 3GB when having problems).
These daemons doesn't exist in Kolab 3.4, does they eat all that memory ?
/usr/bin/mongod --quiet -f /etc/mongod.conf run
Well it was not fixed. Doesn't anyone else have this same problem ?
Both my 3.4 and Kolab 16 vmware servers are with 1 user 1 domain, with allmoust nothing emails coming thru. Kolab 16 getting huge load and same time 3.4 having 0.05 load with same ??
When i login in morning it takes like 1 minute with Kolab 16. More i use kolab pages, they start to answer quicker, I don't understand ?
With 3.4 login and roundcube messages goes fast......
I can give developers login id:s to try ?
We're not experiencing these problems ourselves, and our setup scripts for virtual machines do precisely what it says on the tin:
Yes, that's 1GB of RAM precisely. As for vCPUs:
I did increase my server from 1G to 2G of RAM when I upgraded (from 3.0!)
My memory use currently looks like this, so I'm currently using 1.5G of RAM
[root@mail ~]# free -m total used free shared buff/cache available Mem: 2001 1330 182 96 488 387 Swap: 1023 227 796
My top memory users are:
[root@mail ~]# ps aux --sort -rss | head USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND amavis 6273 0.0 15.5 629920 318108 ? Ssl Feb17 0:28 /usr/sbin/clamd -c /etc/clamd.d/amavisd.conf --nofork=yes amavis 643 0.0 7.0 408268 145308 ? S 12:15 0:11 /usr/sbin/amavisd (ch19-avail) amavis 5880 0.0 5.6 379756 116636 ? S 16:00 0:01 /usr/sbin/amavisd (ch2-avail) amavis 23207 0.0 5.2 367316 107188 ? Ss 04:30 0:01 /usr/sbin/amavisd (master) root 32011 0.0 3.0 477796 61904 ? Ssl Feb20 1:27 /opt/kolab_guam/erts-6.3/bin/beam -Bd -- -root /opt/kolab_guam -progname kolab_guam -- -home /opt/kolab_guam/ -- -noshell -noinput -boot /opt/kolab_guam/releases/0.7.1/kolab_guam -config /opt/kolab_guam/releases/0.7.1/sys.config -name email@example.com -setcookie kolab_guam -- foreground nobody 2586 0.0 2.6 987300 53772 ? Ssl Feb06 0:33 /bin/node /usr/share/manticore/server/app.js pdns-re+ 1024 0.0 1.6 247520 33148 ? Ssl Feb06 4:04 /usr/sbin/pdns_recursor --daemon apache 5449 0.0 1.5 631508 32204 ? S Feb21 0:01 /usr/sbin/httpd -DFOREGROUND apache 5452 0.0 1.5 711100 31432 ? S Feb21 0:01 /usr/sbin/httpd -DFOREGROUND
So clamd is the biggest user by far. But new components of manticore node server and guam are not insignificant.
@chris I reckon you haven't been doing anything special, but have you been actively using the system (mainly, sending/receiving mails with, perhaps, some large attachments)?
If you have, and you can describe some of what has been the load on the system so far, perhaps we can reproduce the increased memory consumption on one of our VMs by letting it sit for a little while longer (and give it the same / similar load).
@kanarip Looking at some stats using pflogsum for yesterday, generally failrly modest use. Only additional software I have is pdns and fail2ban.
Grand Totals ------------ messages 393 received 391 delivered 0 forwarded 0 deferred 0 bounced 168 rejected (30%) 0 reject warnings 0 held 0 discarded (0%) 19734k bytes received 19734k bytes delivered 103 senders 83 sending hosts/domains 17 recipients 13 recipient hosts/domains
Memory looks much the same as yesterday, guam use is up to 77760
[root@mail log]# date Wed 24 Feb 13:29:48 GMT 2016 [root@mail log]# ps aux --sort -rss | head USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND amavis 6273 0.0 15.5 629920 318060 ? Ssl Feb17 0:32 /usr/sbin/clamd -c /etc/clamd.d/amavisd.conf --nofork=yes amavis 31935 0.0 6.2 391600 128552 ? S 11:16 0:06 /usr/sbin/amavisd (ch14-avail) amavis 31929 0.0 6.1 387976 125052 ? S 11:16 0:06 /usr/sbin/amavisd (ch14-avail) amavis 24165 0.0 5.2 367316 107188 ? Ss 05:27 0:01 /usr/sbin/amavisd (master) root 32011 0.0 3.7 485988 77760 ? Ssl Feb20 2:20 /opt/kolab_guam/erts-6.3/bin/beam -Bd -- -root /opt/kolab_guam -progname kolab_guam -- -home /opt/kolab_guam/ -- -noshell -noinput -boot /opt/kolab_guam/releases/0.7.1/kolab_guam -config /opt/kolab_guam/releases/0.7.1/sys.config -name firstname.lastname@example.org -setcookie kolab_guam -- foreground nobody 2586 0.0 2.5 987300 52520 ? Ssl Feb06 0:35 /bin/node /usr/share/manticore/server/app.js pdns-re+ 1024 0.0 1.6 247520 33368 ? Ssl Feb06 4:18 /usr/sbin/pdns_recursor --daemon apache 5449 0.0 1.4 631508 30368 ? S Feb21 0:01 /usr/sbin/httpd -DFOREGROUND apache 5452 0.0 1.3 711100 27944 ? S Feb21 0:01 /usr/sbin/httpd -DFOREGROUND
@chris Further to this, I've done a bit more digging. Using smem to get "real" memory use and aggregating multiple processes, The following is a summary of my memory:
Process Sum of Swap Sum of PSS Count of Command clamd 5900 313550 1 imapd -s 0 226632 183 amavisd (ch15-ava 0 172112 2 kolabd 113872 133522 14 httpd 0 87998 11 guam 0 77210 2 wallaced 313308 63034 26 amavisd (master 0 61475 1 node 28876 51725 1 mysql 70276 27839 1 ns-slapd 0 25012 1 tuned 0 13148 1 fail2ban 4836 10002 1 mongod 22364 7989 1 smem 0 6454 1 others 12984 36336 36 ----------------------------------------------- TOTAL 572416 1314038 283
The Main thing that sticks out to me is that I have a big chunk taken up by imapd. Not sure if this is related to my guam problems. Doing a grep through the maillog, I can find 11 users logging in on imap.
@chris As way of followup, I've been having a look. I can only see 11 users logging in via imap in total, which makes the 183 imapd instances feel very high. Looking at the oldest connection, it me checking my mailbox from my home laptop on Feb 20th. I know this machine is currently sitting at home closed.
I can follow the connection to guam, but can't see any way of following the connection in the guam logs. But I do know the IP I would have been connecting from and can't find any current connections coming in from that IP.
So I think that something is going wrong in guam, and dead connection are getting kept open, when they should be closed?
I still have same problem every day ( im only user). Login takes 30-60 seconds.
I try bypass guam setting in roundcube port to straight to 9993 , it didn't help.
Do you development guys have any machine that have only 1 user and do not use/login like 6 hours like me? I really don't get hold of this. i have 4GB mem in VM but after no one hasn't use server like 6 hours, i login and i get loads like 4-6.00 mostly because of swap process which is killer for VM.. (swap is 8GB), and after waiting for 2 minutes, load goes down and everything starts to work as they should.
In same "mainserver" are running 2 medium loaded apache/mysql servers without problems same time...
And 3.4 version in same vsphere "mainserver" doesn't have any problem. I can login to that when Kolab16 is just "clocking" !?!
Well now i installed new fresh Kolab16 + CentOS7 with SSD disks to another server, and it works little faster than these, let's see tomorrow morning how long it takes to login.....
Do you have enough entropy?
@Kimmo I wonder if stuff is getting moved out onto swap if it's not getting used?
I would be tempted to install smem and work out what's going on/using your memory? Also free -m will give you an idea. If you have 4G then you might be able to turn swap off entirely. Otherwise you can set the swappiness value:
echo 1 > /proc/sys/vm/swappiness
(a value of 1 is mimimum without quite completly disabling swap)
This can be made permament by adding vm.swappiness = 1 to /etc/sysctl.conf
Do you have enough entropy?
I don't understand what you mean with entropy, but anyway just let everyone know, with SSD disks problem has gone away now.
I still have that slow VM running if someone ones to play with it, there is nothing in production. I think it was related to swapping, because in morning you could't even login server via SSH also. And load was up.
And not sure if this should be new subject:
I'm new with Kolab , but familiar with webservers, CentOS 7 is running php 5.4 which doesn't have php-xcache integrated , like php 5.5 does, question goes, does Kolab benefit anything by installing php-xcache or other php cache ?
@Kimmo What you have installed as part of Kolab are web applications just like you would have them installed if it were without Kolab. There's no Kolab-specific extras or conditions for as far as the principle category of "a web server serving a web application written in PHP" goes.