Kolab 16 slow performance vs 3.4


  • Global Moderator

    We're not experiencing these problems ourselves, and our setup scripts for virtual machines do precisely what it says on the tin:

    https://cgit.kolab.org/stick/tree/vms/bin/setup-kolab-vm#n14

    Yes, that's 1GB of RAM precisely. As for vCPUs:

    https://cgit.kolab.org/stick/tree/vms/bin/setup-kolab-vm#n415

    That's 2.



  • I did increase my server from 1G to 2G of RAM when I upgraded (from 3.0!)

    My memory use currently looks like this, so I'm currently using 1.5G of RAM

    [root@mail ~]# free -m
                  total        used        free      shared  buff/cache   available
    Mem:           2001        1330         182          96         488         387
    Swap:          1023         227         796
    

    My top memory users are:

    [root@mail ~]# ps aux --sort -rss | head
    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    amavis    6273  0.0 15.5 629920 318108 ?       Ssl  Feb17   0:28 /usr/sbin/clamd -c /etc/clamd.d/amavisd.conf --nofork=yes
    amavis     643  0.0  7.0 408268 145308 ?       S    12:15   0:11 /usr/sbin/amavisd (ch19-avail)
    amavis    5880  0.0  5.6 379756 116636 ?       S    16:00   0:01 /usr/sbin/amavisd (ch2-avail)
    amavis   23207  0.0  5.2 367316 107188 ?       Ss   04:30   0:01 /usr/sbin/amavisd (master)
    root     32011  0.0  3.0 477796 61904 ?        Ssl  Feb20   1:27 /opt/kolab_guam/erts-6.3/bin/beam -Bd -- -root    /opt/kolab_guam -progname kolab_guam -- -home /opt/kolab_guam/ -- -noshell -noinput -boot /opt/kolab_guam/releases/0.7.1/kolab_guam -config /opt/kolab_guam/releases/0.7.1/sys.config -name kolab_guam@127.0.0.1 -setcookie kolab_guam -- foreground
    nobody    2586  0.0  2.6 987300 53772 ?        Ssl  Feb06   0:33 /bin/node /usr/share/manticore/server/app.js
    pdns-re+  1024  0.0  1.6 247520 33148 ?        Ssl  Feb06   4:04 /usr/sbin/pdns_recursor --daemon
    apache    5449  0.0  1.5 631508 32204 ?        S    Feb21   0:01 /usr/sbin/httpd -DFOREGROUND
    apache    5452  0.0  1.5 711100 31432 ?        S    Feb21   0:01 /usr/sbin/httpd -DFOREGROUND
    

    So clamd is the biggest user by far. But new components of manticore node server and guam are not insignificant.


  • Global Moderator

    @chris I reckon you haven't been doing anything special, but have you been actively using the system (mainly, sending/receiving mails with, perhaps, some large attachments)?

    If you have, and you can describe some of what has been the load on the system so far, perhaps we can reproduce the increased memory consumption on one of our VMs by letting it sit for a little while longer (and give it the same / similar load).



  • @kanarip Looking at some stats using pflogsum for yesterday, generally failrly modest use. Only additional software I have is pdns and fail2ban.

    Grand Totals
    ------------
    messages
    393   received
    391   delivered
      0   forwarded
      0   deferred
      0   bounced
    168   rejected (30%)
      0   reject warnings
      0   held
      0   discarded (0%)
        
      19734k  bytes received
      19734k  bytes delivered
      103   senders
       83   sending hosts/domains
       17   recipients
       13   recipient hosts/domains
    

    Memory looks much the same as yesterday, guam use is up to 77760

    [root@mail log]# date
    Wed 24 Feb 13:29:48 GMT 2016
    [root@mail log]# ps aux --sort -rss | head
    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    amavis    6273  0.0 15.5 629920 318060 ?       Ssl  Feb17   0:32 /usr/sbin/clamd -c /etc/clamd.d/amavisd.conf --nofork=yes
    amavis   31935  0.0  6.2 391600 128552 ?       S    11:16   0:06 /usr/sbin/amavisd (ch14-avail)
    amavis   31929  0.0  6.1 387976 125052 ?       S    11:16   0:06 /usr/sbin/amavisd (ch14-avail)
    amavis   24165  0.0  5.2 367316 107188 ?       Ss   05:27   0:01 /usr/sbin/amavisd (master)
    root     32011  0.0  3.7 485988 77760 ?        Ssl  Feb20   2:20 /opt/kolab_guam/erts-6.3/bin/beam -Bd -- -root /opt/kolab_guam -progname kolab_guam -- -home /opt/kolab_guam/ -- -noshell -noinput -boot /opt/kolab_guam/releases/0.7.1/kolab_guam -config /opt/kolab_guam/releases/0.7.1/sys.config -name kolab_guam@127.0.0.1 -setcookie kolab_guam -- foreground
    nobody    2586  0.0  2.5 987300 52520 ?        Ssl  Feb06   0:35 /bin/node /usr/share/manticore/server/app.js
    pdns-re+  1024  0.0  1.6 247520 33368 ?        Ssl  Feb06   4:18 /usr/sbin/pdns_recursor --daemon
    apache    5449  0.0  1.4 631508 30368 ?        S    Feb21   0:01 /usr/sbin/httpd -DFOREGROUND
    apache    5452  0.0  1.3 711100 27944 ?        S    Feb21   0:01 /usr/sbin/httpd -DFOREGROUND


  • @chris Further to this, I've done a bit more digging. Using smem to get "real" memory use and aggregating multiple processes, The following is a summary of my memory:

    Process            Sum of Swap  Sum of PSS  Count of Command
    clamd              5900         313550      1
    imapd -s           0            226632      183
    amavisd (ch15-ava  0            172112      2
    kolabd             113872       133522      14
    httpd              0            87998       11
    guam               0            77210       2
    wallaced           313308       63034       26
    amavisd (master    0            61475       1
    node               28876        51725       1
    mysql              70276        27839       1
    ns-slapd           0            25012       1
    tuned              0            13148       1
    fail2ban           4836         10002       1
    mongod             22364        7989        1
    smem               0            6454        1
    
    others             12984        36336       36
    -----------------------------------------------
    TOTAL              572416       1314038     283
    

    The Main thing that sticks out to me is that I have a big chunk taken up by imapd. Not sure if this is related to my guam problems. Doing a grep through the maillog, I can find 11 users logging in on imap.



  • @chris As way of followup, I've been having a look. I can only see 11 users logging in via imap in total, which makes the 183 imapd instances feel very high. Looking at the oldest connection, it me checking my mailbox from my home laptop on Feb 20th. I know this machine is currently sitting at home closed.

    I can follow the connection to guam, but can't see any way of following the connection in the guam logs. But I do know the IP I would have been connecting from and can't find any current connections coming in from that IP.

    So I think that something is going wrong in guam, and dead connection are getting kept open, when they should be closed?



  • @kanarip
    I still have same problem every day ( im only user). Login takes 30-60 seconds.

    I try bypass guam setting in roundcube port to straight to 9993 , it didn't help.

    Do you development guys have any machine that have only 1 user and do not use/login like 6 hours like me? I really don't get hold of this. i have 4GB mem in VM but after no one hasn't use server like 6 hours, i login and i get loads like 4-6.00 mostly because of swap process which is killer for VM.. (swap is 8GB), and after waiting for 2 minutes, load goes down and everything starts to work as they should.

    In same "mainserver" are running 2 medium loaded apache/mysql servers without problems same time...

    And 3.4 version in same vsphere "mainserver" doesn't have any problem. I can login to that when Kolab16 is just "clocking" !?!

    Well now i installed new fresh Kolab16 + CentOS7 with SSD disks to another server, and it works little faster than these, let's see tomorrow morning how long it takes to login.....


  • Global Moderator

    Do you have enough entropy?



  • @Kimmo I wonder if stuff is getting moved out onto swap if it's not getting used?

    I would be tempted to install smem and work out what's going on/using your memory? Also free -m will give you an idea. If you have 4G then you might be able to turn swap off entirely. Otherwise you can set the swappiness value:

    echo 1 > /proc/sys/vm/swappiness

    (a value of 1 is mimimum without quite completly disabling swap)

    This can be made permament by adding vm.swappiness = 1 to /etc/sysctl.conf



  • @kanarip said:

    Do you have enough entropy?

    I don't understand what you mean with entropy, but anyway just let everyone know, with SSD disks problem has gone away now.

    I still have that slow VM running if someone ones to play with it, there is nothing in production. I think it was related to swapping, because in morning you could't even login server via SSH also. And load was up.



  • And not sure if this should be new subject:

    I'm new with Kolab , but familiar with webservers, CentOS 7 is running php 5.4 which doesn't have php-xcache integrated , like php 5.5 does, question goes, does Kolab benefit anything by installing php-xcache or other php cache ?


  • Global Moderator

    @Kimmo What you have installed as part of Kolab are web applications just like you would have them installed if it were without Kolab. There's no Kolab-specific extras or conditions for as far as the principle category of "a web server serving a web application written in PHP" goes.


Log in to reply