2016. május 27., péntek

How to set the maximum amount of memory what a Linux process can use


If I want to let a python process to use maximum the 90% of the total memory available, I'm using this command:

ulimit -Sv $((`python3 -c'import psutil;print(psutil.phymem_use().total)'`*9/10/1024)); python3 mystuff.py

With ulimit you can set the limit parameter for the current session.

To dump the corrent config, simple run the following command:

>ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 63785
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 63785
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited


You can change the values with the -S and use the param argument from the brackets. 
In this case we set the virtual memory parameter with -v

To get the total available memory, you have multiple options:
  1. Option
    cat /proc/meminfo | awk '/MemTotal/ {print $2}'
  2. Option
    free | awk '/^Mem:/ {print $2}'
  3. Option
    python3 -c 'import psutil; print(psutil.phymem_use().total)
Please keep in mind, you need to change this value to KB, 
Examples in a 16 GB RAM machine:
>cat /proc/meminfo | awk '/MemTotal/ {print $2}'
16351304
>free | awk '/^Mem:/ {print $2}'
16351304
>python3 -c 'import psutil; print(psutil.phymem_usage().total)'
16743735296

As you can see, in the last option, it is in bytes, so need to divide with 1024. 

If your application reaches the limit, it will get a signal 6 - Abort signal from the Linux.

Notes:
  • You can use bc to calculate the value to set, but in this example it was not available, so simple bash calculation was used
  • You can calculate the value with python, but this example keeps the chance to replace that part.