How to set the maximum amount of memory what a Linux process can use

If I want to let a python process to use maximum the 90% of the total memory available, I'm using this command:

ulimit -Sv $((`python3 -c'import psutil;print(psutil.phymem_use().total)'`*9/10/1024)); python3 mystuff.py

With ulimit you can set the limit parameter for the current session.

To dump the corrent config, simple run the following command:

>ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 63785
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 63785
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

You can change the values with the -S and use the param argument from the brackets. 

In this case we set the virtual memory parameter with -v

To get the total available memory, you have multiple options:

1. Option
   cat /proc/meminfo | awk '/MemTotal/ {print $2}'
2. Option
   free | awk '/^Mem:/ {print $2}'
3. Option
   python3 -c 'import psutil; print(psutil.phymem_use().total)

Please keep in mind, you need to change this value to KB, 

Examples in a 16 GB RAM machine:

>cat /proc/meminfo | awk '/MemTotal/ {print $2}'

16351304

>free | awk '/^Mem:/ {print $2}'

16351304

>python3 -c 'import psutil; print(psutil.phymem_usage().total)'

16743735296

As you can see, in the last option, it is in bytes, so need to divide with 1024. 

If your application reaches the limit, it will get a signal 6 - Abort signal from the Linux.

Notes:

• You can use bc to calculate the value to set, but in this example it was not available, so simple bash calculation was used
• You can calculate the value with python, but this example keeps the chance to replace that part.

Scripting LVM Snapshot Backups

In this article I will show you how you can simply backup an LV.
In this example we are going to script a mail server's mailbox store LVM backup.

The LVM backup script

#!/bin/bash
SIZE=1G            # Size of a snapshor. Make sure, there is space for all 7 snapshot
VGNAME=/dev/vg00    # /dev/VGNAME
LVTOBACKUP=www        # only the name of LV
DAY=`date +%a`
SNAPSHOTNAME="$LVTOBACKUP-$DAY"

# ---------------------- Do not edit under this line  -------------------

SCRIPTNAME=$0
SENDMONITORINGDATA=true
FAILOCCURED=0
# Detailed log to syslog
#exec > >(logger -t "$SCRIPTNAME" -p local3.info ) 2> >(logger -t "$SCRIPTNAME" -p local3.info)
set -x
ZBXSENDER=`which zabbix_sender`
if [ $? -ne 0 ]
then
    SENDMONITORINGDATA=false
    echo "Not sending data to Zabbix"
fi
ZBXCONFIG=`find /etc/zabbix/ -maxdepth 1  -name '*.conf' | tail -n 1`
sendMonitoringData(){
    if [ "$SENDMONITORINGDATA" = true ]
    then
        echo "Sending monitoring data: $SCRIPTNAME - $1"
        $ZBXSENDER --config $ZBXCONFIG --key scripterror --value $1
        $ZBXSENDER --config $ZBXCONFIG --key scriptname --value $SCRIPTNAME
    fi
}
checkResult(){
    EXITSTATUS="$1"
    shift
    MSG="$*"
    if [ $EXITSTATUS -eq 0 ]
    then
        echo "$MSG - OK"
    else
        echo "$MSG - FAIL"
        sendMonitoringData $EXITSTATUS
        FAILOCCURED=1
    fi
}
lvs "$VGNAME/$SNAPSHOTNAME" 2>&1 > /dev/null
RETCODE=$?
checkResult $RETCODE "Failed to check $VGNAME/$SNAPSHOTNAME"
if [ $RETCODE -eq 0 ]
then
    lvremove -f "$VGNAME/$SNAPSHOTNAME"
    checkResult $? "Failed to remove $VGNAME/$SNAPSHOTNAME"
fi
sync
lvcreate -L "$SIZE" -s -n "$SNAPSHOTNAME" "$VGNAME/$LVTOBACKUP"
checkResult $?  "Failed to create $VGNAME/$SNAPSHOTNAME"
if [ $FAILOCCURED -eq 0 ]
then
    sendMonitoringData 0
else
    echo "Due to failure no more alert to monitoring"
fi

Explanation

It seems a bit complicated script, let me explain:

LVM part

The basic backup script looks like this
 #!/bin/bash
SIZE=1G            # Size of a snapshot. Make sure, there is space for all 7 snapshot
VGNAME=/dev/vg00    # /dev/VGNAME
LVTOBACKUP=mail        # only the name of LV
DAY=`date +%a`
SNAPSHOTNAME="$LVTOBACKUP-$DAY"
# ---- 
lvs "$VGNAME/$SNAPSHOTNAME" 2>&1 > /dev/null
RETCODE=$?
if [ $RETCODE -eq 0 ]
then
    lvremove -f "$VGNAME/$SNAPSHOTNAME"
fi
sync
lvcreate -L "$SIZE" -s -n "$SNAPSHOTNAME" "$VGNAME/$LVTOBACKUP"

For LVM snapshot you need to define the snapshot Logical Volume size.
It depends on the application witch data you are backing up.
After the initialization period you can check the used amount of Snapshot space like this:
# lvs
  LV       VG   Attr   LSize  Origin Snap%  Move Log Copy%  Convert
  mysql    vg00 -wi-ao  5,00g                                     
  swap     vg00 -wi-ao  4,00g                                     
  mail     vg00 owi-ao 30,00g                                     
  mail-Sun vg00 swi-a-  1,00g mail    0,70
You can see that, it uses the 0.7% of the configured size. I could configure smaller size...

I'm using daily backup, it creates a new LV snapshot every day. The name is came from the original Logical Volume's name and the sort version of day of week.

The script first check whether the last weeks LVM snapshot is available. If exist, then first it removes that.
Then the last lines are coming:
sync
lvcreate -L "$SIZE" -s -n "$SNAPSHOTNAME" "$VGNAME/$LVTOBACKUP"
Sync is important, because we need the data on disk, not in cache ;-)

Monitoring

You can see that, there are more lines connecting to monitoring than for the backup.
To make sure, this script can be used on different distributions and different versions, the path are not burned into the code.
Every step of the backup process is monitored and reported to Zabbix and of course to syslog.

Please feel free to use it.

All feedback are welcome.

Cheap car GPS tracking with OpenWRT I. The basics

My goal to assembly a reliable GPS tracking system by cheap components with OpenWRT.
I would like to install it into my car. It would record the GPS data (latitude, longitude, speed) in 10 seconds and uploads to a server when it hase WIFI connection.
In my firs run it would scan the available wifi connections frequently and try to connect and upload the recorded data trough non-secure access points.
I would like to make a simple web page where I can follow the car's movement on the google map.


I'm good with it. It is running in pilot mode.

The parts of the project:

• TP-LINK  TL-MR3020 router running OpenWRT
• USB Hub to share to multiple the USB connectors
• Flash drive to store recorded data
• GPS receiver  to collect data
• Power bank to work in case the car's battery disconnected
• PCA with power connectors, fuse, DC-DC converter, etc
• Webserver listening on the Internet

Here you can see a picture from the current status:

First I present the components then I write the "why" in the following posts.

TP-LINK  TL-MR3020 
This is a cheap, portable router which can share your 3G mobile connection as a wifi access point. It has one piece of USB connector, one peace 10/100Mb/s LAN connector and built-in 5DBm WIFI antenna.
An advantage of this device is it can run by USB power (5V, max 500mA).
I've replaced the original firware to OpenWRT by this tutorial.
It was very simply, works out of the box.

USB Hub
It is a cheap one, but can work with external power. The external power is important because the GPS receiver use approx 300mA. Without the extra power the Linux wrote I/O error into the log...
In the first step I use one connector for the USBdrive and another for the GPS receiver. (I plan to complete the system with a 3G stick to send the recorded data to the server immediately)

Flash drive
The router have only 4MB Flash storage, therefore it needs more space to store the recorded data. I've used a flashdrive 2GB space.

GPS receiver
During the period I've collected data to my project I found this and this article. They used the Globalsat BU-353 GPS Reciever. It is the most expensive part of the project, but it works pretty good.

Power bank
I've chosen a 5V and 2600mAh  power bank which can charge from USB connector. Optimally it will not works because I will connect the stuff onto constant current source in the car. One of my expectation is it have to works for a while when  external power is not available. It has a normal USB connector for power input and a micro USB for power output. Both connectors are connected constantly.

PCA
As you can see in the picture I soldered some components onto a project board. The most important component is the DC-DC converter which makes stable 5V  from the car's 12-14V.

Webserver
Basically the system store the collected data on the flash drive but in case it has connection to the internet it synchronize the local data with the remote database.
I would like to make a simple web page where i can follow the car's movement on the google map.
It would inform the visitor about the car's current position, the last "hearthbeat" when the car logged in the page, etc


I will coming soon with the presentation of the components