Zero-1-Earth!

To content | To menu | To search

Bash scripting

Entries feed - Comments feed

Monday, September 3 2012

Hidden behavior in a bash script

Today I stumbled upon the following challenge: I have some scripts that are supposed to receive 0 or 1 parameter. If the script receives 0 parameter, it returns a "usage" message and exit. I needed to change my scripts so that they can return a public interface specification without introducing new parameters (I have tenth of similar script and a mechanism calling them).

The basic idea is to set a global variable $SCRIPT_PUBLICINTERFACE to change the script behavior, so that an information-collecting script can get information from the scripts, without having to pass any parameter. This approach avoid changing significantly the structure of my scripts.

Here is my implementation for the scripts, followed by the "function" library.

#!/bin/bash
NPARAM=$#
# load library
source functions

function usageMessage(){
    echo "Script stoped with error: "$1
    echo "Script usage:"
    echo "  please set/unset variable SCRIPT_PUBLICINTERFACE to show/hidde publicInterface"
    echo "  example: "
    echo "   export SCRIPT_PUBLICINTERFACE=1"
    echo "   ./script.sh"
    echo "   unset SCRIPT_PUBLICINTERFACE"
    echo "   ./script.sh"
    exit 1
}

echo "The script is running and calling the usage function"
if [ $NPARAM -eq 0 ]; then
       usage "calling the usage function"
fi

Hereafter the 'function' library:

function usageMessage(){
         # default usageMessage. Overridden by user function
         echo "Script ${0##*/} received the error message:" $1
         echo "Please define a specific usageMessage function for this script."
}

function usage(){
        if [ -z "$SCRIPT_PUBLICINTERFACE" ]; then
            usageMessage "$1"
            return 0
         else
            printPublicInterface
            exit 1
         fi
}

function printPublicInterface(){
cat << EOF
<?xml version="1.0">
<dict>
...write here your xml file content...
</dict>
EOF

The user defines a "userMessage' function which contains the text to display in case of error, or if the script is called with no parameter. In case that the user forgets declaring such a function, there is a default definition in the library. The usage function checks if the global variable is set, and calls usageMessage or the function to output the public interface.

Sunday, May 2 2010

testing files integrity across an archive of data

I currently cooking a system, largely based on bash, to collect remote sensing data from the web. Since I’m using my personal ADSL connection, I can expect to have many corrupted downloads. I made a little script to check files integrity, and trigger again their download.

First, how to know if a file is corrupted or not? Two technics: either your collect the error code from you download software (ftp here) and log them somewhere to try again or you are able to assess the integrity of a file simply by scanning it. Let’s consider the second case.

The central problem is that you may have various kind of file, so there is not a method per kind of file to check. For example, if we download modis files we have an xml and an hdf file. For a given file, the script must first guess the type of file, then choose an ad-hoc function for checking this file. We assume here that the file type is found by considering the file extension.

To get the file extension from a full file path, simply remove the file path and keep what you find after the last ‘.’, which is a job for sed:

echo $fileFullPath | sed 's/^.*\///' | sed 's/^.*\.//' | tr '[:upper:]' '[:lower:]'

Command

sed 's/^.*\///'

removes the file path, by substituting repetitions (*) of any characters (.*) with nothing, starting from the string beginning (^). Then anything up to the last point is removed (think to escape the point \.).
Note that the regular expression of your system may give different result: give it some tries.

Now we need to call the appropriate test routine as a function of the detected file extension: a simple case function will do this job. Finally, let’s wrap-up everything in a single function (selectFunc): call it with a file name, and it returns the test function to call.

function selectFunc(){
# receives a file name and decides which integrity test function corresponds
if [ $# -ne 1 ]; then
echo "selectFunc is missing a single parameter. Exit."
return -1
fi
selector=$(echo $1 | sed 's/^.*\///' | sed 's/^.*\.//' | tr '[:upper:]' '[:lower:]')
case $selector in
xml) selectFunc='doTestXML';;
hdf) selectFunc='doTestImg';;
tif | tiff ) selectFunc='doTestImg';;
esac
# return the selected function
echo $selectFunc
}

We can see that there is a pending problem with respect to file without an extension, like ENVI native file format (it does not require an extension, only a companion text file). To improve this situation, you can either force an extension to this kind of files (like .bil or .bsq for ENVI files), or handle the case of missing extension with additional tests. For example, one could image to call gdalinfo in this case.

Now we just have to write some test functions.

xml files are rather easy to test. Your OS should have a function for that. For Linux, consider xml Starlet, which command line is

xml -val $file

For images, you should be able to test most of them with gdalinfo.
The function return 0 is everything was ok, 1 else. Actually, the tests functions return the return code of xml starlet and gdalinfo. If you use other kind of test, you may need to translate their exit codes.
At the end, we’ve got something like:

function doTestXML(){
if [ $# -ne 1 ]; then
echo "doTestXML is missing a single parameter. Exit."
exit -1
fi
xmlwf $1 >& /dev/null
return $?
}

function doTestImg(){
if [ $# -ne 1 ]; then
echo "doTestXML is missing a single parameter. Exit."
exit -1
fi
gdalinfo $1 >& /dev/null
return $?
}

Note we sent to null any output of the function and take care only of the return code ($?).

Now, all to use this code:
get the name of the test function to call:

myTest=$(selectFunc $file)

and call the script:

$myTest $file

A functional copy of the code is found there.


Tuesday, April 20 2010

Removing 10 first lines of a text file

How to remove the 10 first lines of a text file? Easy job with sed (Stream EDitor):

sed '1,10d' myFile

Example: removing 10 first lines of all text files in a directory.

for file in *.txt
do
sed '1,10d' $file > output_dir/new_${file}
done

Further readings: 

Tuesday, April 6 2010

ntfs hard drive on Snow Leopard

Note: this post proposes a trick at your own risks!

I’m willing to backup-up 1.5Tb of data I have on my office computer, and use them on my Mac OS X (Snow Leopard) computer at home. By default, Mac OS X can format, read and write Fat32 (MS-DOS) to share an hard drive with a Windows PC. Unfortunately, MS-DOS format is a bit outdated, and would not support partition larger than 1Gb, which would force me to make two partitions on my drive, while I would prefer only one.

Although it is not visible by default, Mac OS X Snow Leopard (10.6) support NTFS. To activate the support, you must declare your new drive in /etc/fstab (the file used by Unix/Linux systems to mount devices).

You must first get some infos: plug in your hard drive. In my case, the drive mounts with the name “Elements”.

open a terminal and type the following command:

diskutil info /Volumes/Elements

you’ll see more information about the drive, check it is NTFS formatted.

By default, file /etc/fstab is not created on Mac OS X. If it already exists, make a copy first:

sudo cp /etc/fstab /etc/fstab_org

sudo command will prompt for the administrator password (to run the copy command in directory /etc where you should not have rights to write as a normal user).

edit it (you can use nano to edit the file, or any other plain text editor):

sudo nano /etc/fstab

and add the line:

LABEL=Elements none ntfs rw

(replace Elements with the label name of your hard drive).

Reboot the computer to force reading /etc/fstab and mounting your drive as a read/write NTFS drive.

Job done!

For perfectionists: if your drive shows a UUID when you enter command diskutil info /Volumes/Elements, then you can edit your /etc/fstab with

UUID=XXXXXXX none ntfs rw

where XXXXXXX is the UUID number shown by diskutil. This should allow recognizing the drive even if you change its label name.

Thursday, December 25 2008

DIY backup system

Continue reading...