Since January 2010, Google docs has allowed you to store any type of file, even arbitrary binary files. However, there are a couple of gotchas: one cannot upload files greater than 1GB in size, and you may want to encrypt your files so that not just anyone can read them (for instance server backup files).

The two bash scripts below provide a solution for the above. I call them the ‘mince’ scripts ‘cos they slice and dice your files and hopefully you’ll get hamburgers back at the end of the day. These scripts depend on you having a fairly new version of bash on your unix-like system, the ‘split’ utility and gnupg (GPG) which is used for the encryption/decryption. If you’re unsure of GPG, a good getting started guide can be found here.

It must be said that google docs is (in my opinion) currently not the best way to store your files in the cloud. In fact, I wrote another blog post describing the “google storage” options in greater depth.

The encrypt&split script is mince.sh and it takes two parameters, the first one a directory or archive, the second the email address for an already imported public key:

#!/bin/bash
# gpg encrypt archive and split into chunks
# $1 specifies base directory or compressed archive to encrypt.
# $2 is the recipients public key, eg. 'myfriend@his.isp.net'
set -e


CHUNK_SIZE=1000000000 #1000000000==1GB (not 1GiB!)
SCRATCH_DIR=~/scratch_space
TAR_REGEX='\.tar'


usage() {
    echo "ERROR: "
    echo " $*"
    echo "USAGE: "
    echo " mince.sh [DIRECTORY|ARCHIVE] PUBLIC_KEY_NAME"
    echo "EXAMPLE: "
    echo " ./mince.sh directory myfriend@her.isp.net"
    echo "FURTHER COMMENTS: "
    echo " if an ARCHIVE is supplied instead of a directory, it must have a name like file.tar or file.tar.gz or file.tar.bz2 "
}


#check parameters entered are valid
[ $# -ne 2 ] && usage "Two parameters are required." && exit 1
if [ ! -d "$1" ] && [[ ! $1 =~ $TAR_REGEX ]]; then
    usage "$1 is not a directory or tar/tar.gz/tar.bz2 archive."
    exit 1
fi


#if 1st parameter is a directory, then tar it up in the scratch space
if [ -d "$1" ]; then
    absolute="`cd $1; pwd` "
    mkdir -p $SCRATCH_DIR
    cd $SCRATCH_DIR
    nameonly=${absolute##*/}
    nameonly=${nameonly/ /} #remove trailing spaces
    tar -cf $nameonly.tar $absolute
    arch="${SCRATCH_DIR}/${nameonly}.tar"
    created=true
    echo "Created temporary archive $arch"
else
    arch="`readlink -f $1`"
    created=false
    echo "Working with existing archive $arch"
fi


#call for GPG encryption and compression of the archive
target="${arch##*/}.gpg"
name=${target%\.gpg}
mkdir -p $SCRATCH_DIR
cd $SCRATCH_DIR
echo "Commencing GPG encryption, please be patient"
gpg --bzip2-compress-level 6 --compress-algo bzip2 --output $target --encrypt --recipient $2 $arch


#split .gpg file into chunks of size CHUNK_SIZE
outdir="${SCRATCH_DIR}/output"
mkdir -p "$outdir"
mkdir -p "$outdir/$name"
cd $outdir/$name && rm -f $name*
echo "Splitting files"
split -b $CHUNK_SIZE "${SCRATCH_DIR}/$target"
for x in *
do
  mv $x "../${name}__$x"
done


#clean up - remove .gpg and temporary archive and temporary directory
cd $SCRATCH_DIR
rmdir "$outdir/$name"
rm $target
if [ $created == true ]; then
    echo "Removing temporary archive $arch"
    rm $arch
fi

echo "All file splits produced placed in $outdir"

Download link for mince.sh

The bash script to reconstitute the file is called unmince.sh and takes one parameter – the name of the first file downloaded from google docs:

#!/bin/bash
# reassemble an archive from chunks of a file that have been gpg-encrypted
# $1 specifies the first file produced from the mincing process, eg, file__xaa
set -e


SCRATCH_DIR=~/scratch_space
REGEX='__xaa'


usage() {
    echo "ERROR: "
    echo " $*"
    echo "USAGE: "
    echo " unmince.sh file$REGEX"
    echo "WHERE: "
    echo " file$REGEX is the first file produced by the mince script"
}


#check parameters
[ $# -ne 1 ] && usage "Only one parameter is required" && exit 1
if [ -d "$1" ] || [[ ! $1 =~ '_xaa' ]]; then
    usage "$1 cannot be a directory and must end in _xaa."
    exit 1
fi


#combine all chunks of the file
sourcepath="`readlink -f $1`"
pathonly="`dirname $sourcepath`"


just=${sourcepath##*/}
basenam=${just%$REGEX}
indir="${SCRATCH_DIR}/reconstituted"
mkdir -p $indir
cd $indir
[ -e $indir/combined.gpg ] && rm $indir/combined.gpg
for x in $pathonly/$basenam*
do
    cat $x >> combined.gpg
done


#decrypt the .gpg file
echo "Commencing GPG decryption, please be patient"
gpg --output $basenam --decrypt combined.gpg


#tidy up - remove the gpg file
rm combined.gpg


echo "The reconstituted archive $indir/$basenam was created"

Download link for unmince.sh

Since I might still tinker and improved these scripts, to get the newest version of these files take a look at my github repo at http://github.com/eyesonly/kenwood (Named after the Kenwood Chef, a famous mincer!)