# Getting the Data Out

Do Backups!

First of all, it is your responsibility to store your experimental data in several places. Ideally, you want the data to be in several computers that are in different buildings. You also want to have archives (snapshots) of the results that cannot easily be modified by anyone (e.g. cloud backup solutions).

File System

The Ethoscope system saves by default the data in two custom directories:

  • /ethoscope_results -- the results of tracking experiments
  • /ethoscope_videos -- raw video files (when recording)

Both directory have the same substructure:

<machine_id>/<machine_name>/<datetime>/<file(s)>

Where:

  • <machine_id> is a unique identifier for the machine (e.g. 0001eeee10184bb39b0754e75cef7900)
  • <machine_name> is a human friendly name (e.g. ETHOSCOPE_001)
  • <datetime> is a date and time formated as YYYY-MM-DD_HH-MM-SS (e.g. 2016-03-21_20-55-10)

Then, each file inside the directory will be uniquely and unambiguously named:

For instance, we could have a file named 2016-03-21_20-55-10_0001eeee10184bb39b0754e75cef7900.db. This redundancy means that if this file is sent to a collaborator, it preserves its uniqueness (plus we have some metadata in the file).

Example of Backups Script

This is the sort of scripts we use in the lab. You are welcome to take it and modify it. You will have to change things according to what sort of remote computer you save your data on (in the lab, we have our own NAS).

#!/bin/bash
# we ensure the backup service is stopped during backup, so we do not keep modifying files
# during backup
backup_service=ethoscope_backup
result_dir=ethoscope_results/
target_prefix=/mnt/nas/auto_generated_data/
(
## we use flock to ensure no concurent backups
# see http://stackoverflow.com/questions/169964/how-to-prevent-a-script-from-running-simultaneously
# Wait for lock on /var/lock/.myscript.exclusivelock (fd 200) for 10 seconds
flock -x -w 10 200 || exit 1
systemctl stop $backup_service &&
sleep 30 &&
python2 /root/incremental_size_upload.py -s /$result_dir/ -d $target_prefix/$result_dir/
Rscript /root/make_db_index.R
cp /$result_dir/index.txt $target_prefix/$result_dir/index.txt
sync
systemctl start $backup_service

) 200>/var/lock/.$backup_service.exclusivelock

Complementary scripts (cron table, fstab) are available on our labscript repository.

results matching ""

    No results matching ""