# Getting the Data Out
First of all, it is your responsibility to store your experimental data in several places. Ideally, you want the data to be in several computers that are in different buildings. You also want to have archives (snapshots) of the results that cannot easily be modified by anyone (e.g. cloud backup solutions).
The Ethoscope system saves by default the data in two custom directories:
/ethoscope_results-- the results of tracking experiments
/ethoscope_videos-- raw video files (when recording)
Both directory have the same substructure:
<machine_id>is a unique identifier for the machine (e.g.
<machine_name>is a human friendly name (e.g.
<datetime>is a date and time formated as
Then, each file inside the directory will be uniquely and unambiguously named:
For instance, we could have a file named
2016-03-21_20-55-10_0001eeee10184bb39b0754e75cef7900.db. This redundancy means that if this file is sent to a collaborator, it preserves its uniqueness (plus we have some metadata in the file).
Example of Backups Script
This is the sort of scripts we use in the lab. You are welcome to take it and modify it. You will have to change things according to what sort of remote computer you save your data on (in the lab, we have our own NAS).
#!/bin/bash # we ensure the backup service is stopped during backup, so we do not keep modifying files # during backup backup_service=ethoscope_backup result_dir=ethoscope_results/ target_prefix=/mnt/nas/auto_generated_data/ ( ## we use flock to ensure no concurent backups # see http://stackoverflow.com/questions/169964/how-to-prevent-a-script-from-running-simultaneously # Wait for lock on /var/lock/.myscript.exclusivelock (fd 200) for 10 seconds flock -x -w 10 200 || exit 1 systemctl stop $backup_service && sleep 30 && python2 /root/incremental_size_upload.py -s /$result_dir/ -d $target_prefix/$result_dir/ Rscript /root/make_db_index.R cp /$result_dir/index.txt $target_prefix/$result_dir/index.txt sync systemctl start $backup_service ) 200>/var/lock/.$backup_service.exclusivelock
Complementary scripts (cron table, fstab) are available on our labscript repository.