Monday, 4 March 2013

Raspberry Pi - Google Drive & Grive

I use google drive a lot and I wanted to use my raspberry pi to automatically backup some of my more important files (i.e. documents and pictures) to google's cloud.

Now google hasn't released a google drive client for linux, as yet, but there is a really good open source linux client called grive which allows you to sync with google drive.

Install grive
I got most of this info from, http://www.lbreda.com/grive/installation.

Install dependencies
Use apt-get to install grive's dependencies, I already had some of these installed, but it won't hurt to re-run if your not sure.

sudo apt-get install git cmake build-essential libgcrypt11-dev libjson0-dev libcurl4-openssl-dev libexpat1-dev libboost1.50-all libboost-filesystem-dev libboost-program-options-dev binutils-dev libyajl-dev

Get the grive source code

cd ~
git clone git://github.com/Grive/grive.git

Compile grive

cd grive
cmake .
make

Wait a while for it to compile (10-20 mins)

Setup your google drive folder and authenticate 
You need to create a directory which grive will sync Google Drive with; anything in Google Drive will be synced into this folder and anything in this folder will be synced to Google Drive.

mkdir ~/googleDrive

In order to make grive work it needs to be in your google drive, so copy the grive executable to the google drive directory:

cp ~/grive/grive/grive ~/googleDrive

Authenticate grive to work with your Google Drive account.

cd ~/googleDrive
./grive -a

Grive will now give you a URL and ask you to provide an access code.  Cut and paste the URL into a browser, if your not logged in to google, login, and it will present you with an access code, cut and paste this access code into grive.

NOTE - In order to run grive, you must be in the google drive directory, for some reason if you are not, grive doesn't work and moans about having to authenticate even if you have!

Test
Grive can be run in a test mode, which will do everything other than uploading / downloading.

./grive --dry-run

Run

./grive

Once completed, your ~/googleDrive directory should now hold a copy of everything which is in your google drive.  Also any files you put in your googleDrive directory will also get synced to google drive.

Backup other files and folders
I wanted to backup other files and folders and didn't want to put them in my ~/googleDrive folder so I did this by using symbolic links.  Using symbolic links we can make it appear that a file or folder from a different location is in the google drive folder and as such will get synced to google drive.  I used this technique to create a link to my Pictures folder which is stored on my NAS drive, that way all my pictures are backed up to google drive.

ln -s /path/to/my/pictures ~/googleDrive/pictures

For more information on ln command.

Restricting bandwidth using trickle
One of the challenges I found in backing up my files to google drive was a simple one of bandwidth, or more accurately a lack of bandwidth.  When grive first ran, it uploaded a LOT of data to google drive and totally saturated by broadband (particularly as upload speeds are generally a lot less than download).

I came across a great utility called "trickle" which allows you to specify a maximum upload and download speed for any command, see http://www.tuxradar.com/content/control-your-bandwidth-trickle for more info.

Install trickle

sudo apt-get install trickle

Run grive using trickle
The following command will specify a maximum download (-d) and upload (-u) limit for grive of 50k.

cd ~/googleDrive
trickle -d 50 -u 50 ./grive

Schedule
I then chose to schedule grive to run once a week, using cron, thereby automating my file backup to google drive.


37 comments:

  1. Great idea, just stumbled across you site and this was one of the things on my "to do" list. Many thanks for putting this together.

    One correction though, libcurlost-filesystem-dev should be libboost-filesystem-dev.

    Others mileage may vary too, I had to include another couple of libraries that I didn't already have, specifically libexpat1-dev and libcurl4-openssl-dev.

    ReplyDelete
    Replies
    1. Thanks for the info, are you running Raspbian?

      Delete
  2. FYI and for all other noobs like me

    there is a typo in your dependencies, after looking for libcurlost-filesystem-dev package I realised that you should replace this with

    - libcurl4-openssl-dev
    - libexpat1-dev
    - libboost-filesystem-dev

    packages (I spent a half hour on this, yes i'm a tux noob)

    thanks for the work

    regards,
    Cem

    ReplyDelete
    Replies
    1. I'm a bit of an uber-noob to this. I've been stuck at the 'Compile' step for a while and haven't been able to make any progress. I'm guessing it's a libraries/dependencies issue?

      When I type in 'cmake .' it says 'bash: cmake: command not found'

      Delete
    2. It sounds like cmake isnt installed... try:

      sudo apt-get install cmake

      to install it.

      Delete
  3. Martin - thanks for this; have recreated this using the latest raspbian build (as of 22/03/2013) without trouble.

    One concern though - I wondered if you had given this thought - my gdrive is pretty big (>5GB now) and I don't want my poor pi to have to sync to the whole thing. I would like a 'pi' subdirectory, say, on GDrive that will get synchronised with the pi's local filesystem (or part of it). Have you seen anyone achieve this ? I don't think grive supports it although there was some discussion about this idea.

    Without this I think I will have to resort to googlecl, which has been a bit flakey for the last nine months or so (largely because it ought to be obsolete really). Any bright ideas ?



    ReplyDelete
    Replies
    1. This comment has been removed by the author.

      Delete
    2. Same issue. Have you found a solution? Thanks

      Delete
  4. I get an error when I run this command

    sudo apt-get install libcurl4-openssl-dev

    ReplyDelete
  5. I also had to manually install libboost test as well as qt4, 10 mins of googling told me how

    ReplyDelete
  6. same Problem with qt4. plz help me

    ReplyDelete
    Replies
    1. sudo aptitude install libqt4-dev

      Delete
    2. CMake Error at /usr/share/cmake-2.8/Modules/FindBoost.cmake:1194 (message):
      Unable to find the requested Boost libraries.

      Boost version: 1.49.0

      Boost include path: /usr/include

      The following Boost libraries could not be found:

      boost_unit_test_framework

      Some (but not all) of the required Boost libraries were found. You may
      need to install these additional Boost libraries. Alternatively, set
      BOOST_LIBRARYDIR to the directory containing Boost libraries or BOOST_ROOT
      to the location of Boost.



      the old version is working


      Delete
    3. This comment has been removed by the author.

      Delete
    4. 'sudo apt-get install libboost-test-dev' solved this for me.

      Delete
  7. I'm running raspbian and I keep getting these errors.

    /home/pi/grive/libgrive/src/json/JsonWriter.cc:24:27: fatal error: yajl/yajl_gen.h: No such file or directory
    compilation terminated.
    make[2]: *** [libgrive/CMakeFiles/grive.dir/src/json/JsonWriter.cc.o] Error 1
    make[1]: *** [libgrive/CMakeFiles/grive.dir/all] Error 2

    ReplyDelete
    Replies
    1. Try:

      sudo apt-get install libyajl-dev

      ... and then go back to building grive.

      I get a linking error at this point, but the solution to that can be found here:

      https://github.com/Grive/grive/issues/168

      Making the edits suggested there allows grive to compile completely.

      Delete
    2. That did the trick. Thanks.
      I followed the suggestions at that link and now it's compiled.

      Delete
    3. Thanks for the info guys. Its difficult to keep on top of the changes to grive, and comments like this really help.

      Delete
  8. I cannot to compile. Could you please help me?

    root@raspberrypi:/home/pi# sudo apt-get install git cmake build-essential libgcrypt11-dev libjson0-dev libcurl4-openssl-dev libexpat1-dev libboost-filesystem-dev libboost-program-options-dev binutils-dev libyajl-dev
    Czytanie list pakietów... Gotowe
    Budowanie drzewa zależności
    Odczyt informacji o stanie... Gotowe
    binutils-dev jest już w najnowszej wersji.
    build-essential jest już w najnowszej wersji.
    cmake jest już w najnowszej wersji.
    git jest już w najnowszej wersji.
    libboost-filesystem-dev jest już w najnowszej wersji.
    libboost-program-options-dev jest już w najnowszej wersji.
    libcurl4-openssl-dev jest już w najnowszej wersji.
    libexpat1-dev jest już w najnowszej wersji.
    libgcrypt11-dev jest już w najnowszej wersji.
    libjson0-dev jest już w najnowszej wersji.
    Zostaną zainstalowane następujące NOWE pakiety:
    libyajl-dev
    0 aktualizowanych, 1 nowo instalowanych, 0 usuwanych i 0 nieaktualizowanych.
    Konieczne pobranie 31,8 kB archiwów.
    Po tej operacji zostanie dodatkowo użyte 127 kB miejsca na dysku.
    Kontynuować [T/n]? t
    Pobieranie:1 http://mirrordirector.raspbian.org/raspbian/ wheezy/main libyajl-dev armhf 2.0.4-2 [31,8 kB]
    Pobrano 31,8 kB w 1s (17,1 kB/s)
    Wybieranie wcześniej niewybranego pakietu libyajl-dev.
    (Odczytywanie bazy danych ... 103826 plików i katalogów obecnie zainstalowanych.)
    Rozpakowywanie pakietu libyajl-dev (z .../libyajl-dev_2.0.4-2_armhf.deb) ...
    Konfigurowanie pakietu libyajl-dev (2.0.4-2) ...
    root@raspberrypi:/home/pi# cd ~
    root@raspberrypi:~# git clone git://github.com/Grive/grive.git
    fatal: destination path 'grive' already exists and is not an empty directory.
    root@raspberrypi:~# cd grive
    root@raspberrypi:~/grive# cmake .
    -- The C compiler identification is GNU 4.6.3
    -- The CXX compiler identification is GNU 4.6.3
    -- Check for working C compiler: /usr/bin/gcc
    -- Check for working C compiler: /usr/bin/gcc -- works
    -- Detecting C compiler ABI info
    -- Detecting C compiler ABI info - done
    -- Check for working CXX compiler: /usr/bin/c++
    -- Check for working CXX compiler: /usr/bin/c++ -- works
    -- Detecting CXX compiler ABI info
    -- Detecting CXX compiler ABI info - done
    -- Found libgcrypt: -L/lib/arm-linux-gnueabihf -lgcrypt
    -- Found JSON-C: /usr/lib/arm-linux-gnueabihf/libjson.so
    -- Found CURL: /usr/lib/arm-linux-gnueabihf/libcurl.so (found version "7.26.0")
    -- Found EXPAT: /usr/lib/arm-linux-gnueabihf/libexpat.so (found version "2.1.0")
    CMake Error at /usr/share/cmake-2.8/Modules/FindBoost.cmake:1194 (message):
    Unable to find the requested Boost libraries.

    Boost version: 1.49.0

    Boost include path: /usr/include

    The following Boost libraries could not be found:

    boost_unit_test_framework

    Some (but not all) of the required Boost libraries were found. You may
    need to install these additional Boost libraries. Alternatively, set
    BOOST_LIBRARYDIR to the directory containing Boost libraries or BOOST_ROOT
    to the location of Boost.
    Call Stack (most recent call first):
    libgrive/CMakeLists.txt:9 (find_package)


    -- Found libbfd: /usr/lib/libbfd.so
    -- Found libiberty: /usr/lib/libiberty.a
    -- Found ZLIB: /usr/lib/arm-linux-gnueabihf/libz.so (found version "1.2.7")
    -- Boost version: 1.49.0
    -- Found the following Boost libraries:
    -- program_options
    -- Looking for Q_WS_X11
    -- Looking for Q_WS_X11 - found
    -- Looking for Q_WS_WIN
    -- Looking for Q_WS_WIN - not found.
    -- Looking for Q_WS_QWS
    -- Looking for Q_WS_QWS - not found.
    -- Looking for Q_WS_MAC
    -- Looking for Q_WS_MAC - not found.
    -- Found Qt4: /usr/bin/qmake (found version "4.8.2")
    -- Boost version: 1.49.0
    -- Configuring incomplete, errors occurred!

    ReplyDelete
  9. Hmm...

    sudo apt-get install libboost1.50-all

    helped a lot:

    root@raspberrypi:/# cd root
    root@raspberrypi:~# cd grive
    root@raspberrypi:~/grive# cmake .
    -- Found libgcrypt: -L/lib/arm-linux-gnueabihf -lgcrypt
    -- Found JSON-C: /usr/lib/arm-linux-gnueabihf/libjson.so
    -- Boost version: 1.50.0
    -- Found the following Boost libraries:
    -- program_options
    -- filesystem
    -- unit_test_framework
    -- system
    -- Found libbfd: /usr/lib/libbfd.so
    -- Found libiberty: /usr/lib/libiberty.a
    -- Boost version: 1.50.0
    -- Found the following Boost libraries:
    -- program_options
    -- Boost version: 1.50.0
    -- Configuring done
    -- Generating done
    -- Build files have been written to: /root/grive

    ReplyDelete
  10. But still not working:

    ...

    [ 41%] Building CXX object libgrive/CMakeFiles/grive.dir/src/util/log/DefaultLog.cc.o
    [ 42%] Building CXX object libgrive/CMakeFiles/grive.dir/src/util/log/Log.cc.o
    [ 44%] Building CXX object libgrive/CMakeFiles/grive.dir/src/util/log/CompositeLog.cc.o
    [ 46%] Building CXX object libgrive/CMakeFiles/grive.dir/src/xml/TreeBuilder.cc.o
    [ 48%] Building CXX object libgrive/CMakeFiles/grive.dir/src/xml/NodeSet.cc.o
    [ 50%] Building CXX object libgrive/CMakeFiles/grive.dir/src/xml/Node.cc.o
    [ 51%] Building CXX object libgrive/CMakeFiles/grive.dir/src/xml/String.cc.o
    [ 53%] Building CXX object libgrive/CMakeFiles/grive.dir/src/bfd/SymbolInfo.cc.o
    [ 55%] Building CXX object libgrive/CMakeFiles/grive.dir/src/bfd/Backtrace.cc.o
    [ 57%] Building CXX object libgrive/CMakeFiles/grive.dir/src/bfd/Debug.cc.o
    Linking CXX static library libgrive.a
    [ 82%] Built target grive
    Scanning dependencies of target btest
    [ 83%] Building CXX object libgrive/CMakeFiles/btest.dir/test/btest/UnitTest.cc.o
    [ 85%] Building CXX object libgrive/CMakeFiles/btest.dir/test/btest/ValTest.cc.o
    [ 87%] Building CXX object libgrive/CMakeFiles/btest.dir/test/btest/JsonValTest.cc.o
    Linking CXX executable btest
    [ 87%] Built target btest
    Scanning dependencies of target grive_executable
    [ 89%] Building CXX object grive/CMakeFiles/grive_executable.dir/src/main.cc.o
    Linking CXX executable grive
    ../libgrive/libgrive.a(State.cc.o): In function `gr::v1::State::Write(boost::filesystem::path const&) const':
    State.cc:(.text+0x19bc): undefined reference to `gr::Json::Json(long const&)'
    State.cc:(.text+0x1a4c): undefined reference to `gr::Json::Json(unsigned long const&)'
    State.cc:(.text+0x1b2c): undefined reference to `gr::Json::Json(long const&)'
    collect2: ld returned 1 exit status
    make[2]: *** [grive/grive] Błąd 1
    make[1]: *** [grive/CMakeFiles/grive_executable.dir/all] Błąd 2
    make: *** [all] Błąd 2
    root@raspberrypi:~/grive#

    ReplyDelete
    Replies
    1. your solution can be found at https://github.com/Grive/grive/issues/168
      See the post by chubbyupchuck

      Delete
  11. i have this error when i tri to do: cmake .

    pi@raspberrypi ~/grive $ cmake .
    -- The C compiler identification is GNU 4.6.3
    -- The CXX compiler identification is GNU 4.6.3
    -- Check for working C compiler: /usr/bin/gcc
    -- Check for working C compiler: /usr/bin/gcc -- works
    -- Detecting C compiler ABI info
    -- Detecting C compiler ABI info - done
    -- Check for working CXX compiler: /usr/bin/c++
    -- Check for working CXX compiler: /usr/bin/c++ -- works
    -- Detecting CXX compiler ABI info
    -- Detecting CXX compiler ABI info - done
    CMake Error at cmake/Modules/FindLibGcrypt.cmake:46 (message):
    Could not find libgcrypt libraries
    Call Stack (most recent call first):
    libgrive/CMakeLists.txt:5 (find_package)


    -- Configuring incomplete, errors occurred!

    ReplyDelete
  12. i can't do:
    sudo apt-get install libcurl4-openssl-dev

    Need to get 1,137 kB/5,459 kB of archives.
    After this operation, 10.1 MB of additional disk space will be used.
    Do you want to continue [Y/n]? y
    Err http://mirrordirector.raspbian.org/raspbian/ wheezy/main libgnutls-openssl27 armhf 2.12.20-7
    404 Not Found
    Err http://mirrordirector.raspbian.org/raspbian/ wheezy/main libgnutlsxx27 armhf 2.12.20-7
    404 Not Found
    Err http://mirrordirector.raspbian.org/raspbian/ wheezy/main libgnutls-dev armhf 2.12.20-7
    404 Not Found
    Failed to fetch http://mirrordirector.raspbian.org/raspbian/pool/main/g/gnutls26/libgnutls-openssl27_2.12.20-7_armhf.deb 404 Not Found
    Failed to fetch http://mirrordirector.raspbian.org/raspbian/pool/main/g/gnutls26/libgnutlsxx27_2.12.20-7_armhf.deb 404 Not Found
    Failed to fetch http://mirrordirector.raspbian.org/raspbian/pool/main/g/gnutls26/libgnutls-dev_2.12.20-7_armhf.deb 404 Not Found
    E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

    ReplyDelete
    Replies
    1. Your apt-get directories are out of date. Have you tried running sudo apt-get update as per the error message, and trying again?

      Delete
  13. I'm trying to follow the guide and when i'm typing MAKE in "Compile Grive" i get following message.
    Du anybody have an idea???


    [ 81%] Built target grive
    [ 86%] Built target btest
    Linking CXX executable grive
    ../libgrive/libgrive.a(State.cc.o): In function `gr::v1::State::Write(boost::filesystem3::path const&) const':
    State.cc:(.text+0x19bc): undefined reference to `gr::Json::Json(long const&)'
    State.cc:(.text+0x1a4c): undefined reference to `gr::Json::Json(unsigned long const&)'
    State.cc:(.text+0x1b2c): undefined reference to `gr::Json::Json(long const&)'
    collect2: ld returned 1 exit status
    grive/CMakeFiles/grive_executable.dir/build.make:95: recipe for target 'grive/grive' failed
    make[2]: *** [grive/grive] Error 1
    CMakeFiles/Makefile2:162: recipe for target 'grive/CMakeFiles/grive_executable.dir/all' failed
    make[1]: *** [grive/CMakeFiles/grive_executable.dir/all] Error 2
    Makefile:113: recipe for target 'all'

    ReplyDelete
    Replies
    1. No sorry, grive is a constantly developing application though. I would raise an issue on the github project and see what feedback you get https://github.com/Grive

      Delete
  14. Is there a size limitation?
    I have this running to a mounted 2TB USB drive on my Pi.
    I'm getting about 16GB of data, but then the /grive process is stopping. I have 132GB of total content I'm trying to sync.

    ReplyDelete
  15. Grive upload only, not need download to google drive. Who can help me, please?. i don't need download from google drive.

    ReplyDelete
    Replies
    1. Take a look at the grive usage page, http://www.lbreda.com/grive/usage, it might help, the only option I saw though which looked useful was "-f [ –force ] - Force grive to always download a file from Google Drive instead of uploading it."

      Delete
  16. We are currently working on project which includes uploading images from raspberry pi to google drive.We are getting an error while executing cmake and make command.So can you please suggest proper solution for it??

    ReplyDelete
    Replies
    1. What error?

      I suspect it is probably down to missing dependencies. Grive is in a constant state of development so its difficult to keep up with all the dependencies.

      Delete
  17. Please help. I get below error while trying to authenticate and syncup using the google drive authentication key.

    Reading local directories
    Synchronizing folders
    exception: /home/pi/grive/libgrive/src/protocol/AuthAgent.cc(174): Throw in function long int gr::AuthAgent::CheckHttpResponse(long int, const string&, const gr::http::Header&)
    Dynamic exception type: N5boost16exception_detail10clone_implIN2gr4http5ErrorEEE
    [PN2gr4expt12BacktraceTagE] = #0 0xff688 :0 gr::Exception::Exception()
    #1 0xe0e88 :0 gr::http::Error::Error()
    #2 0xe8540 :0 gr::AuthAgent::CheckHttpResponse(long, std::string const&, gr::http::Header const&)
    #3 0xe7fe4 :0 gr::AuthAgent::Get(std::string const&, gr::DataStream*, gr::http::Header const&)
    #4 0xdcd50 :0 gr::v1::Drive::SyncFolders()
    #5 0xdd254 :0 gr::v1::Drive::DetectChanges()
    #6 0xa79f0 :0 Main(int, char**)
    #7 0xa7f94 :0 main
    #8 0x7689e294 /lib/arm-linux-gnueabihf/libc.so.6 __libc_start_main

    [PN2gr4http15HttpResponseTagE] = 400

    ReplyDelete
    Replies
    1. Try raising an issue on grive's github - https://github.com/Grive/grive

      Delete