Opensource slowing down? "GoogleDrive" private cloud

As I see it there are two parts to the problem: storage and syncing.

Storage is covered by the NAS. That's the master copy, everything should migrate there. You can nuke all the other copies in the world and you'll only lose the most recent changes. You can do a bulk download quite easily over gigabit ethernet/wifi/USB/SATA.

Tools like Google Drive, OneDrive and Dropbox aren't a good idea for the master copy: they're small and they're at the wrong end of the internet. They're also vulnerable to 'ransom risks': if they announce pricing for next month will be $500pm, what do you do?

Another risk is site failure: fotopic.net went under and it turned out the bandwidth bill for sucking all the photos off it was beyond the means of the company that had just called in the receiver because they had no money. Likewise Code Spaces got hacked and the attacker just deleted all their files - there was nothing left. The message from this is that if a cloud provider goes under, it's simply gone, you don't get your data back.

However they could be useful for transit. For instance, git is a nice protocol because every working copy contains a full copy of the history. That means I don't care if github goes under - I have everything locally. So github is simply a nice website for publishing the sources, and if github went away it wouldn't be much effort to republish elsewhere. If the files are encrypted you also don't need to worry about privacy (though metadata is still an issue, and always the devil is in the key management).

So I could see how something like this could work - Dropbox et al are just a transit provider for sharing stuff back to base. This also means they're a commodity conduit - if Dropbox are charging £silly, just switch to OneDrive (or whoever). The one thing I don't see this doing is sending a request to the NAS saying 'please put file XXX in Dropbox so I can see it' - maybe there's a way round that. Or maybe you just say 'you can only view the photos when you're at home'.

On the wider philosophical question, I found it depressing to read of an app for sending encrypted SMS where the developers are tightly wedded to the Google Play infrastructure, while seemingly not realising the tensions between these two directions:

Theo

No caching, but the well-known gftp FTP client also supports SFTP. IME it gives a seamless, trouble-free experience with either protocol.

And whilst Drive is *probably* not going away soon, Google have form for randomly pulling projects at short notice and for selling stuff on.

It would be fun to sniff the conversation between Drive and my computer

- with a minimal client so I don't see all the web supporting stuff.

I *wonder* if it is something that could be reverse engineered into a home server...

Another item that has not been mentioned yet is git annex:

Disclaimer, never used it. Also don't know how well it 'interfaces' with android/chromebooks. Both of which, while running Free Software Linux underneath the hood have much more in common with proprietary software than Free Software. Note that on both, even though the engine/transmission is Linux, ship in their default config with the hood (bonnet) firmly locked down against end user modification.

Git annex's webpage description does talk about keeping sync in the face of intermittent/mobile network connection.

Another one that has not come up yet (at least that I recall):

Tahoe-LAFS

No, no, I'm interested in the same thing. Just a bit less vociferous in my disappointment at not finding anything ready-made.

It's not -- I hasten to add -- that I feel any entitlement to a ready-made Open Source solution to the problem ... just that there is so much goos OS software out there that frankly I'm surprised NOT to find one.

Maybe all the good developers with an interest in this sort of thing already work for Google ...

To be fair to the OP (Tim Watts) the issue seems not to be one of cost, but rather of privacy and control.

Google make all this lovely free stuff available (at some cost to them, in terms of development and hosting) so that we will be encouraged to use their services and partake of their delicious advertising. it's a business model, and rather less obnoxious than some. One thing that Google (and others like them) are NOT likely to do is to share their IP so that we can all run lookalike ad-free servers for our own convenience.

Pity, I'd love to be able to use software as nice as Google Docs from a private cloud so that my data wasn't on Google's servers (or, indeed, and servers outside my own country of residence).

That does, indeed, work fairly well ... though that it takes a little more effort to make it safely available remotely (VPN).

The problem here is the Chromebook, which won't talk CIFS. It's easy enough to put files on a local server (Apache, whatever) and access them from the browser on a Chromebook, but some server-side cloudiness is needed to make it possible to upload them back to the server afterwards, and it won't be as transparent as using Google Docs on Google Drive.

I would have thought something like OwnCloud would provide that, though? (OK, I note the OP's issue with the number of shares OwnCloud Free allows.)

A Chromebook is not a computer: Its a stripped down access device to GoogleCloud.

And Google will actively prevent it accessing anything else as well as that.

There are clients like:

I don't know of their maturity. Bolting on encryption is another question.

However, even encrypted I don't think this is the full solution. Because the model is that everything lives primarily in the cloud, whereas we're using the cloud as simply a disposable cache of what we have locally. That means they don't present the unified worldview of 'here are all the files you can have, but some of them will take longer' that a remote FS would do.

Theo

I've been meaning to look at that for a while. It does seem interesting in that it supports protocols like Google Drive for storing files. However, I don't know if they're usable on Android/etc if encrypted - you just have a pile of encrypted files and hashes.

There's an Android app:

Theo

Oh - that does look interesting!

Aye - could be.

I'd work for Google if they let me :) (I tried) Yiu should see their London office!

In a nutshell. I'd pay a reasonable amount for owning a copy of something like Google Drive for my own servers. But not $9000!!!

I'm mostly opensource, but I do pay for a few apps quite happily.

Totally...

But as you say, Google's main income is advertising, not selling software - or even services (though they do do the latter).

It did occur to me I could run multiple instances of OwnCloud, one per share - but it's a bodge...

Just to make things more complicated, there's a Tahoe backend for git-annex

Theo (time to lie down and recover from architecture diagram overload ;-)

Hot-Damn - that is very very interesting - that demands a play...

Wow - thanks folks, turning up some cool stuff :)

It worries me when they make statements like..

"When Bob needs access to some files, git-annex can tell him which drive(s) they're on, and easily make them available. Indeed, every drive knows what is on every other drive."

Just how do the offline drives know what he has just done on the online ones. Its not actually possible. The best they can do is sync them so they know the current state is when you connect one online until then some of the drives have no idea what is on the others.

Check out git-annex assistant

Don't be daft. Where does it mention 'offline' drives? It's Bob that wants to know where various files are, not the drives.

Run rsync in a cron job?

Use puppet - it does all that and more.

I copped out and did my own using gambas. It took me about 3 days to write scripts that do do file copying automatically across continents with ssh, email log files with sendemail, etc etc etc.

Nothing ever dies in open source, and if it does, there is usually a patch for it available in 24 hours. What is happening is that all commonly requested wants have been satisfied and thus topics go quiet. So you need to get out and ask.