Opensource slowing down? "GoogleDrive" private cloud

[Tim Watts]

I have raised this before... And tried some stuff. And I cannot believe there is no solution... hence the first part of the subject.

The problem: "How do I make an existing linux file server be a private cloud file server, targeting other linux, ChromeBooks and Android clients?"

There is a LOT of stuff that misses out of critical features - or those features need a paid-for version that costs $1700-9000 per year.

I've played with Tonido and OwnCloud so far.

Obvious omissions in the free versions:

a Limited shares (OwnCloud only seems to support 1 exported directory - I have lots);

b (Serious) Do not integrate with POSIX users. The idea is the file sharing should integrate with linux, not try and take over.

I did look at OpenAFS but there's no Android client.

SFTP so far offers the simplest solution but I can find no nice clients that offer local copies (caching) so it is still somewhat half arsed. And i suspect the basic nature of the protocol is it would be hard work to make caching multiple client copies work.

OK - allowing for the fact that "I could write it myself dammit" (actually, no, I don't have that type of coding skill) I'm very surprised such a killer app is missing from the opensource stable.

This is the same opensource that brought us *BSD, Linux, perl, python, 2 super RDBMSs, top rate image editing, rock solid *connected* fileservices (NFS, SMB) allowing for the fact the latter had to be reverse engineered.

Have all the creative types died?

[Jeroen Belleman]

Bah. Linux and Unix have all sorts of remote/shared filesystems. Of course, some 'creative' minds thought it'd be a good idea to invent yet another such thing. That's the bane of computing. It's so often easier to reinvent something than it is to find out how to use the existing tricks.

Jeroen Belleman

[The Natural Philosopher]

Indeed. I find that a 'remote backup of local data' which allows global access to that data is easily achieved with rsync.

Assuming Linux CLIENTS

For 'within these walls' services I simply have a file server running NFS and samba which covers all the bases.

And its own internal backup policy and a nice minidnla server to export videos.

[Tim Watts]

OK - tell me one that works well with Android, Chromebooks with a semi-connected internet?

Because that's what I want.

And I have not found any.

SMB - no chance NFS - not the slightest bit suitable OpenAFS - the right logic is there but no client support

Most of the cloud-fs solutions seem to hang around WebDAV - but probably augmented with update hints to aid caching and push delivery of updates.

It is a specialist network filesystem - that is for sure. It strikes me as strange that there are 101 commercial suppliers (Dropbox, Google, Evernote, Spideroak...) and yet no one has come up with a nice well defined open protocol that addresses replication and syncing.

[Tim Watts]

And that does not meet my criteria.

It's kind of what I do with a laptop - but with Unison rather than rsync.

But when you have TBs of data (many media files - mostly photos) you will never be able to sync that 100% to random devices - they simply don't have the storage.

[Jethro_uk]

Not quite sure what your need is, but for some reason I've remembered that it's possible to run a single RAID across many devices under mdadm ... never done it, but would that help ?

[Robert Heller]

What about something like git or subversion? *Subversion* uses a WebDAV-based protocol for svn-over-http(s).

What is sounds like is that you are looking for is something like subversion's WebDAV-based protocol. The bits and pieces are there. WebDAV is reasonably well documented. Subversion is open source, so one can have a look at what it does.

As others have said, programs like rsync seem to cover the bases that Linux

*Developers* are interested in. I suspect that few of the Linux *developers* are interested in yet another pointy-clicky 'toy'. *I* certainly have no interest in such a toy. OTOH, I have dial-up Internet access at home, so any sort of 'Cloud' services are pretty much not much use to me. Subversion (or git) are about the most I can deal with.

[Dan Espen]

Both my Kindle and Android have no problem with Samba shares over wireless.

[Tim Watts]

No. The clients include Android with tiny local storage.

I'll restate the requirements:

1) Central server;

2) Internet access;

3) Android and Chromebook clients with local caching (like Google Drive)

4) Linux client (obviously)

=======

It's not actually outlandish. If it were, there would not be a load of companies doing it. In fact it is a very well defined problem.

However, some of us prefer to keep out data at home (not to mention that TBs of data in the public cloud is actually not cheap!).

Remember that whilst lots of us do have TBs of data, we only access a small amount at any time.

eg: I might mark my "documents" folder and "data/house-plans" folder to be fully sync'd (offline mode) as I access spreadsheets and "word" documents in both very regularly. I might mark one photo sub directory as "sync offline mode" if I were going to show some stuff at a remote site.

The rest I am happy to take my chances of internet connectivity.

It could also be viewed as an HSM problem...

In fact I'm surprised that Western Digital or Buffalo have not sponsored some development because it's exactly what you'd want for a little NAS/cloud box.

I also wonder why noone's tried reverse engineering GoogleDrive protocol

- a lot of effort was made to hack SMB. That way, the clients are a done deal and Google Drive works really well[1] - I can edit a document on my computer and watch it update automagically on my phone.

[1] apart from the fact that free storage is untrustworthy, both for reliability and security and buying TBs would cost.

[Tim Watts]

Hmm - I do wonder how that would scale with big binary files. But I agree, there is almost certainly something useful in the meta data that would assist sync-ing. if we dispensed with the ability to roll back and simply worried about every client's idea of the HEAD version, then it would be easy for clients to work out who had the most recent copy.

But I think the mercurial logic would be better suited compared to SVN as Mercurial is an HTTP based version of Git and Git was designed from the ground up to be devoid of any central authority.

With respect I would hardly describe it as a "toy"

It is a real world problem.

Case in point:

10 years ago my house had desktops, connected by NFS to a central server and all was well.

Now, we have people (4) passing laptops, pads and phones around. And we may be needing a chromebook for the kids (schooling). NFS simply does not work.

I find myself uploading stuff ad hoc to Google Drive just so SWMBO can look at a copy on her phone.

if I don't get a solution in place we will have 7 devices with random bits of random peoples files all over them.

Open source dev folk use pads and phones - maybe not Chromebooks - but I refuse to believe they are so stuck in the stone age they cannot see where the future is going. Right now it's a mess.

This is most definitely a problem that needs a solution even if not everyone knows it yet.

[HarpingOn]

I use Open Media Vault on a HP Microserver.

WebDAV might give you what you want

Chromebooks aren't very friendly to anything that isn't Google Drive, although I have used webdav with mine.

[Tim Watts]

OK - I'm interesed - please tell me more.

What client are you using and can it offline sync certain files?

That was an assumption on my part - I have not seen SMB jump out at me with my interaction with Android.

However, it's no bother to install SaMBa on teh servers if it solves the problem... At least I know it works correctly at the back end (user auth, POSIX UID/GID mapping etc).

Cheers!

Tim

[HarpingOn]

My Xperia Android devices even have an option to mount SMB shares as if they were SD cards. Bit surprising to find that feature.

[Tim Watts]

That's interesting - I have not come across that. I will spin that up on a virtual machine and see who it works - if it does, I can probably cherry pick the bit I want and shove it onto my existing Debian server (which coincidently runs on an HP Micro Server!).

May I ask - what client are you using? I tried that once (about a year back) but the couple of clients I tried were ropey.

That is also very useful - thank you.

[Tim Watts]

How do they respond when the network goes wobbly? Graceful failure and back up when the network is stable again?

[The Natural Philosopher]

This is so much a wintroll type post.

"I want EXACTLY what I used to have in Windows for nothing on Linux, and if Linux can't do it exactly .linux sucks'

In short if you want a commercial cloud behaviour stick to commercial clouds.

[Theo Markettos]

What's interesting about OMV in regards to this discussion? It looks like A.N.Other NAS Linux distro. I can't see anything there that handles offline files, which is the critical part of the problem.

There is an OpenAFS FUSE client, so that could work for Android, Mac, etc though it's still experimental (no write support). And no pretty GUI.

Very interesting thread, BTW - I have exactly the same problems...

Theo

[Tim Watts]

Thank you - will check that out later.

Now you're making me laugh - because you of all people know that I am a linux die hard!

Note I am not criticising linux, but making a pertinent observation that, after all the massive effort that went into cracking SMB (and I remember that effort, late 90's) I am SO surprised that noone (not even an MIT student project) has come along and said - this is actually not so hard, and just did it.

Bit like the way we all put up with that PoS WU-IMAPd then Timo Sirainen came along and said "this is bollocks - here I just wrote dovecot and solved all the logic problems cleanly".

Ditto Philip Hazel and exim.

The complication here is the massive array of client devices that need supporting. It would probably be a good approach to be able to emulate a Google Drive server protocol (then we don't care about teh client software because it exists).

There is a strong risk that Google will then say "f*ck you, we just upgraded the protocol". However, by that time, if the server is proven, people would have a chance to write replacement clients without it being an "all at once" problem.

That was rather the point. Those guys have proved it can be done well. But there needs to be an opensource version that works as well (and there isn't).

[The Natural Philosopher]

There are some very good reasons for that.

An internet based cloud is by definition reliant on a high bandwidth server somewhere on the internet backbone.

At some level that is a commercial entity.

Adding free software to a rented (virtual) server to make a global cloud is not actually any cheaper than renting a bit of someone elses cloud.

If its domestic file sharing you want, stuff a domestic files server in.

[Tim Watts]

I was assuming they had a tuned WebDav server instance - that might be worth ripping off as it's probably only going to be an apache config.

I was hoping that there might be a client that could handle the caching

- though without certain explicit server support it may not be very efficient.

Thank you - I am surprised it's only the two of us though. Perhaps others are less picky?

The commercial guys clearly see a LOT of milage in this. However, even Tonido came back to me and said their Enterprise FileCloud (that costs $$$$) can only handle NTFS user file perms and ownership - POSIX may happen in the future.

I have seen this moment before, at least twice:

1) The great MS office desert - if you were on linux/BSD and everyone else was throwing Word files around. It was a long time before something better than "strings" came along and it was largely thanks to Sun. 2) Email. There was a very dark time in the late 90's/early 00's when we had a piss poor IMAP server (WU-IMAPd) and very ropey client support.

Then Thunderbird got its act together and Timo knocked out dovecot and all because sane again.

SMTP was less of a problem even before Philip Hazel as sendmail works, even though it had a config from hell.

It does seem like many of the great solutions are down to one person, Linus, Timo, Philip. It can take a while for the momentum to roll.

In some cases, a great project simply sits at risk of dying. case in point: MoSH -

- I use it all the time - it is fantastic when you have a tethered internet connection with the IP changing all the time. However, it has not gained much traction and even though adding new features like tunnels has been mooted, noone wants to do it and I expect the original guys are off doing other interesting stuff.