[#SFOS-934] Create a component for file transfers and file system operations on a remote SmartFrog-enabled host

SmartFrog

Create a component for file transfers and file system operations on a remote SmartFrog-enabled host

Created: 04/Aug/08 04:38 PM (BST) Updated: 05/Aug/08 12:43 PM (BST)

Component/s:

.sfCore

Affects Version/s:

None

Fix Version/s:

None

Time Tracking:

Not Specified

Compatibility:

new feature

Description

« Hide

A lot of deployments need to transfer files to remote hosts. Currently one uses the scp components for this, but we already have communication between hosts through SmartFrog, so it seems an unnecessary inconvenience to have to maintain user accounts for SSH on the side.

My initial idea for this is to have 2 components : one that is deployed on the initial host, and another deployed on the remote host by specifying a different sfProcessHost attribute. The interface for the remote component would then include upload (and also download, and other useful filesystem operations).

Of course this is going to be horribly inefficient since all data will be marshalled through RMI... but for small file transfers it shouldn't matter too much, and would be very convenient. One can always revert to the scp components if things get too slow.

Also, I'm opening a separate issue about adding recursive operation to the Files component, since this is how I think files to be uploaded (and maybe dowloaded too) should be specified.
Comments ?

All

Comments

Work Log

Change History

Sort Order:

[ Permlink | « Hide ]

Julio Guijarro added a comment - 04/Aug/08 05:08 PM (BST) - edited

You already said it. It would be extremely inefficient to use RMI for that but very quick to implement.
It would be better create an abstract file system and use something like Jetty or and FTP lib to serve the data. The security keys could be exchanged using the encrypted RMI connection. Another option would be to create components to use the Hadoop file system.

Example to start Jetty as a file server: http://jetty.mortbay.org/jetty-6/xref/org/mortbay/jetty/example/FileServer.html. No security though.

[ Permlink | « Hide ]

Steve Loughran added a comment - 04/Aug/08 05:14 PM (BST)

RMI may be very inefficient, but this could be useful for long-haul filesystem operations without opening any more ports on a machine

[ Permlink | « Hide ]

Olivier Pernet added a comment - 04/Aug/08 05:19 PM (BST)

Well, this idea comes from my need to transfer a couple files a a small part of my small deployment that I was hoping to get running quickly. So I was thinking of something simple, to implement, but more importantly to use. I feel like SmartFrog often lacks simple components... That's probably because it was initially intended as a framework and not as a tool, but I want to use it as a tool now :) And I think there's a need for it too. That's the kind of use Steve describes in Ant in Action, but I find it hard to make it work myself.

Now that I'm done with my rant, I think this would still be worth trying at least. The performance shouldn't matter too much when transferring a couple configuration files and small JAR files, which is what I need.

The real, long-term, break-everything-and-start-over solution to this kind of problem would be to do all communication between SmartFrog components through HTTP and sending dumped component descriptions over the wire, and also allowing access to the raw HTTP channel for anything that isn't a component description. But that's not quite the quick solution I was looking for initially :)

[ Permlink | « Hide ]

Olivier Pernet added a comment - 05/Aug/08 12:43 PM (BST)

This has some similarities with the LocalFilestore component, some things can probably be shared.