Issue Details (XML | Word | Printable)

Key: SFOS-1055
Type: Sub-task Sub-task
Status: Closed Closed
Resolution: Fixed
Priority: Major Major
Assignee: Steve Loughran
Reporter: Steve Loughran
Votes: 0
Watchers: 0
Operations

If you were logged in you would be able to see more operations.
SmartFrog
SFOS-780

write component to load in settings from hadoop's XML files

Created: 14/Jan/09 11:48 AM (GMT)   Updated: 13/Feb/09 02:57 PM (GMT)
Component/s: _service_hadoop
Affects Version/s: None
Fix Version/s: 3.17.010

Time Tracking:
Not Specified

Issue Links:
Depends
 

Compatibility: unknown


 All   Comments   Work Log   Change History      Sort Order: Ascending order - Click to sort in descending order
Steve Loughran added a comment - 14/Jan/09 11:50 AM (GMT)
If we could read a hadoop-site.xml and similar, existing Hadoop settings could be picked up in an SF deployment.

This would let us
-keep up to date with hadoop default.xml changes
-import site specific settings, without having to re-enter stuff.

Steve Loughran added a comment - 14/Jan/09 12:02 PM (GMT)
Proposal

A component "HadoopXmlFiles" which takes a list of filenames/FileIntf/resources.

These are loaded in order. Alternatively, run through all attributes with the name fileXXX or resourceXXX in the order they are declared. This makes inheritance easier and avoids complex file-or-resource logic.

fileXXX: FileIntf or string filename. if "" or NULL then skip
resourceXXX: string resource name, sfClass classloader. If "" or NULL then skip
failOnError true/false: should failure to read a file/resource be an error?

urlXXX: read in a URL (java.net?).

Steve Loughran added a comment - 14/Jan/09 01:30 PM (GMT)
even better, an anubis reference :)

Steve Loughran added a comment - 14/Jan/09 03:54 PM (GMT)
Started this; using Hadoop's own logic to load files as that will do the inline variable expansion the default files require.

Also consider:
1. adding a validate option that does some validation of required attributes
2. add a list of required attributes which are resolved; if not local are set in place here.

Steve Loughran added a comment - 15/Jan/09 01:20 PM (GMT)
Done, checked in