Issue Details (XML | Word | Printable)

Key: SFOS-855
Type: Improvement Improvement
Status: Resolved Resolved
Resolution: Fixed
Priority: Major Major
Assignee: Steve Loughran
Reporter: Steve Loughran
Votes: 0
Watchers: 0
Operations

If you were logged in you would be able to see more operations.
SmartFrog

Hadoop client-side components should be able to pick up properties from a cluster CD

Created: 14/May/08 05:02 PM (BST)   Updated: 25/Feb/09 04:46 PM (GMT)
Return to search
Component/s: _service_hadoop
Affects Version/s: None
Fix Version/s: 3.17.010

Time Tracking:
Not Specified

Issue Links:
Depends
 
Metabug
 

Compatibility: backwards compatible


 Description  « Hide
This is related to a design choice of hadoop: your clients need to have a correct server configuration for job submission to work. This is not (yet) implemented from communication with the server. so the client config files need to be right.

This is very brittle, and forces us to build up the production components through inheritance, placing in all the various attributes that are needed to work.

If we could build our conf not just through inheritance from a parent (and nesting), but by referencing a Cluster CD/Prim, then we woudn't need to patch every template to deploy work to different customers. something like

SubmitJob {
 cluster LAZY HplbCluster;
 job "...";
}

This could be done with a tweaked sfResolve derivative (that also tried to resolve via the cluster ), and with the code to build a ManagedConfiguration to read in Cluster content (first)

 All   Comments   Work Log   Change History      Sort Order: Ascending order - Click to sort in descending order
Steve Loughran made changes - 14/May/08 05:02 PM (BST)
Field Original Value New Value
Link This issue is depended on by SFOS-780 [ SFOS-780 ]
Steve Loughran made changes - 18/Nov/08 04:42 PM (GMT)
Link This issue is covered under SFOS-905 [ SFOS-905 ]
Steve Loughran added a comment - 25/Feb/09 04:46 PM (GMT)
done

Steve Loughran made changes - 25/Feb/09 04:46 PM (GMT)
Status Open [ 1 ] Resolved [ 5 ]
Fix Version/s 3.17.x [ 10160 ]
Resolution Fixed [ 1 ]