Issue Details (XML | Word | Printable)

Key: SFOS-1101
Type: Bug Bug
Status: Resolved Resolved
Resolution: Fixed
Priority: Major Major
Assignee: Steve Loughran
Reporter: Steve Loughran
Votes: 0
Watchers: 0
Operations

If you were logged in you would be able to see more operations.
SmartFrog

wrong termination semantics in the HadoopServices

Created: 06/Feb/09 07:09 PM (GMT)   Updated: 13/Feb/09 02:54 PM (GMT)
Component/s: _service_hadoop
Affects Version/s: 3.17.010
Fix Version/s: 3.17.010

Time Tracking:
Not Specified

Compatibility: unknown


 Description  « Hide
the HadoopService components have a flag that has been always set to true saying they expect node termination; this should be be read from the configuration file, and we may want to consider whether an termination with an exception (that is, the service went through the failed state) to be abnormal

 All   Comments   Work Log   Change History      Sort Order: Ascending order - Click to sort in descending order
Steve Loughran added a comment - 06/Feb/09 08:16 PM (GMT)
changed to
 -use an attribute (always set to true)
 -always throw if the service has failed


Steve Loughran added a comment - 06/Feb/09 08:16 PM (GMT)
but it is bailing on JobSubmissionTest on a slow machine

system-tests:
[sf-startdaemon-debug]
[sf-startdaemon-debug] SmartFrog 3.17.005dev (2009-02-05 12:07:34 GMT)
[sf-startdaemon-debug] (C) Copyright 1998-2009 Hewlett-Packard Development Company, LP
[sf-startdaemon-debug]
[sf-startdaemon-debug] 2009/02/06 20:13:59:202 GMT [WARN ][main] SFCORE_LOG - SmartFrog security is NOT active
[sf-startdaemon-debug] 2009/02/06 20:13:59:222 GMT [WARN ][main] SFCORE_LOG - Warning: stack trace logging enabled
[sf-startdaemon-debug] SmartFrog ready... [rootProcess:3800] Fri Feb 06 20:14:00 GMT 2009
[sf-system-test-junit] WARNING: multiple versions of ant detected in path for junit
[sf-system-test-junit] jar:file:/home/slo/Java/Apache/ant/lib/ant.jar!/org/apache/tools/ant/Project.class
[sf-system-test-junit] and jar:file:/home/slo/.ivy/cache/org.apache.ant/ant/jars/ant-1.7.1.jar!/org/apache/tools/ant/Project.class
[sf-system-test-junit] Testsuite: org.smartfrog.services.hadoop.test.system.local.tracker.JobSubmissionTest
[sf-startdaemon-debug] 09/02/06 20:14:18 [NameNode] INFO namenode.NameNode : State change: NameNode is now STARTED
[sf-startdaemon-debug] 09/02/06 20:14:18 [DataNode] INFO datanode.DataNode : State change: DataNode is now STARTED
[sf-startdaemon-debug] 09/02/06 20:14:18 [NameNode] INFO metrics.RpcMetrics : Initializing RPC Metrics with hostName=ExtNameNode, port=8020
[sf-startdaemon-debug] 09/02/06 20:14:18 [NameNode] INFO namenode.NameNode : Namenode up at: localhost/127.0.0.1:8020
[sf-startdaemon-debug] 09/02/06 20:14:18 [NameNode] INFO jvm.JvmMetrics : Initializing JVM Metrics with processName=NameNode, sessionId=null
[sf-startdaemon-debug] 09/02/06 20:14:18 [NameNode] INFO metrics.NameNodeMetrics : Initializing NameNodeMeterics using context object:org.apache.hadoop.metrics.spi.NullContext
[sf-startdaemon-debug] 09/02/06 20:14:19 [NameNode] INFO metrics.FSNamesystemMetrics : Initializing FSNamesystemMetrics using context object:org.apache.hadoop.metrics.spi.NullContext
[sf-startdaemon-debug] 09/02/06 20:14:19 [NameNode] INFO common.Storage : Image file of size 93 saved in 0 seconds.
[sf-startdaemon-debug] 09/02/06 20:14:19 [NameNode] INFO common.Storage : Storage directory /tmp/tempdir591 has been successfully formatted.
[sf-startdaemon-debug] 09/02/06 20:14:19 [NameNode] INFO common.Storage : Storage directory /tmp/hadoop/dfs/name has been successfully formatted.
[sf-startdaemon-debug] 09/02/06 20:14:19 [NameNode] INFO common.Storage : Number of files = 1
[sf-startdaemon-debug] 09/02/06 20:14:19 [NameNode] INFO common.Storage : Number of files under construction = 0
[sf-startdaemon-debug] 09/02/06 20:14:19 [NameNode] INFO common.Storage : Image file of size 93 loaded in 0 seconds.
[sf-startdaemon-debug] 09/02/06 20:14:19 [NameNode] INFO common.Storage : Edits file /tmp/hadoop/dfs/name/current/edits of size 4 edits # 0 loaded in 0 seconds.
[sf-startdaemon-debug] 09/02/06 20:14:19 [NameNode] INFO common.Storage : Image file of size 93 saved in 0 seconds.
[sf-startdaemon-debug] 09/02/06 20:14:20 [NameNode] INFO mortbay.log : Logging to org.slf4j.impl.JCLLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
[sf-startdaemon-debug] 09/02/06 20:14:20 [NameNode] INFO mortbay.log : jetty-6.1.14
[sf-startdaemon-debug] 09/02/06 20:14:20 [NameNode] INFO mortbay.log : Extract jar:file:/home/slo/.ivy/cache/org.apache.hadoop/hadoop-core/jars/hadoop-core-0.21.0-alpha-4.jar!/webapps/hdfs to /tmp/Jetty_localhost_8021_hdfs____xav36j/webapp
[sf-startdaemon-debug] 09/02/06 20:14:21 [NameNode] INFO mortbay.log : Started SelectChannelConnector@localhost:8021
[sf-startdaemon-debug] 09/02/06 20:14:21 [NameNode] INFO namenode.NameNode : Web-server up at: localhost:8021
[sf-startdaemon-debug] 09/02/06 20:14:21 [IPC Server Responder] INFO ipc.Server : IPC Server Responder: starting
[sf-startdaemon-debug] 09/02/06 20:14:21 [IPC Server listener on 8020] INFO ipc.Server : IPC Server listener on 8020: starting
[sf-startdaemon-debug] 09/02/06 20:14:21 [IPC Server handler 0 on 8020] INFO ipc.Server : IPC Server handler 0 on 8020: starting
[sf-startdaemon-debug] 09/02/06 20:14:21 [NameNode] INFO namenode.NameNode : State change: NameNode is now LIVE
[sf-startdaemon-debug] 09/02/06 20:14:21 [IPC Server handler 3 on 8020] INFO ipc.Server : IPC Server handler 3 on 8020: starting
[sf-startdaemon-debug] 09/02/06 20:14:21 [IPC Server handler 2 on 8020] INFO ipc.Server : IPC Server handler 2 on 8020: starting
[sf-startdaemon-debug] 09/02/06 20:14:21 [IPC Server handler 1 on 8020] INFO ipc.Server : IPC Server handler 1 on 8020: starting
[sf-startdaemon-debug] 09/02/06 20:14:21 [DataNode] INFO common.Storage : Storage directory /tmp/tempdir592 is not formatted.
[sf-startdaemon-debug] 09/02/06 20:14:21 [DataNode] INFO common.Storage : Formatting ...
[sf-startdaemon-debug] 2009/02/06 20:14:21:658 GMT [INFO ][NameNode] HOST k2:rootProcess:testJobSubmission - NameNode deployment complete: service is: ExtNameNode instance ExtNameNode instance org.apache.hadoop.hdfs.server.namenode.ExtNameNode@76ab2f in state LIVE in state LIVE at localhost/127.0.0.1:8021 , , IPC localhost/127.0.0.1:8020
[sf-startdaemon-debug] 09/02/06 20:14:21 [DataNode] INFO datanode.DataNode : Registered FSDatasetStatusMBean
[sf-startdaemon-debug] 09/02/06 20:14:21 [DataNode] INFO datanode.DataNode : Opened info server at 8024
[sf-startdaemon-debug] 09/02/06 20:14:21 [DataNode] INFO datanode.DataNode : Balancing bandwith is 1048576 bytes/s
[sf-startdaemon-debug] 09/02/06 20:14:21 [DataNode] INFO mortbay.log : jetty-6.1.14
[sf-startdaemon-debug] 09/02/06 20:14:22 [DataNode] INFO mortbay.log : Extract jar:file:/home/slo/.ivy/cache/org.apache.hadoop/hadoop-core/jars/hadoop-core-0.21.0-alpha-4.jar!/webapps/datanode to /tmp/Jetty_localhost_8022_datanode____.sbeinb/webapp
[sf-startdaemon-debug] 09/02/06 20:14:22 [DataNode] INFO mortbay.log : Started SelectChannelConnector@localhost:8022
[sf-startdaemon-debug] 09/02/06 20:14:22 [DataNode] INFO jvm.JvmMetrics : Cannot initialize JVM Metrics with processName=DataNode, sessionId=null - already initialized
[sf-startdaemon-debug] 09/02/06 20:14:27 [DataNode] INFO metrics.RpcMetrics : Initializing RPC Metrics with hostName=ExtDataNode, port=50020
[sf-startdaemon-debug] 09/02/06 20:14:27 [IPC Server Responder] INFO ipc.Server : IPC Server Responder: starting
[sf-startdaemon-debug] 09/02/06 20:14:27 [IPC Server handler 0 on 50020] INFO ipc.Server : IPC Server handler 0 on 50020: starting
[sf-startdaemon-debug] 09/02/06 20:14:27 [IPC Server handler 1 on 50020] INFO ipc.Server : IPC Server handler 1 on 50020: starting
[sf-startdaemon-debug] 09/02/06 20:14:27 [IPC Server listener on 50020] INFO ipc.Server : IPC Server listener on 50020: starting
[sf-startdaemon-debug] 09/02/06 20:14:27 [DataNode] INFO datanode.DataNode : dnRegistration = DatanodeRegistration(k2:8024, storageID=, infoPort=8022, ipcPort=50020)
[sf-startdaemon-debug] 09/02/06 20:14:27 [IPC Server handler 0 on 8020] INFO net.NetworkTopology : Adding a new node: /default-rack/127.0.0.1:8024
[sf-startdaemon-debug] 09/02/06 20:14:27 [DataNode] INFO datanode.DataNode : New storage id DS-1547360181-127.0.1.1-8024-1233951267801 is assigned to data-node 127.0.0.1:8024
[sf-startdaemon-debug] 09/02/06 20:14:27 [DataNode] INFO datanode.DataNode : State change: DataNode is now LIVE
[sf-startdaemon-debug] 09/02/06 20:14:27 [ExtDataNode] INFO datanode.DataNode : DatanodeRegistration(127.0.0.1:8024, storageID=DS-1547360181-127.0.1.1-8024-1233951267801, infoPort=8022, ipcPort=50020)In DataNode.run, data = FSDataset{dirpath='/tmp/tempdir592/current'}
[sf-startdaemon-debug] 09/02/06 20:14:27 [ExtDataNode] INFO datanode.DataNode : using BLOCKREPORT_INTERVAL of 10000msec Initial delay: 0msec
[sf-startdaemon-debug] 2009/02/06 20:14:27:884 GMT [INFO ][DataNode] HOST k2:rootProcess:testJobSubmission - DataNode deployment complete: service is: DataNode {data=FSDataset{dirpath='/tmp/tempdir592/current'}, localName='127.0.0.1:8024', storageID='DS-1547360181-127.0.1.1-8024-1233951267801', xmitsInProgress=0, state=LIVE}
[sf-startdaemon-debug] 09/02/06 20:14:28 [org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@12e7c6a] ERROR datanode.DataNode : DatanodeRegistration(127.0.0.1:8024, storageID=DS-1547360181-127.0.1.1-8024-1233951267801, infoPort=8022, ipcPort=50020):DataXceiveServer: Exiting due to:java.nio.channels.ClosedSelectorException
[sf-startdaemon-debug] at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:66)
[sf-startdaemon-debug] at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:88)
[sf-startdaemon-debug] at sun.nio.ch.Util.releaseTemporarySelector(Util.java:135)
[sf-startdaemon-debug] at sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:120)
[sf-startdaemon-debug] at org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:130)
[sf-startdaemon-debug] at java.lang.Thread.run(Thread.java:619)
[sf-startdaemon-debug]
[sf-startdaemon-debug] 09/02/06 20:14:28 [ExtDataNode] INFO datanode.DataNode : BlockReport of 0 blocks got processed in 130 msecs
[sf-startdaemon-debug] 09/02/06 20:14:28 [ExtDataNode] INFO datanode.DataNode : Starting Periodic block scanner.
[sf-startdaemon-debug] 09/02/06 20:14:28 [org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@1931579] INFO datanode.DataBlockScanner : Exiting DataBlockScanner thread.
[sf-startdaemon-debug] 09/02/06 20:14:28 [JobTracker] INFO mapred.ExtJobTracker : State change: JobTracker is now STARTED
[sf-startdaemon-debug] 09/02/06 20:14:28 [JobTracker] INFO metrics.RpcMetrics : Initializing RPC Metrics with hostName=ExtJobTracker, port=8012
[sf-startdaemon-debug] 09/02/06 20:14:28 [TaskTracker] INFO mapred.ExtTaskTracker : State change: TaskTracker is now STARTED
[sf-startdaemon-debug] 09/02/06 20:14:28 [ExtDataNode] INFO datanode.DataNode : DatanodeRegistration(127.0.0.1:8024, storageID=DS-1547360181-127.0.1.1-8024-1233951267801, infoPort=8022, ipcPort=50020):Finishing DataNode in: FSDataset{dirpath='/tmp/tempdir592/current'}
[sf-startdaemon-debug] 09/02/06 20:14:28 [ExtDataNode] INFO datanode.DataNode : State change: DataNode is now CLOSED
[sf-startdaemon-debug] 09/02/06 20:14:28 [ExtDataNode] INFO datanode.DataNode : Terminating ExtDataNode
[sf-startdaemon-debug] 2009/02/06 20:14:28:900 GMT [INFO ][TerminatorThread] HOST k2:rootProcess:testJobSubmission - Initiating DataNode termination; serviceThread=Thread[DataNode,5,] service=DataNode {data=FSDataset{dirpath='/tmp/tempdir592/current'}, localName='127.0.0.1:8024', storageID='DS-1547360181-127.0.1.1-8024-1233951267801', xmitsInProgress=0, state=CLOSED}
[sf-startdaemon-debug] 2009/02/06 20:14:28:925 GMT [INFO ][TerminatorThread] HOST k2:rootProcess:testJobSubmission - Terminating deployer thread
[sf-startdaemon-debug] 2009/02/06 20:14:28:926 GMT [INFO ][TerminatorThread] HOST k2:rootProcess:testJobSubmission - Requesting thread termination
[sf-startdaemon-debug] 2009/02/06 20:14:28:929 GMT [INFO ][TerminatorThread] HOST k2:rootProcess:testJobSubmission - waiting for thread to finish
[sf-startdaemon-debug] 2009/02/06 20:14:28:930 GMT [WARN ][TerminatorThread] HOST k2:rootProcess:testJobSubmission - Hadoop Service thread did not terminate within the expected shutdown period. Service is DataNode {data=FSDataset{dirpath='/tmp/tempdir592/current'}, localName='127.0.0.1:8024', storageID='DS-1547360181-127.0.1.1-8024-1233951267801', xmitsInProgress=0, state=CLOSED}
[sf-startdaemon-debug] 09/02/06 20:14:28 [ExtDataNode] INFO ipc.Server : Stopping server on 50020
[sf-startdaemon-debug] 09/02/06 20:14:28 [IPC Server handler 1 on 50020] INFO ipc.Server : IPC Server handler 1 on 50020: exiting
[sf-startdaemon-debug] 09/02/06 20:14:28 [IPC Server listener on 50020] INFO ipc.Server : Stopping IPC Server listener on 50020
[sf-startdaemon-debug] 2009/02/06 20:14:28:933 GMT [INFO ][TerminatorThread] HOST k2:rootProcess:testJobSubmission - Terminating hadoopService service ExtDataNode
[sf-startdaemon-debug] 09/02/06 20:14:28 [IPC Server handler 0 on 50020] INFO ipc.Server : IPC Server handler 0 on 50020: exiting
[sf-startdaemon-debug] 09/02/06 20:14:28 [IPC Server Responder] INFO ipc.Server : Stopping IPC Server Responder
[sf-startdaemon-debug] 09/02/06 20:14:28 [ExtDataNode] INFO datanode.DataNode : Waiting for threadgroup to exit, active threads is 0
[sf-startdaemon-debug] 2009/02/06 20:14:29:075 GMT [ERROR][TerminatorThread] HOST k2:rootProcess:testJobSubmission - Unexpected termination of action
[sf-startdaemon-debug] 2009/02/06 20:14:29:076 GMT [ERROR][TerminatorThread] HOST k2:rootProcess:testJobSubmission - Termination Record: HOST k2:rootProcess:testJobSubmission:action:datanode, type: abnormal, description: Service has halted
[sf-startdaemon-debug] 2009/02/06 20:14:29:137 GMT [INFO ][TerminatorThread] HOST k2:rootProcess:testJobSubmission - Initiating NameNode termination; serviceThread=Thread[NameNode,5,] service=ExtNameNode instance ExtNameNode instance org.apache.hadoop.hdfs.server.namenode.ExtNameNode@76ab2f in state LIVE in state LIVE at localhost/127.0.0.1:8021 , , IPC localhost/127.0.0.1:8020
[sf-startdaemon-debug] 2009/02/06 20:14:29:138 GMT [INFO ][TerminatorThread] HOST k2:rootProcess:testJobSubmission - Terminating deployer thread
[sf-startdaemon-debug] 2009/02/06 20:14:29:138 GMT [INFO ][TerminatorThread] HOST k2:rootProcess:testJobSubmission - Requesting thread termination
[sf-startdaemon-debug] 2009/02/06 20:14:29:139 GMT [INFO ][TerminatorThread] HOST k2:rootProcess:testJobSubmission - waiting for thread to finish
[sf-startdaemon-debug] 2009/02/06 20:14:29:140 GMT [WARN ][TerminatorThread] HOST k2:rootProcess:testJobSubmission - Hadoop Service thread did not terminate within the expected shutdown period. Service is ExtNameNode instance ExtNameNode instance org.apache.hadoop.hdfs.server.namenode.ExtNameNode@76ab2f in state LIVE in state LIVE at localhost/127.0.0.1:8021 , , IPC localhost/127.0.0.1:8020
[sf-startdaemon-debug] 2009/02/06 20:14:29:141 GMT [INFO ][TerminatorThread] HOST k2:rootProcess:testJobSubmission - Terminating hadoopService service ExtNameNode
[sf-startdaemon-debug] 09/02/06 20:14:29 [TerminatorThread] INFO namenode.NameNode : State change: NameNode is now CLOSED
[sf-startdaemon-debug] 09/02/06 20:14:29 [TerminatorThread] INFO namenode.NameNode : Closing NameNode
[sf-startdaemon-debug] 2009/02/06 20:14:29:231 GMT [INFO ][TerminatorThread] HOST k2:rootProcess:testJobSubmission - Initiating TaskTracker termination; serviceThread=Thread[TaskTracker,5,RMI Runtime] service=Task Tracker instance org.apache.hadoop.mapred.ExtTaskTracker@11b86c7 in state STARTED; web port=0
[sf-startdaemon-debug] 2009/02/06 20:14:29:232 GMT [INFO ][TerminatorThread] HOST k2:rootProcess:testJobSubmission - Terminating deployer thread
[sf-startdaemon-debug] 2009/02/06 20:14:29:232 GMT [INFO ][TerminatorThread] HOST k2:rootProcess:testJobSubmission - Requesting thread termination
[sf-startdaemon-debug] 2009/02/06 20:14:29:234 GMT [INFO ][TerminatorThread] HOST k2:rootProcess:testJobSubmission - waiting for thread to finish
[sf-startdaemon-debug] 09/02/06 20:14:29 [org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@fdb00d] WARN namenode.FSNamesystem : ReplicationMonitor thread received InterruptedException.java.lang.InterruptedException: sleep interrupted
[sf-startdaemon-debug] 09/02/06 20:14:29 [org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@1c65216] INFO namenode.DecommissionManager : Interrupted Monitor
[sf-startdaemon-debug] java.lang.InterruptedException: sleep interrupted
[sf-startdaemon-debug] at java.lang.Thread.sleep(Native Method)
[sf-startdaemon-debug] at org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
[sf-startdaemon-debug] at java.lang.Thread.run(Thread.java:619)
[sf-startdaemon-debug] 2009/02/06 20:14:29:309 GMT [INFO ][TerminatorThread] HOST k2:rootProcess:testJobSubmission - Initiating JobTracker termination; serviceThread=Thread[JobTracker,5,RMI Runtime] service=JobTracker instance org.apache.hadoop.mapred.ExtJobTracker@7e75d2 in state STARTED
[sf-startdaemon-debug] 2009/02/06 20:14:29:310 GMT [INFO ][TerminatorThread] HOST k2:rootProcess:testJobSubmission - Terminating deployer thread
[sf-startdaemon-debug] 2009/02/06 20:14:29:311 GMT [INFO ][TerminatorThread] HOST k2:rootProcess:testJobSubmission - Requesting thread termination
[sf-startdaemon-debug] 2009/02/06 20:14:29:336 GMT [INFO ][TerminatorThread] HOST k2:rootProcess:testJobSubmission - waiting for thread to finish
[sf-startdaemon-debug] 09/02/06 20:14:29 [TerminatorThread] INFO ipc.Server : Stopping server on 8020
[sf-startdaemon-debug] 09/02/06 20:14:29 [IPC Server handler 0 on 8020] INFO ipc.Server : IPC Server handler 0 on 8020: exiting
[sf-startdaemon-debug] 09/02/06 20:14:29 [IPC Server handler 1 on 8020] INFO ipc.Server : IPC Server handler 1 on 8020: exiting
[sf-startdaemon-debug] 09/02/06 20:14:29 [IPC Server handler 2 on 8020] INFO ipc.Server : IPC Server handler 2 on 8020: exiting
[sf-startdaemon-debug] 09/02/06 20:14:29 [IPC Server handler 3 on 8020] INFO ipc.Server : IPC Server handler 3 on 8020: exiting
[sf-startdaemon-debug] 09/02/06 20:14:29 [IPC Server listener on 8020] INFO ipc.Server : Stopping IPC Server listener on 8020
[sf-startdaemon-debug] 09/02/06 20:14:29 [IPC Server Responder] INFO ipc.Server : Stopping IPC Server Responder
[sf-system-test-junit] (unknown) -TestCompletedEvent at Fri Feb 06 20:14:29 GMT 2009 alive: true
[sf-system-test-junit] status:
[sf-system-test-junit] Termination Record: HOST k2:rootProcess:testJobSubmission:action:datanode, type: abnormal, description: Service has halted
[sf-system-test-junit] Submit a job to a cluster and copy out the result
[sf-system-test-junit]
[sf-system-test-junit] succeeded:false
[sf-system-test-junit] forcedTimeout:false
[sf-system-test-junit] skipped:false
[sf-system-test-junit] )
[sf-system-test-junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 28.062 sec
[sf-system-test-junit]
[sf-system-test-junit] Testcase: testJobSubmission took 27.91 sec
[sf-system-test-junit] Caused an ERROR
[sf-system-test-junit] Test failed
[sf-system-test-junit] (unknown) -TestCompletedEvent at Fri Feb 06 20:14:29 GMT 2009 alive: true
[sf-system-test-junit] status:
[sf-system-test-junit] Termination Record: HOST k2:rootProcess:testJobSubmission:action:datanode, type: abnormal, description: Service has halted
[sf-system-test-junit] Submit a job to a cluster and copy out the result
[sf-system-test-junit]
[sf-system-test-junit] succeeded:false
[sf-system-test-junit] forcedTimeout:false
[sf-system-test-junit] skipped:false
[sf-system-test-junit]
[sf-system-test-junit] org.smartfrog.test.TerminationRecordException: Test failed
[sf-system-test-junit] (unknown) -TestCompletedEvent at Fri Feb 06 20:14:29 GMT 2009 alive: true
[sf-system-test-junit] status:
[sf-system-test-junit] Termination Record: HOST k2:rootProcess:testJobSubmission:action:datanode, type: abnormal, description: Service has halted
[sf-system-test-junit] Submit a job to a cluster and copy out the result
[sf-system-test-junit]
[sf-system-test-junit] succeeded:false
[sf-system-test-junit] forcedTimeout:false
[sf-system-test-junit] skipped:false
[sf-system-test-junit] Termination Record: HOST k2:rootProcess:testJobSubmission:action:datanode, type: abnormal, description: Service has halted
[sf-system-test-junit] at org.smartfrog.test.DeployingTestBase.completeTestDeployment(DeployingTestBase.java:315)
[sf-system-test-junit] at org.smartfrog.test.DeployingTestBase.runTestsToCompletion(DeployingTestBase.java:338)
[sf-system-test-junit] at org.smartfrog.test.DeployingTestBase.expectSuccessfulTestRunOrSkip(DeployingTestBase.java:439)
[sf-system-test-junit] at org.smartfrog.services.hadoop.test.system.local.tracker.JobSubmissionTest.testJobSubmission(JobSubmissionTest.java:40)
[sf-system-test-junit]
[sf-system-test-junit] Test org.smartfrog.services.hadoop.test.system.local.tracker.JobSubmissionTest FAILED

Steve Loughran added a comment - 06/Feb/09 08:32 PM (GMT)
tweaked this; think it is as should be.
-failures are always trouble
-termination without going through a terminateService call may or may not be unexpected