Issue Details (XML | Word | Printable)

Key: SFOS-1124
Type: Bug Bug
Status: Resolved Resolved
Resolution: Fixed
Priority: Major Major
Assignee: Steve Loughran
Reporter: Steve Loughran
Votes: 0
Watchers: 0
Operations

If you were logged in you would be able to see more operations.
SmartFrog

Add component to do directory copy between any two supported filesystems

Created: 24/Feb/09 02:18 PM (GMT)   Updated: 25/Feb/09 04:36 PM (GMT)
Component/s: _service_hadoop
Affects Version/s: 3.17.010
Fix Version/s: 3.17.010

Time Tracking:
Not Specified

Compatibility: backwards compatible


 Description  « Hide
At the end of a job, we want to copy out everything

  CopyDataOut extends DfsCopyFileOutWorkflow {
    cluster LAZY PARENT:cluster;
    source mapred.output.dir;
    dest outputDirLocal;
  }

but it is failing as the operations only work for a single fule.

at org.smartfrog.services.hadoop.components.dfs.DfsCopyFileOutImpl.performDfsOperation(DfsCopyFileOutImpl.java:75)
at org.smartfrog.services.hadoop.components.dfs.DfsOperationImpl.performDfsOperation(DfsOperationImpl.java:97)
at org.smartfrog.services.hadoop.components.dfs.DfsOperationImpl$DfsWorkerThread.execute(DfsOperationImpl.java:146)
at org.smartfrog.sfcore.utils.SmartFrogThread.run(SmartFrogThread.java:279)
at org.smartfrog.sfcore.utils.WorkflowThread.run(WorkflowThread.java:117)
Caused by: java.io.IOException: Target file:/home/slo/Projects/SmartFrog/Forge/core/components/hadoop/build/test/work/out/out is a directory
at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:352)
at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:354)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:201)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:191)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:142)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1175)
at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1156)
at org.smartfrog.services.hadoop.components.dfs.DfsCopyFileOutImpl.performDfsOperation(DfsCopyFileOutImpl.java:73)

 All   Comments   Work Log   Change History      Sort Order: Ascending order - Click to sort in descending order
[ Permlink | « Hide ]
Steve Loughran added a comment - 25/Feb/09 12:01 PM (GMT)
Hadoop has the copymerge operation to do this; the main reason to do it ourselves would be to handle termination mid-copy better and support patterns for the copy

[ Permlink | « Hide ]
Steve Loughran added a comment - 25/Feb/09 12:02 PM (GMT)
The current code assumes every URL is to an HDFS filesystem, fails on file:///

[sf-system-test-junit] SFHadoopException:: Failed to initialise filesystem, cause: java.io.IOException: Incomplete HDFS URI, no host: file:///, SmartFrog 3.17.005dev (2009-02-13 11:47:21 GMT)
[sf-system-test-junit] at org.smartfrog.services.hadoop.core.SFHadoopException.forward(SFHadoopException.java:279)
[sf-system-test-junit] at org.smartfrog.services.hadoop.common.DfsUtils.createFileSystem(DfsUtils.java:140)
[sf-system-test-junit] at org.smartfrog.services.hadoop.components.dfs.DfsCopyFilesImpl.performDfsOperation(DfsCopyFilesImpl.java:105)
[sf-system-test-junit] at org.smartfrog.services.hadoop.components.dfs.DfsOperationImpl$DfsWorkerThread.execute(DfsOperationImpl.java:115)
[sf-system-test-junit] at org.smartfrog.sfcore.utils.SmartFrogThread.run(SmartFrogThread.java:279)
[sf-system-test-junit] at org.smartfrog.sfcore.utils.WorkflowThread.run(WorkflowThread.java:117)
[sf-system-test-junit] Caused by: java.io.IOException: Incomplete HDFS URI, no host: file:///
[sf-system-test-junit] at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:77)
[sf-system-test-junit] at org.smartfrog.services.hadoop.common.DfsUtils.createFileSystem(DfsUtils.java:138)
[sf-system-test-junit] at org.smartfrog.test.DeployingTestBase.completeTestDeployment(DeployingTestBase.java:315)
[sf-system-test-junit] at org.smartfrog.test.DeployingTestBase.runTestsToCompletion(DeployingTestBase.java:338)
[sf-system-test-junit] at org.smartfrog.test.DeployingTestBase.expectSuccessfulTestRunOrSkip(DeployingTestBase.java:439)
[sf-system-test-junit] at org.smartfrog.services.hadoop.test.system.local.tracker.JobSubmissionTest.testJobSubmission(JobSubmissionTest.java:39)
[sf-system-test-junit] Caused by: SFHadoopException:: Failed to initialise filesystem, cause: java.io.IOException: Incomplete HDFS URI, no host: file:///, SmartFrog 3.17.005dev (2009-02-13 11:47:21 GMT)
[sf-system-test-junit] at org.smartfrog.services.hadoop.core.SFHadoopException.forward(SFHadoopException.java:279)
[sf-system-test-junit] at org.smartfrog.services.hadoop.common.DfsUtils.createFileSystem(DfsUtils.java:140)
[sf-system-test-junit] at org.smartfrog.services.hadoop.components.dfs.DfsCopyFilesImpl.performDfsOperation(DfsCopyFilesImpl.java:105)
[sf-system-test-junit] at org.smartfrog.services.hadoop.components.dfs.DfsOperationImpl$DfsWorkerThread.execute(DfsOperationImpl.java:115)
[sf-system-test-junit] at org.smartfrog.sfcore.utils.SmartFrogThread.run(SmartFrogThread.java:279)
[sf-system-test-junit] at org.smartfrog.sfcore.utils.WorkflowThread.run(WorkflowThread.java:117)
[sf-system-test-junit] Caused by: java.io.IOException: Incomplete HDFS URI, no host: file:///
[sf-system-test-junit] at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:77)
[sf-system-test-junit] at org.smartfrog.services.hadoop.common.DfsUtils.createFileSystem(DfsUtils.java:138)
[sf-system-test-junit] Test org.smartfrog.services.hadoop.test.system.local.tracker.JobSubmissionTest FAILED

[ Permlink | « Hide ]
Steve Loughran added a comment - 25/Feb/09 04:36 PM (GMT)
done!


Powered by a free Atlassian JIRA open source license for SmartFrog. Try JIRA - bug tracking software for your team.
Atlassian JIRA the Professional Issue Tracker. (Professional Edition, Version: 3.13.1-#333) - Bug/feature request - Atlassian news - Contact Administrators