append operation in hadoop webhdfs client
hadoop, java, python, webhdfs
Solution
Assuming your user name is `hdfs`, add `&user.name=hdfs` to your URL. Write operations require a valid user.
Your java code works because it pulls your user information from the unix environment.
If you see the user `dr.who` anywhere, it's probably because you haven't set a `user.name` in your request.
Problem
A Java client I threw together works: ``` import java.io.File; import java.io.IOException; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.FSDataOutputStream; import org.apache.hadoop.fs.Path; public class HdfsAppend { public static final String hdfs = "hdfs://my222host.com"; public static final String hpath = "/tmp/odp/testfile"; public static final String message = "Hello, world!\n"; public static void main(String[] args) throws IOException { Configuration conf = new Configuration(); conf.set("fs.defaultFS", hdfs); FileSystem fs = FileSystem.get(conf); Path filenamePath = new Path(hpath); FSDataOutputStream out = fs.append(filenamePath); out.writeBytes("DUPA DUPA DUPA\n"); } } ``` But both curl and Python whoops client fail in similar way, curl here: ``` curl -i -X POST "http://my222host:50070/webhdfs/v1/tmp/odp/testfile?op=APPEND" HTTP/1.1 307 TEMPORARY_REDIRECT Cache-Control: no-cache Expires: Tue, 13 Aug 2013 13:26:22 GMT Date: Tue, 13 Aug 2013 13:26:22 GMT Pragma: no-cache Expires: Tue, 13 Aug 2013 13:26:22 GMT Date: Tue, 13 Aug 2013 13:26:22 GMT Pragma: no-cache Content-Type: application/octet-stream Location: http://my333host:50075/webhdfs/v1/tmp/odp/testfile?op=APPEND&namenoderpcaddress=my222host:8020 Content-Length: 0 Server: Jetty(6.1.26.cloudera.2) curl -i -X POST -T /tmp/abc "http://my333host:50075/webhdfs/v1/tmp/odp/testfile?op=APPEND&namenoderpcaddress=my222host:8020" HTTP/1.1 100 Continue HTTP/1.1 403 Forbidden Cache-Control: no-cache Expires: Tue, 13 Aug 2013 13:26:26 GMT Date: Tue, 13 Aug 2013 13:26:26 GMT Pragma: no-cache Expires: Tue, 13 Aug 2013 13:26:26 GMT Date: Tue, 13 Aug 2013 13:26:26 GMT Pragma: no-cache Content-Type: application/json Transfer-Encoding: chunked Server: Jetty(6.1.26.cloudera.2) {"RemoteException":{"exception":"AccessControlException","javaClassName":"org.apache.hadoop.security.AccessControlException","message":"Permission denied: user=dr.who, access=WRITE, inode=\"/tmp/odp/testfile\":root:hadoop:-rw-r--r--\n\tat org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:224)\n\tat org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:155)\n\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:4716)\n\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:4698)\n\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:4660)\n\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:1837)\n\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInt(FSNamesystem.java:2105)\n\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:2081)\n\tat org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(NameNodeRpcServer.java:434)\n\tat org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.append(ClientNamenodeProtocolServerSideTranslatorPB.java:224)\n\tat org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:44944)\n\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)\n\tat org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1002)\n\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1701)\n\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1697)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat javax.security.auth.Subject.doAs(Subject.java:396)\n\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)\n\tat org.apache.hadoop.ipc.Server$Handler.run(Server.java:1695)\n"}} ``` whoops client fails with "connection refused". What can be the problem here? The only clue I have is "user=dr.who" in java exception when using curl, but I have no idea what is the user used by Configuration class or how to get it (if that is the root of the problem). Pls help!