Create new file operation is used to create a file to the directory. Following interfaces can be used to add file in the directory.
Apache Hadoop
and java.io
classes are used to create file in the directory.
import java.io.File; import java.io.FileInputStream; import java.io.IOException; import java.io.InputStream; import java.io.OutputStream; import org.apache.commons.io.IOUtils; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.hdfs.DFSConfigKeys; public class PutObject { /* * This program reads a file from the local file system and saves it on HDFS. */ public static void main(String[] args) throws IOException{ Configuration conf = new Configuration(true); //Create a configuration object to define hdfs properties conf.set(DFSConfigKeys.FS_DEFAULT_NAME_KEY, "hdfs://192.168.0.1:9000"); // URL for your namenode conf.set(DFSConfigKeys.DFS_REPLICATION_KEY, "3"); // Replication count for files you write OutputStream os = null; InputStream is = null; try{ // First initialize DFS FileSystem object FileSystem dfs = FileSystem.get(conf); //Hadoop FileSystem object with QueryIO configuration dfs.mkdirs(new Path("/queryio/demo/")); //created a new directory "demo" os = dfs.create(new Path("/queryio/demo/file1.txt")); //return a OutputStream at indicated path is = new FileInputStream(new File("/local/queryio.txt")); //InputStream from a local filesystem file IOUtils.copy(is, os); //copy bytes from InputStream to OutputStream : create Object Operation } finally { try{ if(is!=null) is.close(); //close InputStream } catch(Exception e){ e.printStackTrace(); } try{ if(os!=null) os.close(); //close OutputStream } catch(Exception e){ e.printStackTrace(); } } } }
Following version of curl
command is used for creating new file.
Step 1: Submit a HTTP PUT request without automatically following redirects and without sending the file data. Request is redirected to datanode where data is to be written.
curl -i -X PUT "http://<HOST>:<PORT>/<PATH>?user.name=<username>&op=CREATE
[&overwrite=<true|false>][&blocksize=<LONG>][&replication=<SHORT>]
[&permission=<OCTAL>][&buffersize=<INT>]"
-i option
: Include the HTTP-header in the output like server-name, date of the document, HTTP-version etc. -X option
: (HTTP) Specifies a custom request method to use when communicating with the HTTP server.PUT
: Submit HTTP PUT request.<HOST>
: Hostname where file has to be created.<PORT>
: Port on which server is working.<PATH>
: A valid path of the file.<DATANODE>:<PORT>
: Datanode host and port.user.name=<username>
: QueryIO account username for authentication.op=CREATE
: Create new file.[&overwrite=<true|false>]
: (Optional)If file already exists, should it be overwritten.[&blocksize=<LONG>]
: (Optional) Block size of file.[&replication=<SHORT>]
: (Optional) The number of replications of a file.[&permission=<OCTAL>]
: (Optional) Permissions to be granted on file in octal format. By default permissions granted to file is 0755.[&buffersize=<INT>]
: (Optional) The size of the buffer used in transferring data.Step 2: Submit another HTTP PUT request using the URL in the location header with the file data to be written.
curl -i -X PUT -T <LOCAL_FILE> "http://<DATANODE>:<PORT>/<PATH>?user.name=<username>op=CREATE..."
-T option
: Upload file.<LOCAL_FILE>
: Local file to be uploaded to hdfs.<DATANODE>:<PORT>
: Datanode host and port.op=CREATE
: create new file.Step 1: Sample Request: curl -i -X PUT "http://192.168.0.1:50070/webhdfs/v1/queryio/demo/file1.txt?user.name=admin&op=CREATE&overwrite=false" HTTP Request: PUT /webhdfs/v1/queryio/demo/file1.txt?user.name=admin&op=CREATE&overwrite=false HTTP/1.1 User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5 Host: 192.168.0.16:50070 Accept: */* HTTPResponse: HTTP/1.1 307 TEMPORARY_REDIRECT Location: http://server.local:50075/webhdfs/v1/queryio/demo/file1.txt?op=CREATE&user.name=admin&namenoderpcaddress=server.local:9000&overwrite=false Content-Type: application/octet-stream Content-Length: 0 Server: Jetty(6.1.26) Step 2: Sample Request: curl -i -X PUT -T /local/queryio.txt "http://server.local:50075/webhdfs/v1/queryio/demo/file1.txt?op=CREATE&user.name=admin&namenoderpcaddress=server.local:9000&overwrite=false" HTTP Request: PUT /webhdfs/v1/queryio/demo/file1.txt?op=CREATE&user.name=admin&namenoderpcaddress=server.local:9000&overwrite=false HTTP/1.1 User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5 Host: server.local:50075 Accept: */* Content-Length: 16011 Expect: 100-continue HTTP Response: HTTP/1.1 100 Continue HTTP/1.1 201 Created Location: webhdfs://0.0.0.0:50070/queryio/demo/file1.txt Content-Type: application/octet-stream Content-Length: 0 Server: Jetty(6.1.26)
The client receives a 201 Created response code and the WebHDFS URI of the file in the Location header