|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Objectorg.apache.hadoop.conf.Configured
org.apache.hadoop.fs.FileSystem
public abstract class FileSystem
An abstract base class for a fairly generic filesystem. It may be implemented as a distributed filesystem, or as a "local" one that reflects the locally-connected disk. The local version exists for small Hadoop instances and for testing.
All user code that may potentially use the Hadoop Distributed File System should be written to use a FileSystem object. The Hadoop DFS is a multi-machine system that appears as a single disk. It's useful because of its fault tolerance and potentially very large capacity.
The local implementation is LocalFileSystem
and distributed
implementation is DistributedFileSystem.
Nested Class Summary | |
---|---|
static class |
FileSystem.Statistics
|
Field Summary | |
---|---|
static org.apache.commons.logging.Log |
LOG
|
protected FileSystem.Statistics |
statistics
The statistics for this file system. |
Constructor Summary | |
---|---|
protected |
FileSystem()
|
Method Summary | |
---|---|
FSDataOutputStream |
append(Path f)
Append to an existing file (optional operation). |
FSDataOutputStream |
append(Path f,
int bufferSize)
Append to an existing file (optional operation). |
abstract FSDataOutputStream |
append(Path f,
int bufferSize,
Progressable progress)
Append to an existing file (optional operation). |
protected void |
checkPath(Path path)
Check that a Path belongs to this FileSystem. |
static void |
clearStatistics()
|
void |
close()
No more filesystem operations are needed. |
static void |
closeAll()
Close all cached filesystems. |
void |
completeLocalOutput(Path fsOutputFile,
Path tmpLocalFile)
Called when we're all done writing to the target. |
void |
copyFromLocalFile(boolean delSrc,
boolean overwrite,
Path[] srcs,
Path dst)
The src files are on the local disk. |
void |
copyFromLocalFile(boolean delSrc,
boolean overwrite,
Path src,
Path dst)
The src file is on the local disk. |
void |
copyFromLocalFile(boolean delSrc,
Path src,
Path dst)
The src file is on the local disk. |
void |
copyFromLocalFile(Path src,
Path dst)
The src file is on the local disk. |
void |
copyToLocalFile(boolean delSrc,
Path src,
Path dst)
The src file is under FS, and the dst is on the local disk. |
void |
copyToLocalFile(Path src,
Path dst)
The src file is under FS, and the dst is on the local disk. |
static FSDataOutputStream |
create(FileSystem fs,
Path file,
FsPermission permission)
create a file with the provided permission The permission of the file is set to be the provided permission as in setPermission, not permission&~umask It is implemented using two RPCs. |
FSDataOutputStream |
create(Path f)
Opens an FSDataOutputStream at the indicated Path. |
FSDataOutputStream |
create(Path f,
boolean overwrite)
Opens an FSDataOutputStream at the indicated Path. |
FSDataOutputStream |
create(Path f,
boolean overwrite,
int bufferSize)
Opens an FSDataOutputStream at the indicated Path. |
FSDataOutputStream |
create(Path f,
boolean overwrite,
int bufferSize,
Progressable progress)
Opens an FSDataOutputStream at the indicated Path with write-progress reporting. |
FSDataOutputStream |
create(Path f,
boolean overwrite,
int bufferSize,
short replication,
long blockSize)
Opens an FSDataOutputStream at the indicated Path. |
FSDataOutputStream |
create(Path f,
boolean overwrite,
int bufferSize,
short replication,
long blockSize,
Progressable progress)
Opens an FSDataOutputStream at the indicated Path with write-progress reporting. |
abstract FSDataOutputStream |
create(Path f,
FsPermission permission,
boolean overwrite,
int bufferSize,
short replication,
long blockSize,
Progressable progress)
Opens an FSDataOutputStream at the indicated Path with write-progress reporting. |
FSDataOutputStream |
create(Path f,
Progressable progress)
Create an FSDataOutputStream at the indicated Path with write-progress reporting. |
FSDataOutputStream |
create(Path f,
short replication)
Opens an FSDataOutputStream at the indicated Path. |
FSDataOutputStream |
create(Path f,
short replication,
Progressable progress)
Opens an FSDataOutputStream at the indicated Path with write-progress reporting. |
boolean |
createNewFile(Path f)
Creates the given Path as a brand-new zero-length file. |
abstract boolean |
delete(Path f)
Deprecated. Use delete(Path, boolean) instead |
abstract boolean |
delete(Path f,
boolean recursive)
Delete a file. |
boolean |
deleteOnExit(Path f)
Mark a path to be deleted when FileSystem is closed. |
boolean |
exists(Path f)
Check if exists. |
static FileSystem |
get(Configuration conf)
Returns the configured filesystem implementation. |
static FileSystem |
get(java.net.URI uri,
Configuration conf)
Returns the FileSystem for this URI's scheme and authority. |
static java.util.List<FileSystem.Statistics> |
getAllStatistics()
Return the FileSystem classes that have Statistics |
long |
getBlockSize(Path f)
Deprecated. Use getFileStatus() instead |
ContentSummary |
getContentSummary(Path f)
Return the ContentSummary of a given Path . |
long |
getDefaultBlockSize()
Return the number of bytes that large input files should be optimally be split into to minimize i/o time. |
short |
getDefaultReplication()
Get the default replication. |
static java.net.URI |
getDefaultUri(Configuration conf)
Get the default filesystem URI from a configuration. |
BlockLocation[] |
getFileBlockLocations(FileStatus file,
long start,
long len)
Return an array containing hostnames, offset and size of portions of the given file. |
FileChecksum |
getFileChecksum(Path f)
Get the checksum of a file. |
abstract FileStatus |
getFileStatus(Path f)
Return a file status object that represents the path. |
Path |
getHomeDirectory()
Return the current user's home directory in this filesystem. |
long |
getLength(Path f)
Deprecated. Use getFileStatus() instead |
static LocalFileSystem |
getLocal(Configuration conf)
Get the local file syste |
java.lang.String |
getName()
Deprecated. call #getUri() instead. |
static FileSystem |
getNamed(java.lang.String name,
Configuration conf)
Deprecated. call #get(URI,Configuration) instead. |
short |
getReplication(Path src)
Deprecated. Use getFileStatus() instead |
static java.util.Map<java.lang.String,FileSystem.Statistics> |
getStatistics()
Deprecated. use getAllStatistics() instead |
static FileSystem.Statistics |
getStatistics(java.lang.String scheme,
java.lang.Class<? extends FileSystem> cls)
Get the statistics for a particular file system |
abstract java.net.URI |
getUri()
Returns a URI whose scheme and authority identify this FileSystem. |
long |
getUsed()
Return the total size of all files in the filesystem. |
abstract Path |
getWorkingDirectory()
Get the current working directory for the given file system |
FileStatus[] |
globStatus(Path pathPattern)
Return all the files that match filePattern and are not checksum files. |
FileStatus[] |
globStatus(Path pathPattern,
PathFilter filter)
Return an array of FileStatus objects whose path names match pathPattern and is accepted by the user-supplied path filter. |
void |
initialize(java.net.URI name,
Configuration conf)
Called after a new FileSystem instance is constructed. |
boolean |
isDirectory(Path f)
Deprecated. Use getFileStatus() instead |
boolean |
isFile(Path f)
True iff the named path is a regular file. |
abstract FileStatus[] |
listStatus(Path f)
List the statuses of the files/directories in the given path if the path is a directory. |
FileStatus[] |
listStatus(Path[] files)
Filter files/directories in the given list of paths using default path filter. |
FileStatus[] |
listStatus(Path[] files,
PathFilter filter)
Filter files/directories in the given list of paths using user-supplied path filter. |
FileStatus[] |
listStatus(Path f,
PathFilter filter)
Filter files/directories in the given path using the user-supplied path filter. |
Path |
makeQualified(Path path)
Make sure that a path specifies a FileSystem. |
static boolean |
mkdirs(FileSystem fs,
Path dir,
FsPermission permission)
create a directory with the provided permission The permission of the directory is set to be the provided permission as in setPermission, not permission&~umask |
boolean |
mkdirs(Path f)
Call mkdirs(Path, FsPermission) with default permission. |
abstract boolean |
mkdirs(Path f,
FsPermission permission)
Make the given file and all non-existent parents into directories. |
void |
moveFromLocalFile(Path[] srcs,
Path dst)
The src files is on the local disk. |
void |
moveFromLocalFile(Path src,
Path dst)
The src file is on the local disk. |
void |
moveToLocalFile(Path src,
Path dst)
The src file is under FS, and the dst is on the local disk. |
FSDataInputStream |
open(Path f)
Opens an FSDataInputStream at the indicated Path. |
abstract FSDataInputStream |
open(Path f,
int bufferSize)
Opens an FSDataInputStream at the indicated Path. |
static void |
printStatistics()
|
protected void |
processDeleteOnExit()
Delete all files that were marked as delete-on-exit. |
abstract boolean |
rename(Path src,
Path dst)
Renames Path src to Path dst. |
static void |
setDefaultUri(Configuration conf,
java.lang.String uri)
Set the default filesystem URI in a configuration. |
static void |
setDefaultUri(Configuration conf,
java.net.URI uri)
Set the default filesystem URI in a configuration. |
void |
setOwner(Path p,
java.lang.String username,
java.lang.String groupname)
Set owner of a path (i.e. |
void |
setPermission(Path p,
FsPermission permission)
Set permission of a path. |
boolean |
setReplication(Path src,
short replication)
Set replication for an existing file. |
void |
setTimes(Path p,
long mtime,
long atime)
Set access time of a file |
void |
setVerifyChecksum(boolean verifyChecksum)
Set the verify checksum flag. |
abstract void |
setWorkingDirectory(Path new_dir)
Set the current working directory for the given file system. |
Path |
startLocalOutput(Path fsOutputFile,
Path tmpLocalFile)
Returns a local File that the user can write output to. |
Methods inherited from class org.apache.hadoop.conf.Configured |
---|
getConf, setConf |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Field Detail |
---|
public static final org.apache.commons.logging.Log LOG
protected FileSystem.Statistics statistics
Constructor Detail |
---|
protected FileSystem()
Method Detail |
---|
public static FileSystem get(Configuration conf) throws java.io.IOException
java.io.IOException
public static java.net.URI getDefaultUri(Configuration conf)
conf
- the configuration to access
public static void setDefaultUri(Configuration conf, java.net.URI uri)
conf
- the configuration to alteruri
- the new default filesystem uripublic static void setDefaultUri(Configuration conf, java.lang.String uri)
conf
- the configuration to alteruri
- the new default filesystem uripublic void initialize(java.net.URI name, Configuration conf) throws java.io.IOException
name
- a uri whose authority section names the host, port, etc.
for this FileSystemconf
- the configuration
java.io.IOException
public abstract java.net.URI getUri()
public java.lang.String getName()
public static FileSystem getNamed(java.lang.String name, Configuration conf) throws java.io.IOException
java.io.IOException
public static LocalFileSystem getLocal(Configuration conf) throws java.io.IOException
conf
- the configuration to configure the file system with
java.io.IOException
public static FileSystem get(java.net.URI uri, Configuration conf) throws java.io.IOException
java.io.IOException
public static void closeAll() throws java.io.IOException
java.io.IOException
public Path makeQualified(Path path)
public static FSDataOutputStream create(FileSystem fs, Path file, FsPermission permission) throws java.io.IOException
fs
- file system handlefile
- the name of the file to be createdpermission
- the permission of the file
java.io.IOException
public static boolean mkdirs(FileSystem fs, Path dir, FsPermission permission) throws java.io.IOException
fs
- file system handledir
- the name of the directory to be createdpermission
- the permission of the directory
java.io.IOException
create(FileSystem, Path, FsPermission)
protected void checkPath(Path path)
public BlockLocation[] getFileBlockLocations(FileStatus file, long start, long len) throws java.io.IOException
java.io.IOException
public abstract FSDataInputStream open(Path f, int bufferSize) throws java.io.IOException
f
- the file name to openbufferSize
- the size of the buffer to be used.
java.io.IOException
public FSDataInputStream open(Path f) throws java.io.IOException
f
- the file to open
java.io.IOException
public FSDataOutputStream create(Path f) throws java.io.IOException
java.io.IOException
public FSDataOutputStream create(Path f, boolean overwrite) throws java.io.IOException
java.io.IOException
public FSDataOutputStream create(Path f, Progressable progress) throws java.io.IOException
java.io.IOException
public FSDataOutputStream create(Path f, short replication) throws java.io.IOException
java.io.IOException
public FSDataOutputStream create(Path f, short replication, Progressable progress) throws java.io.IOException
java.io.IOException
public FSDataOutputStream create(Path f, boolean overwrite, int bufferSize) throws java.io.IOException
f
- the file name to openoverwrite
- if a file with this name already exists, then if true,
the file will be overwritten, and if false an error will be thrown.bufferSize
- the size of the buffer to be used.
java.io.IOException
public FSDataOutputStream create(Path f, boolean overwrite, int bufferSize, Progressable progress) throws java.io.IOException
f
- the file name to openoverwrite
- if a file with this name already exists, then if true,
the file will be overwritten, and if false an error will be thrown.bufferSize
- the size of the buffer to be used.
java.io.IOException
public FSDataOutputStream create(Path f, boolean overwrite, int bufferSize, short replication, long blockSize) throws java.io.IOException
f
- the file name to openoverwrite
- if a file with this name already exists, then if true,
the file will be overwritten, and if false an error will be thrown.bufferSize
- the size of the buffer to be used.replication
- required block replication for the file.
java.io.IOException
public FSDataOutputStream create(Path f, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) throws java.io.IOException
f
- the file name to openoverwrite
- if a file with this name already exists, then if true,
the file will be overwritten, and if false an error will be thrown.bufferSize
- the size of the buffer to be used.replication
- required block replication for the file.
java.io.IOException
public abstract FSDataOutputStream create(Path f, FsPermission permission, boolean overwrite, int bufferSize, short replication, long blockSize, Progressable progress) throws java.io.IOException
f
- the file name to openpermission
- overwrite
- if a file with this name already exists, then if true,
the file will be overwritten, and if false an error will be thrown.bufferSize
- the size of the buffer to be used.replication
- required block replication for the file.blockSize
- progress
-
java.io.IOException
setPermission(Path, FsPermission)
public boolean createNewFile(Path f) throws java.io.IOException
java.io.IOException
public FSDataOutputStream append(Path f) throws java.io.IOException
f
- the existing file to be appended.
java.io.IOException
public FSDataOutputStream append(Path f, int bufferSize) throws java.io.IOException
f
- the existing file to be appended.bufferSize
- the size of the buffer to be used.
java.io.IOException
public abstract FSDataOutputStream append(Path f, int bufferSize, Progressable progress) throws java.io.IOException
f
- the existing file to be appended.bufferSize
- the size of the buffer to be used.progress
- for reporting progress if it is not null.
java.io.IOException
@Deprecated public short getReplication(Path src) throws java.io.IOException
src
- file name
java.io.IOException
public boolean setReplication(Path src, short replication) throws java.io.IOException
src
- file namereplication
- new replication
java.io.IOException
public abstract boolean rename(Path src, Path dst) throws java.io.IOException
java.io.IOException
@Deprecated public abstract boolean delete(Path f) throws java.io.IOException
java.io.IOException
public abstract boolean delete(Path f, boolean recursive) throws java.io.IOException
f
- the path to delete.recursive
- if path is a directory and set to
true, the directory is deleted else throws an exception. In
case of a file the recursive can be set to either true or false.
java.io.IOException
public boolean deleteOnExit(Path f) throws java.io.IOException
f
- the path to delete.
java.io.IOException
protected void processDeleteOnExit()
public boolean exists(Path f) throws java.io.IOException
f
- source file
java.io.IOException
@Deprecated public boolean isDirectory(Path f) throws java.io.IOException
java.io.IOException
public boolean isFile(Path f) throws java.io.IOException
java.io.IOException
@Deprecated public long getLength(Path f) throws java.io.IOException
java.io.IOException
public ContentSummary getContentSummary(Path f) throws java.io.IOException
ContentSummary
of a given Path
.
java.io.IOException
public abstract FileStatus[] listStatus(Path f) throws java.io.IOException
f
- given path
java.io.IOException
public FileStatus[] listStatus(Path f, PathFilter filter) throws java.io.IOException
f
- a path namefilter
- the user-supplied path filter
java.io.IOException
- if encounter any problem while fetching the statuspublic FileStatus[] listStatus(Path[] files) throws java.io.IOException
files
- a list of paths
java.io.IOException
public FileStatus[] listStatus(Path[] files, PathFilter filter) throws java.io.IOException
files
- a list of pathsfilter
- the user-supplied path filter
java.io.IOException
public FileStatus[] globStatus(Path pathPattern) throws java.io.IOException
Return all the files that match filePattern and are not checksum files. Results are sorted by their names.
A filename pattern is composed of regular characters and special pattern matching characters, which are:
pathPattern
- a regular expression specifying a pth pattern
java.io.IOException
public FileStatus[] globStatus(Path pathPattern, PathFilter filter) throws java.io.IOException
pathPattern
- a regular expression specifying the path patternfilter
- a user-supplied path filter
java.io.IOException
- if any I/O error occurs when fetching file statuspublic Path getHomeDirectory()
public abstract void setWorkingDirectory(Path new_dir)
new_dir
- public abstract Path getWorkingDirectory()
public boolean mkdirs(Path f) throws java.io.IOException
mkdirs(Path, FsPermission)
with default permission.
java.io.IOException
public abstract boolean mkdirs(Path f, FsPermission permission) throws java.io.IOException
java.io.IOException
public void copyFromLocalFile(Path src, Path dst) throws java.io.IOException
java.io.IOException
public void moveFromLocalFile(Path[] srcs, Path dst) throws java.io.IOException
java.io.IOException
public void moveFromLocalFile(Path src, Path dst) throws java.io.IOException
java.io.IOException
public void copyFromLocalFile(boolean delSrc, Path src, Path dst) throws java.io.IOException
java.io.IOException
public void copyFromLocalFile(boolean delSrc, boolean overwrite, Path[] srcs, Path dst) throws java.io.IOException
java.io.IOException
public void copyFromLocalFile(boolean delSrc, boolean overwrite, Path src, Path dst) throws java.io.IOException
java.io.IOException
public void copyToLocalFile(Path src, Path dst) throws java.io.IOException
java.io.IOException
public void moveToLocalFile(Path src, Path dst) throws java.io.IOException
java.io.IOException
public void copyToLocalFile(boolean delSrc, Path src, Path dst) throws java.io.IOException
java.io.IOException
public Path startLocalOutput(Path fsOutputFile, Path tmpLocalFile) throws java.io.IOException
java.io.IOException
public void completeLocalOutput(Path fsOutputFile, Path tmpLocalFile) throws java.io.IOException
java.io.IOException
public void close() throws java.io.IOException
close
in interface java.io.Closeable
java.io.IOException
public long getUsed() throws java.io.IOException
java.io.IOException
@Deprecated public long getBlockSize(Path f) throws java.io.IOException
java.io.IOException
public long getDefaultBlockSize()
public short getDefaultReplication()
public abstract FileStatus getFileStatus(Path f) throws java.io.IOException
f
- The path we want information from
java.io.FileNotFoundException
- when the path does not exist;
IOException see specific implementation
java.io.IOException
public FileChecksum getFileChecksum(Path f) throws java.io.IOException
f
- The file path
java.io.IOException
public void setVerifyChecksum(boolean verifyChecksum)
verifyChecksum
- public void setPermission(Path p, FsPermission permission) throws java.io.IOException
p
- permission
-
java.io.IOException
public void setOwner(Path p, java.lang.String username, java.lang.String groupname) throws java.io.IOException
p
- The pathusername
- If it is null, the original username remains unchanged.groupname
- If it is null, the original groupname remains unchanged.
java.io.IOException
public void setTimes(Path p, long mtime, long atime) throws java.io.IOException
p
- The pathmtime
- Set the modification time of this file.
The number of milliseconds since Jan 1, 1970.
A value of -1 means that this call should not set modification time.atime
- Set the access time of this file.
The number of milliseconds since Jan 1, 1970.
A value of -1 means that this call should not set access time.
java.io.IOException
public static java.util.Map<java.lang.String,FileSystem.Statistics> getStatistics()
getAllStatistics()
instead
public static java.util.List<FileSystem.Statistics> getAllStatistics()
public static FileSystem.Statistics getStatistics(java.lang.String scheme, java.lang.Class<? extends FileSystem> cls)
cls
- the class to lookup
public static void clearStatistics()
public static void printStatistics() throws java.io.IOException
java.io.IOException
|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |