Large Files

This section describes how to handle large files when using FCS.

This page discusses:

Requirements

For HTML/JSP-based applications, including the 3DEXPERIENCE platform product suites and custom programs, large file checkins require both of the following:

  • Checkins must be targeted for a FCS-enabled store/location.
  • Checkins must be invoked via the configurable file upload applet (see Installation and Setup - Administrate - 3DEXPERIENCE Platform |-3DSpace Administration - File Collaboration Server - Setting Up the FCS - Migrating FTP Stores to File Stores for setup information.)

We recommend that you enable both FCS and the file upload applet for all implementations, even if you do not foresee working with files larger than 2 Gbs.

For files larger than 2.5 Gb, you should write your own checkout JSPs.

The file transfer mechanism for FCS checkins and checkouts uses Transfer-Encoding=chunked per the HTTP 1.1 specification, which is supported on all 3DSpace supported application servers. However, there are certain third-party vendor components that may not support this protocol correctly. These limitations are described below.

Checkin

If there is a web server located between the client (3DEXPERIENCE platform) and the FCS, some web servers will not pass the file-transfer request through properly.

Apache plugin encountered errors. Typical errors (in the web server's log) may report that chunking is not supported, content-length is not set, or there is a badly formed POST body. These may be issued by the web server itself, or by web server plugins that pass the request on to an Application Server.

STATUS/WORKAROUND: This is not an issue for files < 1.9 gigabytes (2040109465 bytes) which do not use the chunked protocol. If you do not typically handle files larger than this, there is no problem. However, if you frequently handle files larger than this, you must define FCS URLs so they do NOT go through a web server/plugin that does not support this aspect of HTTP 1.1.

There is a delay at the end of a checkin for files < 2 Gb after the server returns a receipt to the client.

Checkin from an FTP store is no longer supported. To use data in FTP stores, you must migrate FTP stores to file stores. For more information, see Collaboration and Approvals Administration Guide : Migrating FTP Stores to File Stores.

Checkout

Downloading a file bigger than 4 Gb should not be handled by Internet Explorer by default.

For files less than 4 gigabytes, no workaround is needed (although the workaround described here is for files over 2 gb). If you need to download larger files, you will need to configure the client-side applet (FileDownloadApplet) to be used instead of emxCommonDocumentCheckout.jsp by setting the following system property to true in emxsystem.properties. The default is false.

emxFramework.UseDownloadApplet = true

When the download applet is enabled, it is used only for downloading files over 2 gb. When it is used, the files are saved to the file system; however, they cannot be opened via the applet without making changes to the registry for the I.E. client. The following two links explain the problem and note some registry settings that can be modified to override this behavior:

http://windowsxp.mvps.org/ie/elevlocalfile.htm

http://www.all-usenet-archive.com/File.asp?service=15823

For Firefox, you can set security.checkloaduri to false (in about:config URL) to allow large files to be opened.

Alternatively, you can open the files outside of the application via the file system.

If you frequently handle files larger than 2 gigabytes, configure your FCS URLs to directly point to the FCS application server, with no intermediate web server. Also, configure both the FileUploadApplet and the FileDownloadApplet to replace the corresponding checkout/checkin JSP pages in the applications.

FcsClientLargeFile Class

The FcsClientLargeFile class can also be used to handle large file checkin and checkout operations. The class contains one checkin method and two checkout methods. These methods improve large file checkin and checkout performance by using hard links to avoid file copy operations and by using an optimized procedure for file copy operations rather than a high-level transport protocol such as HTTP.

Hard links are directory references or pointers to a file on a storage volume. Hard links can only be created on the same file system where the file is stored. The following three directories are used by the checkin and checkout methods. These directories must be on the same file system for hard links to work.

  • Application Working Directory - is used by the FCS client application to read or write files. This directory is managed by the application and can only be accessed by the application. The FCS does not have access to this directory.
  • Stage Directory - is used to exchange files between the Application Working directory and the Store/Location directory. This directory manages file permission changes and can be accessed by both the application and the FCS. This directory is managed by the application.

    Files and hard links are not created in this directory. They are created in the temporary folders created by the command in this directory.

  • Store/Location Directory - is used by the FCS to store files. This directory is managed by the FCS and can only be accessed by the FCS. The application does not have access to this directory.

In UNIX environments, application and FCS users must have access to the ln and chmod commands. In Windows environments, hard links are created through the CreateHardLink executable program.

Note: If you encounter a Java SecurityException when creating hardlinks, you must grant java.nio.file.LinkPermission("hard").

Before you can use the FcsClientLargeFile class, the FCS must have the appropriate access to the file system. File access should be carefully managed for each directory. For information about the FcsClientLargeFile class, see the Studio Customization Toolkit Reference Guide (Java Docs) that are installed with the Studio Customization Toolkit.

Setting Up File Access

The following access should be given to each directory:

On UNIX:

  • Application Working Directory - application user should have full control: drwx------
  • Stage Directory - everyone should have write and execute access: d-wx-wx-wx
  • Store/Location Directory - FCS user should have full control: drwx------

On Windows:

From the Properties window for each directory, access the Security tab and grant the following access:

  • Application Working Directory - application user should have full control
  • Stage Directory - application and FCS users should have full control. A group can be created for application users to make this task easier.
  • Store/Location Directory - FCS user should have full control.

Hard Link for Local Clone

The "hard link for local clone" feature improves the efficiency of the FCS local clone service by creating hard links on copied files. This saves storage space and is particularly important for large file support. MQL commands are provided for evaluating the size of a store or location using 3DEXPERIENCE file metadata to assist System Administrators in planning replication and archiving processes.

The "hard link for local clone" feature improves the performance of FCS clone/copy services in the special case of local clone. "local" means that the source (copied) file and destination (copy) file are managed by the same FCS. The difference between the regular FCS clone/copy and FCS hard link clone/copy is the file-system type of the files created: the former creates files using buffered, bit-wise copy while the latter creates links. Creation of a hard link is possible only when the file that is being copied and its copy belong to the same file system. As the FCS server cannot check for this condition, a regular file copy is performed whenever hard link creation fails.

Hard link creation is not always faster than copying a file, because the performance of hard link creation (from Java code) is both OS and hardware dependent. A hard link threshold property is, therefore, provided to allow setting of the maximum file size for which copying is faster than hard link creation. In other words, copying of a file that is smaller than the threshold value is faster than creating a hard link for that file. You can override the default, OS-dependent threshold values provided in the FCS code, if desired, by setting the ematrix.fcs.hardlink.threshold property in the framework.properties file, which is located in the SERVER_INSTALL\managed\properties\ (Windows) or SERVERHOME/managed/properties/ (UNIX) directory. The value of this property is a file size in bytes: all files smaller than this size are cloned using file copy, files larger than this size are cloned by hard link copy. To disable the hard link for local clone feature altogether, you can set the ematrix.fcs.hardlink.threshold property to -1. This may be useful when using a new prefix for a store, assuming that the new prefix path and the former store path are on different file systems.

Introduction of a hard link in store and location directories creates a gap between the store/location logical size, which is computed from 3DEXPERIENCE metadata, and its file system size. To handle this, MQL commands are provided to retrieve store/location logical size. These commands improve store and location administration tasks, such as store archiving and replication.

In addition, print store/location XXX select size.* commands are provided to compute the total size of the files physically located in a store/location path using 3DEXPERIENCE file metadata. Obsolete files are also included in the computation. For example, if you check a file into store A, the size of the store returned by the print store A select size.byte command is increased by the byte size of this file. If you check a file into location locA and replicate store A, the size of store A does not change but that of locA does.

On UNIX systems, evaluating store XXX size provides the same result as the shell command du -l /store_XXX_path/. The -l option of the du command counts file size as many times as the file is hard linked.

The store/location selectable is size, which is associated with a unit sub-selectable (byte, mb, or gb), which in turn specifies the size unit. For example, the command print store/location XXX select size.byte returns a size in bytes. In like manner, the argument select size.mb returns a size in megabytes, while select size.gb returns a size in gigabytes. The print store/location XXX select size.[byte|mb|gb] command does not create persistent data (hard links).

The files of a hashed store that is managed by FCS and those recorded by the MCS metadata (business object files) have different names. However, there is a one-to-one mapping between metadata files and store files. The life cycle of persistent data (hard links) through FCS operations is as follows:

  • A client application checks out a file that is the result of a hard link local clone/copy. In this case, FCS reads and returns file data as with any other file.
  • A client application updates a file that has been cloned one or more times with "hard link for local clone". In this case, FCS creates a new file in the store folders (with encoded/hashed names) during update checkin, and then deletes the former version of the file. However, as the file is a hard link, it is not removed from the file system. The file system operation performed is, in fact, a (POSIX) unlink operation.
  • A client application updates a file that is the result of a hard link local clone/copy. In this case, FCS creates a new file in the store folders (with encoded/hashed names) during update checkin, and then deletes the former version of the file. However, as the file is a hard link, it is not removed from the system. The delete operation performed is, in fact, a (POSIX) unlink operation.
  • A client application deletes a file that has been cloned one or more times with "hard link for local clone". In this case, FCS deletes the corresponding encoded/hashed file in the store folder. As this file is the target of one or more hard links, this delete operation is, in fact, a (POSIX) unlink operation. So again, the file is not removed from the file system.
  • A client application deletes a file that is the result of a hard link local clone/copy. In this case, FCS deletes the corresponding encoded/hashed file in the store folder. As this file is a hard link, this delete is, in fact, a (POSIX) unlink operation. So once again, the file is not removed from the file system unless it is the last file link.

The following are some limitations of the "hard link for local clone" feature:

  • The "hard link for local clone" feature works only for captured, hashed stores.
  • The "hard link for local clone" feature works only for stores that have a non-void FCS URL.
  • If you create a store prefix that is the mount point of another file system or a junction point of another volume (NTFS), then the "hard link for local clone" feature should be disabled for the FCS managing this store (by setting ematrix.fcs.hardlink.threshold=-1) in the framework.properties file.
  • When performing store or location synchronization, the hard links created in the store or location path are not reported in the sync destination path. Therefore, the volume of this synchronization operation may exceed the file system size of the source store or location. To correctly plan your synchronization operation, you must issue a sync store command together with the validate size option.