You can use CSV data, index files, as well as TIFF and GeoTIFF raster data stored in HDFS for distributed analysis. Among them, after registering the raster data (such as TIFF, GeoTIFF) stored in the HDFS directory to iServer, it can be distributed and stored in the HDFS distributed repository and local file system storage.
Register the HDFS directory
Sign in In the iServer service management server, click "Data"-"Data Registration", enter the register datastore page (http://{ip}:{port}/iserver/admin-ui/data/dataRegistration), click the "Register Datastore" button, and configure the following parameters:
- Storage ID: User-defined storage ID name, a unique identifier created for the database
- Datastore type: Select "Big Data File Share"
- Shared data type: Select "HDFS Directory".
- Share directory: Enter the shared directory address
- If you need to register a separate CSV file stored on HDFS, you can fill it out directly HDFS path + CSV name, for example: hdfs://{ip}:9000/data/a.csv.
- If you register a file directory with CSV files, the directory contains multiple CSV file (only supports opening HDFS directories as read-only):
-
- If the fields, attributes, and other formats of the CSV file are the same, when registering, it is necessary to fill in the upper level directory of the directory where the CSV file is located, such as HDFS:// {ip}:9000/data， Among them, data is the upper level directory of the CSV file.
- When the CSV files are different, you need to fill in the directory where the CSV files are located, such as hdfs://{ip}:9000/data/csvfolder.
- If you are registering a file directory with an index, you need to fill in the directory where the index file is located, such as hdfs:// {ip}:9000/data/indexfolder， Among them, the index folder contains the index file.
- If you are registering TIFF or GeoTIFF data, you need to fill in the directory where the TIFF or GeoTIFF data is located, such as HDFS:// {ip}:9000/data/TIFF， among them, TIFF contains TIFF and GeoTIFF files.
- If the registered HDFS cluster is in high availability mode, simply fill in the high availability HDFS address, such as HDFS://{high availability address}/data.
- Set whether "Kerberos authentication is enabled". If it is not enabled, there is no need to check it; If you need to register an HDFS directory that has enabled Kerberos authentication, you need to check the box for "Enable Kerberos authentication" and configure it as follows:
- Subject name: username @ domain. For example: iServer@SUPERMAP1.COM
- User name: Must be a system user of the HDFS cluster Mater node.
- Domain: The domain configured for the Kerberos service. (Must be in the same domain as the HDFS cluster)
- Key file path: Copy the key file generated during the cluster building process to the file location on the machine where the iServer service is located.
- Kerberos client configuration file path: The file location of the Kerberos client configuration file on the machine where the iServer service is located.
- Set whether the selected HDFS cluster is in 'high availability mode'. If it is not in high availability mode, there is no need to check it; If it is in high availability mode, it needs to be checked and the core-site.xml and hdfs-site.xml in the HDFS cluster configuration file need to be copied to any specified location on the machine where the iServer is located. The following parameters should also be filled in:
- Hadoop cluster core-site.xml file path: core-site.xml file path in the machine where iServer is located
- HDFS cluster HDFS-site.xml file path: HDFS-site.xml file path in the machine where iServer is located
Click the "Register Datastore" button to complete the registration.
Note: 1. If you have registered a CSV data file, it needs to be validated before it can be used for distributed analysis services. See details: CSV data file verification.
2. If the registered HDFS directory has enabled Kerberos authentication and the distributed analysis service uses a Hadoop Yarn cluster with enabled Kerberos authentication, it can be used for distributed analysis.