Image collection supports the addition of image files in a variety of storage media and supports a variety of image file formats, including.tif,.img,.ECW, and.pix. The following describes in detail which "Add Type" is suitable for images of different storage media and how to configure it.
Image files can be sourced from the following storage media:
Locally stored images can be added to the collection in any of the following ways:
Images from a remote shared directory can be added to the collection in any of the following ways:
Remote shared directory path: Windows system: \\172.16.120.34\imageChina.
For Linux system, you need to mount the shared directory, and use the mounted directory path (mount the Windows shared directory, the command behavior is: mount -t cifs //{ip}/share /mnt/share, Where, IP is the IP address of the shared directory, and /mnt/share is the mounted directory, for example, mount -t cifs //172.16.120.34/imageChina /mnt/share. To mount a shared directory on a Unix or Linux system, the command line is: mount -t nfs -o rw {ip}:/share/ /mnt/share, where: IP is the IP address of the shared directory, and /mnt/share is the mounted directory.)
Images on the FTP server can be added to the collection only through the "ListFile" mode. One line in the manifest file (.txt file) is recorded as the FTP full path of an Image Files, and the path is written as ftp://172.16.120.34/imageChina/H48C002002.tif.
Note:
The FTP path contains non-English characters, such as ftp://127.0.0.1/影像数据/dem30米分辨率.tif. Add image may fail, because the encoding of the FTP client is inconsistent with the encoding used by the server, and the path of the FTP server is transcoded, resulting in a path error. Image files cannot be accessed.
Solution: Obtain the FTP server image path after transcoding. For example, the FTP server path is: ftp://127.0.0.1/影像数据/dem30米分辨率.tif, Aft transcoding, that load path is a ftp://127.0.0.1/%E5%BD%B1%E5%83%8F%E6%95%B0%E6%8D%AE/dem30%E7%B1%B3%E5%88%86%E8%BE%A8%E7%8E%87.tif.
The transcoded path can be obtained by entering the FTP server path of image files into the address bar of the browser and pressing Enter. In addition, it can also be transcoded.
Mount command behavior: mount -t cifs //{ip}/share /mnt/share, where IP is the IP Address of the shared directory, and /mnt/share is the mounted directory. For example: mount -t cifs //172.16.120.34/imageChina /mnt/share）
The mounted directory is accessed in the same way as the local storage of the Linux system. Add the image to the collection. See: Local Storage
.The mounted directory is accessed in the same way as the local storage of the Windows system, and the images in it are added to the collection. See: Local Storage
.Images from the object store can be added to the collection in any of the following ways:
The collection supports the addition of images from Huawei OBS object storage service, Alibaba OSS object storage service, and S3 simple storage service.
About the full path writing of image files in the object storage service:
The imaging service supports access to image files in the object store and writing of silent slice results to the object store through the /vsis3/ file system handler of GDAL. The /vsis3/ file system handler supports object storage following the standard S3 (Amazon Simple Storage Service) protocol. Including: Huawei OBS Object Storage Service, Alibaba OSS Object Storage Service, Amazon S3 Simple Storage Service, SeaweedFS Seaweed Object Storage, etc.
The image service can successfully access the images in the object store or write the silent slice results to the object store, which requires the server to complete the configuration of the relevant system environment Variables. The following configuration options apply to the /vsis3/ handler, depending on the specific situation of the object store you use. Select the required configuration items and add them to the system environment variables of the server. The system environment variables will not take effect until the iServer server is restarted after configuration..
For The following variables to be added to the system environment variables, it is necessary to fully understand the access protocol supported by the used object storage service, the URI of the object storage, the authentication method, the access method of the bucket, and other related configurations, in combination with the description of the following variable contents. To properly configure the system environment variables.
"The value of the "AWS_HTTPS" variable is NO and the resource in S3 will be accessed using the HTTP protocol; The value of the "AWS_HTTPS" variable is YES and the resource in S3 will be accessed using the HTTPS protocol.
If the "AWS_HTTPS" variable is not configured, the HTTPS protocol is used by default to access resources in S3.
How to set the value of the "AWS_HTTPS" variable? You need to confirm the request protocol supported by the configuration of the S3 object storage server.
Set the access address of S3 API access object storage, and support two address forms of domain name and "IP+port". The default is s3.amazonaws.com. Example of variable value input:
AWS_S3_ENDPOINT=obs.cn-north-4.myhuaweicloud.com
AWS_S3_ENDPOINT=1.1.1.1:80(accessed using HTTP protocol)
AWS_S3_ENDPOINT=1.1.1.1:443(HTTPS access)
Note: As for the port, the S3 object storage server may provide two access protocols, HTTP and HTTPS, which need to be specified according to the current system environment variables "AWS_HTTPS". Specify the correct port number for the "AWS_S3_ENDPOINT” variable value.
| Give an example |
|
How to determine the access address of object storage? Take MINIO object storage as an example. The open port of object storage service for S3 API access to object storage is 9110, and the IP of object storage server is 172.16.121.40. The object storage server console access address is 172.16.121.40:8110. Enter the MINIO Object Storage Console (CONSOLE) as an administrator. As shown in the figure below, you can also see the address of the S3 API access object storage server it listens to. 0.0.0.0 represents the 9110 port on all networks. The AWS_S3_ENDPOINT variable value in this example should be specified as: 172.16.121.40:9110
|
The object store typically supports multiple authentications by selecting one authentication method and specifying the corresponding credentials through the system environment variables.
If the image service is used to manage the image files in the object store, the iServer server is required to have the permission to read the resources in the object store; If the image service needs to tile the result of the silent slice of image collection in the object storage, and the front end can retrieve the pre-slice result, the iServer server needs to have the permission to read and write the resources in the object storage at the same time.
Determines whether request signing is disabled. This option is available for buckets with public access permissions. If YES, connect anonymously to buckets that do not require authenticated access permissions.
If authentication (AWS_ACCESS_KEY_ID和AWS_SECRET_ACCESS_KEY) is required to access the object bucket, the “AWS_NO_SIGN_REQUEST” variable value is set to NO.
| Give an example |
|
Taking MINIO object storage as an example, as shown in the following figure, the access policy of the bucket is Public, which has both read and write permissions by default. Therefore, no authentication is required to access the resources of the bucket. A system environment variables of AWS_NO_SIGN_REQUEST=YES needs to be added, And doe not need to add AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variable.
|
Access key ID used for authentication. If you are using temporary credentials, you must set a AWS_SESSION_TOKEN.
Access key associated with the AWS_ACCESS_KEY_ID. If you are using temporary credentials, you must set a AWS_SESSION_TOKEN.
| Give an example |
|
Taking MINIO object storage as an example, as shown in the figure below, the access policy of the bucket is Private, which means that no operation can be performed without authorization. Authorization is required to access the resources of this bucket.
The MINIO object store uses IAM (AWS Identity and Access Management) to configure permissions, which are authorized by controlling the login user and the permissions of the user. In the MINIO console, the created User verifies the user's identity through Access Key and Secret Key, and controls the user's permission to operate resources through Policies. As shown in the figure below, in the MINIO console, the IAM Policies page organizes and manages users according to access policies. If you need read and write permissions, you can click the "readwrite" item to view the users with such permissions.
When accessing the resources of this kind of object storage bucket, it is necessary to add the System Environment Variables of AWS_NO_SIGN_REQUEST=NO. According to the object storage resource access permissions required by the image service, select the appropriate login user and add the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variable to specify the user authentication credentials. For example, to configure the authentication credentials of a user with read and write permissions (the ugo user in the figure above), you need to set the following three System Environment Variables: AWS_NO_SIGN_REQUEST=NO AWS_ACCESS_KEY_ID=ugo AWS_SECRET_ACCESS_KEY=ugopassword |
Session token used to validate temporary credentials (AWS_ACCESS_KEY_ID和AWS_SECRET_ACCESS_KEY).
Configure two ways to access an object Bucket. The default is TRUE.
S3 supports Virtual Hosted and Path URL access buckets. If the AWS_VIRTUAL_HOSTING variable is TRUE, it is a virtual hosting type, which means that the Bucket is placed in the access mode of the Host Header, and the Bucket is identified by the host name of the virtual Bucket, for example, mybucket.cname.domain.com, namely "Bucket Name + HostName". If the AWS_VIRTUAL_HOSTING variable is FALSE, it is Path, the Bucket is identified as the top-level directory in the URI, for example, cname.domain.com/mybucket.
| Give an example |
|
Taking the MINIO object storage as an example, the default support format is the path request of the http://example.com/bucket/object. The MINIO_DOMAIN environment variable is used to enable virtual host-style requests. If you configure MINIO_DOMAIN, MinIO supports both path-wise and virtual-host-wise requests from clients. Therefore, you need to add AWS_VIRTUAL_HOSTING variables and specify the correct variable values based on the request style supported by the object store configuration. For example, if the MINIO object store uses the path request style to access buckets by default, you need to add AWS_VIRTUAL_HOSTING=FALSE System environment variables. For example, for the Alibaba OSS object storage service, only the virtual hosting access method is supported. You can select to add AWS_VIRTUAL_HOSTING=TRUE system environment variables, or not to add. By default, the virtual hosting style is used to access the bucket. |
Sets the chunk size for segmented uploads. The default value is 50 MB.
This variable needs to be configured for the image service silent slice write object storage. The recommended value of the variable is 1000 MB.
Determines whether VSIMkdir() is allowed to create an empty object to model an empty directory. If NO, it prevents the VSIMkdir() operation from creating an empty object with a directory name that ends in a slash; by default, YES, GDAL creates such an object, which makes it possible to model an empty directory, but this can cause compatibility issues when application does not need such an empty object.
After testing, when the silent slice result of the image service is written into the SeaweedFS seaweed object storage, this variable needs to be configured, and the value of the variable is NO.
When accessing the data in the object store fails, the Errors are as follows: (CURL error: SSL certificate problem:). You need to download a CURL SSL certificate and complete the configuration.
