When one or more nodes are running on low disk space or have a failed disk, then beyond a certain threshold, Vertica starts rejecting transactions that update the catalog or data files. In Vertica, catalog is a set of files that contains information (metadata) about the objects in a database, such as the nodes, tables, constraints, and projections. The catalog files are replicated on all nodes in a cluster. To avoid low disk space or failed disk related issues, you can mount a new hard disk and store the catalog files on it.
Refer to the following steps to mount a new hard disk. You can ignore these steps if you have already mounted a disk.
Obtain the name of the new hard disk. You can use the command fdisk -l.
Format the new hard disk. For example, if the name of the new hard disk is sdc, then run the command mkfs.ext4 /dev/sdc.
Create a new directory and mount the hard disk to it. For example, if the new directory is /vastorage/home/dbadmin/zenworks/NewDisk then use the commands mkdir -p /vastorage/home/dbadmin/zenworks/NewDisk and mount /dev/sdc /vastorage/home/dbadmin/zenworks/NewDisk.
Refer to the following steps to move the existing data to the new directory location and make it the primary storage location for Vertica. The location /vastorage/home/dbadmin/zenworks/NewDisk has been used as an example in the following steps. Replace this location with the location where the new hard disk has been mounted:
Create a new directory within the mounted location and give permissions to it:
mkdir -p /vastorage/home/dbadmin/zenworks/NewDisk/v_zenworks_node0004_data/ && chown -R dbadmin:verticadba /vastorage/home/dbadmin/zenworks/NewDisk/v_zenworks_node0004_data/
NOTE:In the above command, /vastorage/home/dbadmin/zenworks/NewDisk/ is the directory location where the hard disk is mounted and v_zenworks_node0004_data is the new directory.
Create a new db location by running the following SQL query in Vertica using any connection tool and specify the node to which the data should be moved.
CREATE LOCATION '/vastorage/home/dbadmin/zenworks/NewDisk/v_zenworks_node0004_data/' NODE 'v_zenworks_node0001' USAGE 'DATA,TEMP' LABEL 'T0_DATA_TEMP';
NOTE:If you have more than one node, run this query for all the nodes. For example: CREATE LOCATION '/vastorage/home/dbadmin/zenworks/NewDisk/v_zenworks_node0005_data/' NODE 'v_zenworks_node0002 USAGE 'DATA,TEMP' LABEL 'T0_DATA_TEMP';, indicates that directory /vastorage/home/dbadmin/zenworks/NewDisk/v_zenworks_node0005_data/' is mounted in node 2.
Set object storage policy at db level by running the following query:
select set_object_storage_policy('zenworks','T0_DATA_TEMP', true );
Apply the storage policy by running the following query:
select enforce_object_storage_policy('zenworks');
Ensure that the new storage location is created successfully, by running the following queries:
To check the storage location: select * from storage_locations
To check the storage label: select node_name,location_label from storage_containers where projection_id in (select projection_id from projections);
To check whether the new storage location is created or not (the older location will also be displayed till it is retired): select is_retired,location_path from storage_locations;
Retire the older location by running the following query:
SELECT RETIRE_LOCATION('/vastorage/home/dbadmin/zenworks/v_zenworks_node0001_data/' , 'v_zenworks_node0001', true); If you have more than one node, then repeat this for all the older locations of the nodes.
To verify whether the location is retired, check if retire is marked True by running the following query:
select is_retired,location_path from storage_locations;
The retired location will be displayed as True till the location is deleted.
Delete the retired storage location by running the following query:
SELECT DROP_LOCATION('/vastorage/home/dbadmin/zenworks/v_zenworks_node0001_data/' , 'v_zenworks_node0001');.
To check whether the location is deleted, run the following query:
select is_retired,location_path from storage_locations;
After completing this procedure, it is recommended that you restart the database.