4. Installing the DIRAC File Catalog
4.1. Pre-requisite
You should:
have a machine setup as described in Basic Tutorial setup
be able to install dirac components
have installed a DIRAC SE using the tutorial (Install a DIRAC Storage Element).
4.2. Tutorial goal
The aim of the tutorial is to install the DIRAC FileCatalog (DFC) By the end of the tutorial, you will be able to do all sort of simple Data Management operations.
4.3. More links
More information can be found at the following places:
Introduction to DataManagement: Data Management System
Catalog resource definition Catalogs
How-to datamanagement for user DataManagement
4.4. Installing the DFC
This section is to be executed as diracuser
with a proxy with dirac_admin
group.
The DFC is no different than any other DIRAC service with a database. The installation step are thus very simple:
[diracuser@dirac-tuto ~]$ dirac-admin-sysadmin-cli --host dirac-tuto
Pinging dirac-tuto...
[dirac-tuto]$ install db FileCatalogDB
Adding to CS DataManagement/FileCatalogDB
Database FileCatalogDB from DIRAC/DataManagementSystem installed successfully
[dirac-tuto]$ install service DataManagement FileCatalog
Loading configuration template /home/diracuser/DIRAC/DIRAC/DataManagementSystem/ConfigTemplate.cfg
Adding to CS service DataManagement/FileCatalog
service DataManagement_FileCatalog is installed, runit status: Run
4.5. Adding the FileCatalog resource
In order to be used as a FileCatalog by clients, the DFC needs to be declared. This happens in two places:
/Resources/FileCatalogs/
: in this section, you define how to access the catalog/Operations/Defaults/Services/Catalogs/
: in this section, you define how to use the catalog (for example read/write)
Since we have only one catalog, we will use it as Read-Write
and as Master
.
Using the WebApp (group dirac_admin
), add the following in /Resources/FileCatalogs/
(all options to defaults):
FileCatalog
{
}
Using the WebApp, add the following in /Operations/Defaults/Services/Catalogs
:
FileCatalog
{
AccessType = Read-Write
Status = Active
Master = True
}
From this moment onward, the catalog is totally usable.
4.6. Test the catalog
Since we have a StorageElement at our disposal, we can use the standard dirac-dms-*
script.
First, let us create a file and then “put it on the grid”:
[diracuser@dirac-tuto ~]$ echo "Hello" > /tmp/world.txt
[diracuser@dirac-tuto ~]$ dirac-dms-add-file /tutoVO/user/c/ciuser/world.txt /tmp/world.txt StorageElementOne
Uploading /tutoVO/user/c/ciuser/world.txt
Successfully uploaded file to StorageElementOne
Now, let’s check its replicas and metadata:
[diracuser@dirac-tuto ~]$ dirac-dms-lfn-replicas /tutoVO/user/c/ciuser/world.txt
LFN StorageElement URL
=====================================================
/tutoVO/user/c/ciuser/world.txt StorageElementOne dips://dirac-tuto:9148/DataManagement/StorageElement/tutoVO/user/c/ciuser/world.txt
[diracuser@dirac-tuto ~]$ dirac-dms-lfn-metadata /tutoVO/user/c/ciuser/world.txt
{'Failed': {},
'Successful': {'/tutoVO/user/c/ciuser/world.txt': {'Checksum': '078b01ff',
'ChecksumType': 'Adler32',
'CreationDate': datetime.datetime(2019, 4, 16, 9, 5, 58),
'FileID': 1L,
'GID': 1,
'GUID': '09F7E02F-1290-BE21-1DA7-07A266F153B3',
'Mode': 509,
'ModificationDate': datetime.datetime(2019, 4, 16, 9, 5, 58),
'Owner': 'ciuser',
'OwnerGroup': 'dirac_admin',
'Size': 6L,
'Status': 'AprioriGood',
'UID': 1}}}
Note that these metadata are those registered in the catalog (which hopefully should match the physical one !)
We can also check all the user files that belong to us on the grid:
[diracuser@dirac-tuto ~]$ dirac-dms-user-lfns
Will search for files in /tutoVO/user/c/ciuser
/tutoVO/user/c/ciuser: 1 files, 0 sub-directories
1 matched files have been put in tutoVO-user-c-ciuser.lfns
[diracuser@dirac-tuto ~]$ cat tutoVO-user-c-ciuser.lfns
/tutoVO/user/c/ciuser/world.txt
Finally, let’s remove the file:
[diracuser@dirac-tuto ~]$ dirac-dms-remove-files /tutoVO/user/c/ciuser/world.txt
Successfully removed 1 files