.. _storage_facilities: Data storage ============ The types of storage available to INSIGNEO members are enumerated below. .. todo:: See LINK for a guide to what storage to use for different purposes. .. toctree:: IT Services-managed Research Storage ------------------------------------ Valuable research data for all active INSIGNEO-related projects should be stored on IT Services-managed Research Storage unless that is appropriate. See `IT Services's guide `__ to: * Requesting new storage areas; * Accessing existing storage areas from different operating systems / machines; * Managing permissions. Relevant storage areas ^^^^^^^^^^^^^^^^^^^^^^ The most significant storage volumes that IT Services manage for INSIGNEO are: * ``bone_biomechanics1`` (10TB) - for collaborators of Enrico Dall'Ara. * ``CCE_group1`` (5TB) - for collaborators of Richard Clayton. * ``multisim2`` (20TB) - for members of the IMSB INSIGNEO sub-group including those working on the MultiSim project. * ``polaris2`` (5TB) - for members of the Polaris INSIGNEO sub-group. * ``viceconti_group1`` (20TB) - for collaborators of Marco Viceconti. How do I request access to these? ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ A relevant PI needs to contact the :ref:`INSIGNEO tech team `. Access can be read-only or read plus write and can be granted for an entire volume or just a subfolder. How do I access the data from some machine? ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ See `IT Services's guide `__ for instructions on accessing their Research Storage from :ref:`local HPC (ShARC/Iceberg), shared workstations ` and your own machines. High-performance computing storage ---------------------------------- Various filestores are available on the :ref:`University's HPC clusters `, including *certain* (but not all) `IT Services-Managed Research Storage` areas. These differ in terms of: * Speed (which may differ for small and large files); * Where they can be accessed from (from a single node; from mutiple nodes; from off-HPC); * Whether they are backed up; * Permissions management; * Whether files are deleted after a job or after a period of time. See the main University HPC documentation for a :ref:`guide to HPC storage options `. Google Drive ------------ Advice on effective use of Google Drive will be added in due course. Note that Google Drive, provided by the University's GSuite for Education service, has several limitations you should be aware of with regards to: * The maximum individual file size that can be uploaded or synced; * The maximum volume of data that can be uploaded per day. `More information `__. INSIGNEO NAS ------------ INSIGNEO manage a Network-Attached Storage (NAS) server that provides 55TB of usable storage. This machine, *NAS1* a.k.a. ``nas1-insigneo.sheffield.ac.uk``, was historically used to provide a large amount of shared storage for various INSIGNEO members and INSIGNEO-related projects. It is regularly backed up to a second NAS server. .. warning: NAS1 is being deprecated. For several reasons: * NAS1 is far less resilient than IT Services-managed Research Storage. * IT Services-managed Research Storage allows for much easier recovery of previous versions of files. * NAS1 has a separate set of accounts to IT Services-managed accounts. * NAS1's storage is not accessible from from :ref:`local HPC `. Valuable data for new projects should be saved in `IT Services-managed Research Storage`_. .. todo:: Relevant volumes The most significant storage volumes on NAS1 are: How do I gain access to NAS1? ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ A relevant PI needs to contact the :ref:`INSIGNEO tech team `. Access can be read-only or read plus write and can be granted for an entire volume or just a subfolder. How do I access the data from some machine? ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Data can be accessed from shared workstations and your own machines but *not* from :ref:`ShARC/Iceberg`. Web access is via `https://nas1-insigneo.sheffield.ac.uk/ `__. This could be slow for transferring data to/from NAS1. A guide to accessing NAS1 without using the web interface (i.e. via Windows Explorer) will be provided in due course.