Storage of Data
Table 1: Recommendations based on scale. It is useful for labs to consider that there is more than one option to store data, and the strategy should be chosen based on needs. All different options can be used where appropriate. Here we provide a guide to selecting storage and backup strategy.
Size of dataset
(10 biological samples) <100 GB >100 GB >1 TB
>10 TB
storage decreased the speed of an upload by a factor of 10 aſter the first 100 GB of data were uploaded. Reduced transfer rates may completely halt the use of cloud storage if not planned in advance or when working with large datasets (>100 GBs). Some providers may also offer free institutional access but limit data storage for 24 hrs. Even with unlimited cloud bandwidth, the speed of data transfer can still be limited by fixed institutional internet speed, which is oſten below 1 Gbps. In this situation, just transfer of a single 10 Gb file may take 30 minutes. In summary, cloud storage may be appropriate when data
size for a project is limited to a few GBs and the total amount of data per project is below 1 TB, data acquisition computers are attached to the Internet, and only a few people work with the datasets. Dedicated servers for network-attached storage. Net-
work-attached storage (NAS) is a dedicated data storage com- puter, or bank of computers, located on the same network as the acquisition microscope and image processing workstation. To the user, the storage is presented as a network drive allowing simple “drag-and-drop” workflow as a local drive or USB hard drive functions. A major benefit is that NAS physical storage is independent of data acquisition and image processing systems. Having a dedicated storage server allows gradual storage expan- sion as required without rebuilding the system from an initial ∼100 TB up to and above 500 TB. When the NAS, acquisition microscope, and image processing workstation are physically close to each other, affordable fast networking with speeds of 10 Gbps to 100 Gbps can be installed minimizing delays in data transfer. One approach is to place a data processing server in the
same computer rack with the storage server connected by a 10 Gbps network, which can be accomplished for less than $1,000 (Table 1). Tis allows the end user to access the data processing machine remotely. Backup can be implemented using a second server or via the internet to cloud storage as described above. Te process of data transfer to the cloud can be automated and hidden from the end user. Te administrative interfaces available for such systems make it a relatively simple task for a non-professional to manage data storage and transfer, and the stability of the system ensures that little service is needed aſter initial installation. Te NAS solution can be securely shared between several groups, decreasing costs for the individual research laboratories as described below. It is possible for individual laboratories or small groups
of investigators to develop and share NAS capabilities with a relatively low investment of time and money (Table 2). Man- agement is possible by the laboratory members, but in a better scenario, laboratories should purchase a dedicated server and
44 Mode of transfer
USB drives Network
Optical fiber network (>10 Gbps) Primary storage Workstation
Cloud (Dropbox, etc.) Local server
Centralized server
Cloud Cloud
Cloud provider / second server Cloud
manage the system with assistance from the IT department, since fast networking between floors or buildings will most likely require institutional support. Access from outside the institution can be restricted by the IT department firewall in order to limit vulnerability. In summary, the original price of a NAS system ($20,000
for 250 TB, Table 2) can be intimidating, and institutional IT assistance for setup may be required. Given these limitations, acquiring a personal laboratory NAS is advised only when indi- vidual datasets are larger than tens of GBs, the total amount of data storage is larger than 10 TB, or when storage needs to be secure, fast, and expandable. Details of one implementation of a NAS system is provided in the next section.
Implementation of a Resilient 250-Terabyte Storage NAS To provide storage of large volumes of imaging data, we
have built a centralized storage server using commercially available hardware and free open-source soſtware. Te original goal was to provide storage for fluorescence light-sheet micros- copy data and specifically whole-brain imaging of zebrafish neural activity. Tese experiments routinely generate datasets of approximately 500 GBs. Since implementation three years ago, the server has turned into a shared resource and now serves more than a dozen projects and researchers without data loss or service interruption. To provide this resiliency, we had to account for potential hardware failures, power loss events, and random errors during data reading and writing. Protection against single-drive failure. To create large
centralized storage, we pooled together up to 70 bare spin- ning-disk hard drives (internal SATA drives) for a total storage
Table 2: Estimated cost of server for storage of 250 TB. Building dedicated storage can be expensive, and we provide an example of budget using commodity parts. Prices are in USD.
Item
Main storage server, 24 bays JBOD, 45 bays 10 GbE switch 10 GbE cards Server rack
UPS, 3000 VA
Data hard drive, 4 TB Accessories Total
Count Est. price Total cost 1 $5,000 $5,000 1 $1,700 $1,700 1 2 1
$600 $200 $450
69
$600 $400 $450
2 $1,600 $3,200 $120 $8,280 $1,000 $1,000 $20,630
www.microscopy-today.com • 2020 July Backup
Page 1 |
Page 2 |
Page 3 |
Page 4 |
Page 5 |
Page 6 |
Page 7 |
Page 8 |
Page 9 |
Page 10 |
Page 11 |
Page 12 |
Page 13 |
Page 14 |
Page 15 |
Page 16 |
Page 17 |
Page 18 |
Page 19 |
Page 20 |
Page 21 |
Page 22 |
Page 23 |
Page 24 |
Page 25 |
Page 26 |
Page 27 |
Page 28 |
Page 29 |
Page 30 |
Page 31 |
Page 32 |
Page 33 |
Page 34 |
Page 35 |
Page 36 |
Page 37 |
Page 38 |
Page 39 |
Page 40 |
Page 41 |
Page 42 |
Page 43 |
Page 44 |
Page 45 |
Page 46 |
Page 47 |
Page 48 |
Page 49 |
Page 50 |
Page 51 |
Page 52 |
Page 53 |
Page 54 |
Page 55 |
Page 56 |
Page 57 |
Page 58 |
Page 59 |
Page 60 |
Page 61 |
Page 62 |
Page 63 |
Page 64 |
Page 65 |
Page 66 |
Page 67 |
Page 68 |
Page 69 |
Page 70 |
Page 71 |
Page 72 |
Page 73 |
Page 74 |
Page 75 |
Page 76 |
Page 77 |
Page 78 |
Page 79 |
Page 80 |
Page 81 |
Page 82 |
Page 83 |
Page 84
