What is data warehousing?

As technology advances, more data is being generated, so it is necessary to have efficient storage to be able to save and access all that information.

Data storage has changed a lot, from disk storage systems, which will most likely continue to be used but in a more digital way: today they are connected to a network and are defined by software.

The  storage of data  has a process based on the use of technology. Technology is used to organize, distribute and archive information with the bytes and bits that are part of the systems that people depend on day to day; it becomes so important in all services: from simple applications, multimedia content, addresses, and contacts, to network protocols and everything that has to do with the digital world.

Computers work like a brain, with two types of memory: long term and short term.

The short-term memory is controlled by a random-access memory, better known as RAM. The main function of RAM is to process and remember the actions of a computer while it is active.

A computer distributes data to different volumes of storage depending on the type of data, in the same way that our brain shares long-term memories depending on the type of memory.

Below we present some relevant aspects for optimal data storage, such as architecture, devices, mass storage, among others.

 

Storage Architecture

Storage architecture is a fundamental element, its efficiency and quality directly impacting on the capacity of users to access applications more quickly in order to satisfy their information services needs. Storage systems help any company start with strategies and provide continuity for business processes.

 

Mass Storage

Mass storage of data is a product developed with different types of technology, whether it be mechanical, electronic, magnetic or a more modern version such as hybrid technology, which has capacity for storage, writing and rewriting of bits, in order to obtain and store the information of a user, an organization or an institution.

 

Storage Devices

There are different types of storage devices. Below we list the most significant ones:

• Cloud storage devices: This is storage space that is shared on various very powerful servers, they are located at different points and applications, browsers, authentication and web access are required for its use.

Some of the benefits provided by Cloud storage devices are:

• Total Cost of Ownership: when data is stored in the Cloud, there is no need to purchase hardware. You can increase or eliminate capacity on demand and transfer information that is accessed less frequently.

• Deployment time: The Cloud provides rapid access to the amount of storage needed at the indicated time, allowing you to focus on other application topics instead of wasting time reviewing the management of storage systems.

• Information Management: Focusing information in the Cloud provides greater benefit for new use cases. By using Cloud storage lifecycle management policies, you can run information management tasks.
• Hybrid Storage Devices: These combine 2 technologies to store data permanently.

• The first device combines magnetic and electronic technology.
• The second combines optical and magnetic technology.

• Electronic storage devices: these store data in electronic memory cells, for example, temporary and permanent electronic storage.
• Optical storage devices: These devices allow one to store data on plastic surfaces, thanks to its laser technology.

• Magnetic storage devices: It is possible to save data through a magnetic grouping of ferrous particles; examples are backup tapes, floppy disks and hard drives.

What are primary and secondary storage?

• Primary: this storage can save data temporarily, since when disconnected from electricity, all stored data is lost. RAM-type memories are found within this classification.
• Secondary: this storage can save data permanently, even when disconnected from electricity. Punched cards, magnetic and optical disks are found within this classification.

Storage Infrastructure

There is currently too much information, so it needs to be stored, protected, optimized, and well managed. And to meet this goal, infrastructure becomes one of the essential elements of storage architecture. These systems allow you to perform different tasks:

• Collection of data
• Automated backup

• Information Retrieval

• Data in the context of disasters

There are different types of media within storage infrastructure:  NL-SAS disks, SAS disks, and SAS flash solid-state disks. These disks can store information from different pools depending on capacity and performance needs.

The storage infrastructure provides access to the information through different protocols and access modes:

• CIFS and NFS for file-level access.
• FH, iSCSI, and FCoE to access information at the block level.

Storage infrastructure fulfills functions of security and efficiency such as:

• Perform block-level snapshots and mirrors.
• Checkpoints and replicas at the systems level.
• Optimize and maximize the use of available space.
• Decrease the disk space needed to save information.
• Compression of data.
• Delete duplicate copies

These are some of the security measures that together with the backup to traditional magnetic tape and the backup to disk are complements for the storage infrastructure.

Where is the data stored?

You can store a lot of information inside a computer; for example, the clusters on the hard drive is the place where the computer stores it. Clusters are circular sectors that store a certain amount of data. The amount that can be stored varies and depends solely on the operating system of the machine.

There are options for safeguarding data. The following stand out:

• Encrypted hard drives
• Storage in the Cloud
• Security copies
• Password managers for sensitive data

Type of data that is stored

A lot of information and different types of data are digitally stored in the storage systems; the following stand out:

The personal data of customers:

• The privacy policy must be communicated clearly and precisely to customers, so that they know how their data will be used.
• Preventive measures must be taken for the protection of the personal data of customers.
• Deliver secure functionality to provide a better customer experience.

Financial data:

• There must be total financial protection, without errors.
• Current and future accounting standards must be complied with.
• Comprehensive security of the systems and financial solutions must be provided.

Company information:

• The company's information must maintain scalability for growth in its systems and processes.
• Provide connected business operations for corporate processes.

• Maintain strong internal controls to properly track the stored data.

 

Data Storage Management

Data storage devices have the capacity to store data in their memory, facilitating the transfer and distribution of information to other computers. In addition, these devices work in the management of information storage, as tools of secure storage, also known as backup.

 

Storage in a database

Databases can store a large amount of data permanently. Usually, archived information is accessed and updated constantly.

In general, databases are archived in secondary memories (hard disks, optical disks, flash memories, servers). These databases are stored in files and there are different types of primary organizations that determine how records in a file are placed on a disk.

 

How to Organize Data

Some forms of primary organization are:

 

• Unordered Files: This mode places the records on the disk in no particular order; new records are placed at the end of the file.
• Sequential Files: maintains an ordering of records based on the value of a key code.
• Hash file: uses an addressing function that is applied within a given field to determine the location of records on disk.
• B Trees: they use a tree structure to locate records.

 

Software to manage data

Data management software is a computer system that helps users and applications archive, modify and analyze databases. Database technology, over time, becomes more complex so they manage to support increasingly greater amounts of data, driven by the current demand for permanent connection.

Data management software helps in many ways to simplify complex technology and ensure optimal database performance in a great variety of systems as well as in the Cloud.

 

Data accessibility

Databases arise from the need to save information in an orderly manner, but one of the most important factors is its accessibility, ensuring the immediate availability of the data to optimize the task or need of the user.

The purpose of this function is to provide the best service, being an indispensable resource in any type of company, whether small, medium-sized or large; the objective is to keep the data organized and accessible on a permanent basis.

 

Data Storage Technologies

Over the years, data storage has evolved radically, from magnetic tapes to large magnetic disks and from these to optical storage on CDs  and later to USB, and currently to the concept of the Cloud, where physical storage is not necessary.

The size of these storage devices has also been greatly reduced; now, gigabytes of data can be transported on a small memory card.

 

New Storage Technologies

Over time, the new storage technologies have evolved to facilitate the tasks and needs of the user, in terms of accessibility, speed and size. Some of the most significant and innovative are summarized below:

 

3D NAND:

For a long time, storage devices have used the new flash memory technology called 3D NAND, which evolved from its predecessor 2D NAND, allowing manufacturers to make devices with greater storage capacity.

One of the characteristics, and the way in which 3D NAND has evolved from its predecessor, is that now the cells are connected vertically, allowing connection with a larger number of transistors within the same space.

 

NVDIMM:

This is a new type of memory, a hybrid between the current RAM memory and the NAND memory of the SSDs. NVDIMM has better application performance, as it has a random-access memory for computers, greater verification and it optimizes the recovery time of the system, improving the endurance and reliability of the solid state unit.

There are four types of NVDIMMs:

• NVDIMM-F: available since 2014, system users can link storage DIMM together with traditional DIMM; this type has flash storage.

• NVDIMM-N: features flash storage and traditional DRAM within the same module, it has a backup power source in the event of power failure.
• NVDIMM-P: allows for storage inside the main memory of the computer; it uses ReRAM technology and a DDR5 interface. DRAM can access an external flash memory drive for cache.
• NVDIMM-X: it has NAND flash storage and volatile DRAM within the same module.

RDMA over converged Ethernet: this is a network protocol that provides remote accessibility directly to RDMA memory over an Ethernet network. There are two versions of RoCE, Ethernet link layer protocol (as opposed to RoCE v2, which is an internet layer protocol).

Although the RoCE protocol benefits from the features of a converged Ethernet network, the protocol can be used on a traditional Ethernet network.

Data compression: it allows for a reduction in the total volume of treatable information needed to represent a certain portion of data, using the least amount of space possible; one of its main characteristics is that the code obtained takes up less space than the original.

The compression of data consists mainly in searching for repetitions of information, and then only storing the data once together with the number of times it is repeated.

A compressed file has a greater number of encoded files, which can be extracted from it in different ways to carry out the decompression procedure.

The ultimate goal of compression is simple: to reduce the size of the data. Compressed data may or may not affect the quality of the information, so three types of compression are defined:

• Without real losses
• Subjectively lossless
• Subjectively lossy

All Flash Storage: this memory is a device that provides storage of large amounts of data in a small space, making it possible to read and write through a series of electrical signals.

This chip archives and transfers information from a computer to another type of digital device; it can also be erased or reprogrammed electronically. It is usually found in USB flash drives, MP3 players, digital cameras and in some solid state drives.

      The characteristics of all-flash memory are:

• Memory based on the use of semiconductors
• It can not be rewritten
• Information is not erased when the device is disconnected
• It can be connected to other devices
• It is portable and easy to handle
• It uses little energy
• It is economical and sturdy
• It uses thermal resistance.

The storage of data has evolved greatly over time; it has been a long process of using technology to organize, distribute and store the data and information deemed necessary. For this reason, the use of information storage memory is of utmost importance today.

 

Sources:

https://www.redhat.com/es/topics/data-storage

https://www.orbit.es/arquitectura-ti-como-elegir-el-mejor-almacenamiento-tipos-y-usos/

http://www.informaticamoderna.com/Disp_almacen.htm

https://www.uc3m.es/sdic/nosotros/medios/almacenamiento

https://techlandia.com/almacenan-datos-computadora-como_465731/

https://www.redeszone.net/2019/04/14/proteger-guardar-datos-archivos-segura/

https://www.vocalcom.com/es/blog/digital-customer-engagement/5-consejos-para-manejar-los-datos-personales-de-los-clientes/

https://www.oracle.com/mx/applications/erp/what-is-financial-management-system/

https://sites.google.com/site/seguridadinformaticalias/jnj

http://www.alegsa.com.ar/Dic/almacenamiento_de_bases_de_datos.php

https://www.quest.com/mx-es/solutions/database-management/

https://www.revistabyte.es/comparativa-byte-ti/1bases-de-datos-la-informacion-accesible-y-organizada873/

https://www.elgrupoinformatico.com/repaso-las-tecnologias-almacenamiento-actuales-t19307.html

https://hardzone.es/2018/03/30/que-es-3d-nand/

https://hardzone.es/2018/05/18/intel-3d-xpoint-nvdimm-sustituto-ram/

https://www.ecured.cu/Compresión_de_datos

https://www.euston96.com/memoria-flash/