The classic definition of Big Data
Since the 1990s, the concept of Big Data has been frequently mentioned and has been flourishing in various industries in the next two decades, and its applications can be found in economics, sociology, and technology-related industries. The analysts at Gartner, Inc. proposed a definition of Big Data in the early days of Big Data development: the 3-Vs, and let's take a look at the following Let's take a look at the following excerpts from the original article.
"Big data" is high-volume, -velocity and -variety information assets that demand cost-efficient, innovative forms of information processing for enhanced insight and decision making.
Volume data volume :
The ability to handle extremely large, low-density, and unstructured data is necessary to handle the computation and analysis of big data.
Velocity Data Transfer Speed :
In a big data environment, data often needs to be processed in real time to maximize its value, so a more continuous and fast data flow processing capability is required than traditional data processing. The transfer speed here usually refers to the speed of data input (receiving) and output (responding).
Facing Variety of Data Types :
The variety of data sources in Big Data, from early pure text to current multimedia resources on the Internet (pictures, audio, video, etc.), increases the difficulty of classifying, computing, and analyzing data, so data processing capability is very important.
In the era of information explosion, the information about the concept of big data is also increasingly updated. The 3-Vs definition has been challenged by the authorities in the industry in the past 10 years, and the 4-Vs and 5-Vs characteristics of "Veracity" and "Value" have been added at the later stage, but when referring to big data, the original 3-Vs definition is still the most frequently mentioned and classic big data concept. When referring to Big Data, the original 3-Vs definition is still the most frequently mentioned and classic Big Data concept.
How else can we understand Big Data besides the 3-Vs?
In fact, the term Big Data is a vague conceptual term that generally refers to large amounts of data that are difficult to process using traditional data processing methods. Because of the sheer volume, complexity, and unstructured nature of Big Data, it is difficult for traditional data processing software or systems to cope with and manage Big Data data, but the information hidden beneath the sheer volume of Big Data opens up the possibility of solving many previously intractable problems.
The volume of big data has been increasing cumulatively with recent hardware and software developments. A forecast from the International Data Corporation (IDC) reports that the global data volume will grow from 4.4 Zettabytes exponentially to 44 Zettabytes between 2013 and 2020 (1 Zettabyte = 1012 GB), and is expected to reach 163 Zettabytes by 2025, which means that the global data volume will double about every 2 years.
So ... what is the relationship between AI and Big Data?
Simply put, AI and Big Data complement each other: AI, or more precisely, machine learning and deep learning models require very large amounts of data for computation and improvement. While big data is expected to provide data owners with more information than they were able to uncover before, it is difficult to manage big data databases efficiently without relying on new algorithms such as AI, and traditional algorithms that cannot handle large amounts of real-time data are not helpful when faced with big data. When people need to find out meaningful information behind big data, advanced AI technologies can efficiently process big data and import it into machine learning or deep learning to improve its overall model architecture.
Conclusion
Big Data has opened up new business models and opportunities, but data engineers often spend more than 50% of their time cleaning, filtering, or normalizing raw data before importing it into AI or machine learning algorithms.
AI can effectively help manage big data
The scale of big data will become larger and larger, and the use of AI, machine learning and deep learning to manage big data is the trend of data processing in the future. The interaction between AI and big data will be a constant challenge in the field of data science.
Comments
Post a Comment