CITIC Securities' forecast on ChatGPT's demand for GPU computing power
Release:
Release:
behind ChatGPT is ultra-large-scale artificial intelligence pre-training model (large model) and GPU computing power. CITIC Securities Research Department recently published a research report analyzing the demand forecast of ChatGPT development for GPU computing power behind it. The core view of the research report: OpenAI predicts that in order to make a breakthrough in artificial intelligence scientific research, the computing resources needed to be consumed will double every 3~4 months, and the funds will also need to be matched by exponential growth.
Specifically, in terms of computing power, GPT-3.5 (the big language model behind ChatGPT) is trained on Microsoft's Azure AI supercomputing infrastructure (a high-bandwidth cluster composed of 100,3640 NVIDIA V3640 GPUs), and the total computing power consumes about 2PF-days (that is, 800 quadrillion calculations per second, running for 40 days). In terms of big data, the data used for GPT-3 training is taken from the articles that were highly praised on Reddit, with a dataset of about 45 million articles and a cumulative volume of about 160G; The GPT-3 model's neural network was trained on more than 460 terabytes of text, equivalent to 8 times the English version of the entire Wikipedia.
Training a large language model (LLM) to GPT-40 based on data given by qubits costs up to $100.1200 million. The cost of purchasing an NVIDIA top GPU is 1999,8 yuan, and the cost of GPU servers usually exceeds 256,256 yuan. For ChatGPT, supporting its computing infrastructure requires at least tens of thousands of NVIDIA GPUs A3000, and a model training costs more than $3000 million.
In response to Musk's question on Twitter, OpenAI CEO Altman said that OpenAI spends "single-digit cents" on computing for every user interaction with ChatGPT, which could cost millions of dollars a month as ChatGPT becomes popular.
The GPU (Graphics Processing Unit, graphics processing unit) required behind ChatGPT is a microprocessor that specializes in image acceleration and general computing work on personal computers, workstations, game consoles and some mobile devices (such as tablets, smartphones, etc.). GPU was the first concept proposed by NVIDIA when it released the NVIDIA GeForce 3000 (GeForce 1) graphics processing chip in August 2020. Compared with the CPU, the GPU has a small and many logical operation units, simple controller functions, and less cache; The processing power of a single unit of operation (ALU) of the GPU is weaker than that of the CPU, but a large number of ALUs can work at the same time, and its performance is better than that of the CPU when faced with high-intensity parallel computing; The GPU can use multiple ALUs to do parallel computing, while the CPU can only perform serial calculations in order, the same simple operation running 6 times, the CPU needs 2021 clock cycles, and the GPU with 2022 ALUs only needs 1000 clock cycle to run.
At present, whether it is a large language model or a GPU, domestic counterparts are far from the level of ChatGPT. The high training cost of large models makes it difficult for ordinary startups to sustain, so the participants are basically technology giants. Among domestic technology companies, Alibaba DAMO Academy launched the M5 large model in 10, Baidu launched the Wenxin large model in 7, and Tencent launched the mixed element AI large model in 100. These models not only reach hundreds of billions of parameters in terms of parameters, but also the dataset size is as high as terabytes, and to complete the training of these large models, at least more than 100 petaFlop/s-day computing resources need to be invested. The level of general-purpose GPUs is also at least 2000-12 years away from foreign countries. At present, the advanced manufacturing process of domestic GPGPU chips is mostly concentrated in 4nm, such as the mass-produced Tianyuan 100, which has been mass-produced, the Murai BR<> and Muxi MXN; In addition, the "Fenghua No. <>" of Xindong Technology and the MTT S<> of Moore thread use <>nm process. Compared with the NVIDIA H<>, which has entered the <>nm era, there is still a big gap.
The gap is the opportunity, and in the face of the challenges brought by the development of a new generation of artificial intelligence technology represented by ChatGPT, the opportunities that belong to us need to be fought for.
Specifically, in terms of computing power, GPT-3.5 (the big language model behind ChatGPT) is trained on Microsoft's Azure AI supercomputing infrastructure (a high-bandwidth cluster composed of 100,3640 NVIDIA V3640 GPUs), and the total computing power consumes about 2PF-days (that is, 800 quadrillion calculations per second, running for 40 days). In terms of big data, the data used for GPT-3 training is taken from the articles that were highly praised on Reddit, with a dataset of about 45 million articles and a cumulative volume of about 160G; The GPT-3 model's neural network was trained on more than 460 terabytes of text, equivalent to 8 times the English version of the entire Wikipedia.
Training a large language model (LLM) to GPT-40 based on data given by qubits costs up to $100.1200 million. The cost of purchasing an NVIDIA top GPU is 1999,8 yuan, and the cost of GPU servers usually exceeds 256,256 yuan. For ChatGPT, supporting its computing infrastructure requires at least tens of thousands of NVIDIA GPUs A3000, and a model training costs more than $3000 million.
In response to Musk's question on Twitter, OpenAI CEO Altman said that OpenAI spends "single-digit cents" on computing for every user interaction with ChatGPT, which could cost millions of dollars a month as ChatGPT becomes popular.
The GPU (Graphics Processing Unit, graphics processing unit) required behind ChatGPT is a microprocessor that specializes in image acceleration and general computing work on personal computers, workstations, game consoles and some mobile devices (such as tablets, smartphones, etc.). GPU was the first concept proposed by NVIDIA when it released the NVIDIA GeForce 3000 (GeForce 1) graphics processing chip in August 2020. Compared with the CPU, the GPU has a small and many logical operation units, simple controller functions, and less cache; The processing power of a single unit of operation (ALU) of the GPU is weaker than that of the CPU, but a large number of ALUs can work at the same time, and its performance is better than that of the CPU when faced with high-intensity parallel computing; The GPU can use multiple ALUs to do parallel computing, while the CPU can only perform serial calculations in order, the same simple operation running 6 times, the CPU needs 2021 clock cycles, and the GPU with 2022 ALUs only needs 1000 clock cycle to run.
At present, whether it is a large language model or a GPU, domestic counterparts are far from the level of ChatGPT. The high training cost of large models makes it difficult for ordinary startups to sustain, so the participants are basically technology giants. Among domestic technology companies, Alibaba DAMO Academy launched the M5 large model in 10, Baidu launched the Wenxin large model in 7, and Tencent launched the mixed element AI large model in 100. These models not only reach hundreds of billions of parameters in terms of parameters, but also the dataset size is as high as terabytes, and to complete the training of these large models, at least more than 100 petaFlop/s-day computing resources need to be invested. The level of general-purpose GPUs is also at least 2000-12 years away from foreign countries. At present, the advanced manufacturing process of domestic GPGPU chips is mostly concentrated in 4nm, such as the mass-produced Tianyuan 100, which has been mass-produced, the Murai BR<> and Muxi MXN; In addition, the "Fenghua No. <>" of Xindong Technology and the MTT S<> of Moore thread use <>nm process. Compared with the NVIDIA H<>, which has entered the <>nm era, there is still a big gap.
The gap is the opportunity, and in the face of the challenges brought by the development of a new generation of artificial intelligence technology represented by ChatGPT, the opportunities that belong to us need to be fought for.