As someone who lived through the first wave of supercomputers (I worked with Cray Y-MP models), it makes me very happy to see the second wave. For a while I thought supercomputing was dead and we would just be connecting lots of PCs with a network and calling that "supercomputers".
I still remember how my mind was blown when I first learned that all of the memory in a Cray Y-MP was static RAM. Transistor-based flip-flops: extremely power hungry, but also very fast. Another way of looking at it is that all of its RAM was what we call "cache".
All of a sudden you don't care so much about the inefficiencies of walking linked lists or trees. When everything is "already in cache", you can worry less about cache efficient algorithms!
1 cycle memory access latency is one of the reasons why tiny embedded MCUs can do things with a fraction of the MHZ of their larger counterparts.
Now days of course it is all about tons of memory, tons of bandwidth, craptons of compute, and planning the flow of data ahead of time.
But that's why there's a "super" in "supercomputers"!
Cray loaded a ton of static memory into their computers, then liquid-cooled the whole thing. Sure, the power requirements were through the roof, and you had a whole huge chiller system which you had to run and hope it doesn't fail. If it did fail, you really wanted to shut the machine down fast. From what I recall there was also an emergency propeller inside the back case of the Y-MP 2E, and yes propeller is a much better name for this thing than a "fan". It would delay the inevitable, although dumping those tens of kW of heat into your server room was not something you ever wanted to do.
The whole point of all this was that you could do things that you couldn't with "normal" computers. That's why those were called "supercomputers". And I'm so glad that after a hiatus of about 30 years we're getting another wave of exceptional machines, which aren't just bigger PCs.
L3 cache is all SRAM but you can have pretty significant delays accessing it. Even the fastest memory cells will build up significant addressing delays as you increase in scale.
They presumably where born 20+ years before they started working professionally on the Y-MP. So they could easily have been alive or even a teen in 1964 when the CDC 6600 was released.
It may be presented as seamless unified memory, but it isn’t. Underlying framework still has to figure out how to allocate your data to minimize cross-unit talk. Each unit has independent CPU, GPU and (V)RAM, but units are interconnected via very fast network.
> EDIT: found answer to my own question in the datasheet: "The NVIDIA Grace CPU combines 72 Neoverse V2 Armv9 cores with up to 480GB of server-class LPDDR5X memory with ECC."
So, this is not stacked RAM like HBM, it's LPDDR5X which a quick search says is 8.5Gbps.
I think distributed computing will go away soon. As computers become more powerful, the cost of "distributing" and transferring the data would be more than simply executing everything locally. Yes, you can still split the task amongst different nodes, or give different problems to different nodes, but the use-case would mostly be solving distinct problems on each note, that splitting the same task across multiple computers.
Also, with quantum computers, the parallelization/"distribution" of tasks will be done within the same machine, as it can try all solutions and the same time without having to do divide-et-impera algorithms.
Also, in the future, the algorithms will be a lot simpler, and just have FPGA-s like AI chips, where there is no software, the model is directly modelled in the hardware, so each computation is instant (just the time it takes to propagate the electrons or light through the circuit).
There’s usecases beyond just ML. Sap Hana could theoretically run on this with greater performance. Same goes for a database. Scaling vertically solves a lot of challenges with distributed ledgers.
No, kinda the opposite: mainframes use a litany of sophisticated and special purpose hardware to achieve their tasks (eg hardware io _channels_), while this is a massively overgrown instance of a single kind of hardware (vector processor).
The thing that made the lightbulb go off in my head w.r.t. mainframes was understanding that mainframe I/O channels are computers. The mainframe had several dedicated computers that each specifically handled I/O to a terminal, printer, disk or tape drive, punchcard reader, etc. Made I/O programming a breeze, as you just had to tell the channel to read or write, specifying a block of memory to use as a buffer, and the channel would DMA out the data to be written, or DMA in read-in data.
It also explains the reason why despite having middling CPU power, mainframes had a reputation for stinkloads of I/O bandwidth so they could process everyone's credit card transactions, airline bookings, and that: the mainframe's CPU was involved very little in I/O, that was all handled by the channel processors!
Came here to post exactly the same link. Not just that, but 1 exaflops of sparse FP8.
In comparison, Frontier is 1 exaflops of dense FP64. Try to run this Nvidia system as dense FP64, and it performance will degrade two orders of magnitude.
Don't get me wrong, the machine is really impressive, but the advertisement is quite misleading.
Being single-threaded is why the original Crysis is hard on computers now(!!). When Crysis was being developed people still thought 5-10Ghz was coming Any Day Now.
2007 was well into Intel Core territory, Intel had given up on clockspeed-at-all-costs Pentium 4 Netburst, so it was generally accepted that clocks weren't going to keep up as fast as before.
Even if Crytek took a year off developing new games after the release of Far Cry, that still puts the start of Crysis development before the announcement of Core (early 2005 IIRC).
You might could hit 5Ghz by overclocking the appropriate CPU. I remember 10 years ago looking at the liquid cooling setups needed to hit 5Ghz on a core.
My 12400F boosts to 5GHz for the majority of each day. It's literally normal for me. I've gotten it to almost 5.3GHz before, but it's not really stable above 5.2GHz.
TOP500 uses FP64 performance in ranking. nVidia's 1 exaflop claim is the ~4 petaflops of FP8/INT8 * 256. FP64 performance of modern nvidia gpu's is actually far, far less. The ratio to FP32 isn't even 2:1 anymore (not since Pascal I think) since they realize most machine learning is done with FP32 or less.
64 bit (or 'double precision') is still king in the HPC world though, as it is what you will find in large numerical solutions in fields like computation fluid dynamics, nuclear physics, etc.
Ah fair, I should've known. I suppose the precision is still required for scientific purposes. Thankfully ML stuff now gets more appropriate precision for a speed increase.
1 exaflop + 144TB memory
https://nvidianews.nvidia.com/news/nvidia-announces-dgx-gh20...