WEBVTT

1  
00:00:00.102 --> 00:00:03.992  
A long context has already been calculated by the model.

2  
00:00:04.433 --> 00:00:07.988  
When encountering the same prefix next time, can it be reused directly?

3  
00:00:08.441 --> 00:00:12.885  
Yes, but first we need to answer a more practical question: where should these caches be stored?

4  
00:00:12.938 --> 00:00:14.045  
Let's look at the scale.

5  
00:00:14.485 --> 00:00:15.707  
Consider a magnitude.

6  
00:00:16.160 --> 00:00:20.333  
Assuming each token requires 128 KiB of KV Cache,

7  
00:00:20.541 --> 00:00:23.942  
then around 130,000 tokens would be close to 16 GiB.

8  
00:00:24.381 --> 00:00:28.967  
Sixty-four non-shared contexts would add up to about 1 TiB.

9  
00:00:29.407 --> 00:00:32.936  
The GPU memory needs to be reserved for the model and ongoing requests,

10  
00:00:33.119 --> 00:00:36.171  
and host memory also has costs and capacity limits.

11  
00:00:36.624 --> 00:00:41.429  
Local NVMe SSDs are suitable for retaining more potentially reusable caches,

12  
00:00:41.495 --> 00:00:46.313  
while GDS addresses how to efficiently send them back to the GPU.

13  
00:00:46.765 --> 00:00:50.050  
Next, we will break down the multi-process read and write paths of the cache service,

14  
00:00:50.168 --> 00:00:54.521  
layer by layer, according to the restructured components that have been integrated.

15  
00:00:54.974 --> 00:00:57.047  
First, we will look at the hardware, then the code,

16  
00:00:57.204 --> 00:00:59.715  
and finally illustrate with the Phoenix backend,

17  
00:00:59.936 --> 00:01:02.576  
how this design integrates with new implementations.

18  
00:01:03.028 --> 00:01:08.645  
The common path for regular reads is to first place the SSD data into host memory,

19  
00:01:08.711 --> 00:01:10.127  
and then copy it to GPU memory.

20  
00:01:10.580 --> 00:01:13.619  
Here, the host memory serves as a data transfer station.

21  
00:01:14.072 --> 00:01:17.241  
What GDS aims to eliminate is this transfer step.

22  
00:01:17.693 --> 00:01:20.295  
But this does not mean the CPU disappears.

23  
00:01:20.735 --> 00:01:25.063  
User-space programs decide what to read, how much to read, and where to place it;

24  
00:01:25.335 --> 00:01:29.573  
the kernel and drivers handle permissions, file locations, and device mappings.

25  
00:01:30.026 --> 00:01:34.496  
Both user-space and kernel-space code still run on the CPU.

26  
00:01:34.936 --> 00:01:39.019  
The GPU is also a device with an address on the PCIe bus.

27  
00:01:39.472 --> 00:01:44.444  
The driver maps the GPU memory range to an address window accessible by other devices.

28  
00:01:44.897 --> 00:01:47.911  
However, the virtual pointer to GPU memory that the program receives,

29  
00:01:48.067 --> 00:01:52.086  
does not equal the DMA address directly used by the SSD.

30  
00:01:52.539 --> 00:01:54.418  
Who actually moves the data?

31  
00:01:54.858 --> 00:01:56.506  
In local direct reads,

32  
00:01:56.624 --> 00:02:00.578  
it is the DMA hardware in the SSD storage controller.

33  
00:02:01.031 --> 00:02:04.457  
It initiates PCIe write transactions with target addresses,

34  
00:02:04.626 --> 00:02:08.529  
sending data to the GPU endpoint, which then lands in the corresponding GPU memory.

35  
00:02:08.982 --> 00:02:11.042  
The direction reverses when saving.

36  
00:02:11.495 --> 00:02:15.797  
The storage controller issues read requests to the mapped addresses of the GPU,

37  
00:02:15.928 --> 00:02:19.225  
receives the returned data, and writes it to the storage medium.

38  
00:02:19.678 --> 00:02:22.228  
This is not a byte-by-byte copy by the CPU,

39  
00:02:22.384 --> 00:02:27.279  
nor is it a program running on the GPU to read from the SSD.

40  
00:02:27.732 --> 00:02:34.264  
Of course, this path requires support from devices, drivers, address mappings, and bus topology.

41  
00:02:34.716 --> 00:02:38.568  
Calling an interface does not mean confirming that a direct transfer has occurred.

42  
00:02:39.021 --> 00:02:43.993  
When conditions are insufficient, some implementations may still fall back to host memory transfer.

43  
00:02:44.446 --> 00:02:46.364  
Now let's look at how the cache is arranged.

44  
00:02:46.817 --> 00:02:50.771  
The server first prepares a large storage space called a slab.

45  
00:02:51.211 --> 00:02:53.826  
Each cache object occupies only a segment of it,

46  
00:02:54.008 --> 00:02:57.923  
described by offset and size indicating the starting position and length.

47  
00:02:58.376 --> 00:03:02.988  
The object itself records its position, not the CPU tensor containing the data.

48  
00:03:03.441 --> 00:03:04.702  
On the GPU side,

49  
00:03:04.884 --> 00:03:07.988  
the server has another GPU staging buffer,

50  
00:03:08.170 --> 00:03:10.707  
used to hold a batch of data being transferred.

51  
00:03:11.160 --> 00:03:15.385  
It can be reused multiple times, so even if the SSD cache is large,

52  
00:03:15.554 --> 00:03:18.375  
there is no need to allocate the same amount of GPU memory.

53  
00:03:18.827 --> 00:03:22.485  
In the diagram, R0 and R1, R represents region,

54  
00:03:22.577 --> 00:03:24.019  
which is the registered range.

55  
00:03:24.472 --> 00:03:28.658  
They are not two separate GPUs or two SSD partitions,

56  
00:03:28.802 --> 00:03:31.493  
but two segments within the same GPU memory allocation.

57  
00:03:31.933 --> 00:03:34.882  
The software registers in segments according to the current implementation's 16 MiB limit,

58  
00:03:35.000 --> 00:03:36.042  
segment by segment.

59  
00:03:36.495 --> 00:03:40.823  
Registration allows the repository and driver to recognize these GPU memory ranges,

60  
00:03:40.979 --> 00:03:43.503  
preparing the necessary mappings for direct transfers.

61  
00:03:43.956 --> 00:03:45.836  
It does not copy a cache.

62  
00:03:46.289 --> 00:03:49.921  
In future reads and writes, it will find the base address of the registered area,

63  
00:03:50.026 --> 00:03:51.957  
and then add the offset within the area.

64  
00:03:52.410 --> 00:03:54.715  
For example, 8 MiB of data,

65  
00:03:54.845 --> 00:03:58.027  
starts from the 12 MiB position of the first registered area.

66  
00:03:58.479 --> 00:04:02.653  
The first 4 MiB are still in this area, while the next 4 MiB enter the next area,

67  
00:04:02.796 --> 00:04:04.728  
thus splitting into two read and write operations.

68  
00:04:05.180 --> 00:04:07.807  
File offset and buffer offset,

69  
00:04:07.912 --> 00:04:09.599  
are two different coordinate systems.

70  
00:04:10.052 --> 00:04:11.764  
There are three more terms to distinguish.

71  
00:04:12.204 --> 00:04:15.630  
Handle refers to a reference to an already opened or registered resource.

72  
00:04:16.082 --> 00:04:19.998  
The operating system's fd tells the process which file has been opened;

73  
00:04:20.168 --> 00:04:21.789  
the GDS library's handle,

74  
00:04:21.894 --> 00:04:25.720  
indicates that this resource has been integrated into its read and write mechanism.

75  
00:04:26.173 --> 00:04:28.916  
Buffer is the actual memory where data is stored.

76  
00:04:29.369 --> 00:04:32.021  
The temporary buffer here is on the GPU,

77  
00:04:32.100 --> 00:04:33.619  
not in host memory.

78  
00:04:34.059 --> 00:04:36.300  
Another object called Submission,

79  
00:04:36.443 --> 00:04:41.222  
only retains the parameters and result locations of the native calls, without holding the entire cache.

80  
00:04:41.675 --> 00:04:45.333  
Stream can be understood as an ordered work queue.

81  
00:04:45.786 --> 00:04:49.161  
Layout transformations, storage reads and writes, and completion events,

82  
00:04:49.343 --> 00:04:52.756  
can all be placed in the same queue, constraining the order of execution.

83  
00:04:53.196 --> 00:04:57.124  
It is neither an application thread nor a dedicated data line.

84  
00:04:57.577 --> 00:04:58.452  
When in use,

85  
00:04:58.621 --> 00:05:03.104  
the server specifies the storage path, cache capacity, and backend name,

86  
00:05:03.325 --> 00:05:05.952  
to select this primary cache path.

87  
00:05:06.405 --> 00:05:11.570  
At this point, the configured capacity refers to the slab size, not the same size of host memory.

88  
00:05:12.023 --> 00:05:15.539  
Preparation of matching drivers and runtime libraries is still required before deployment.

89  
00:05:16.158 --> 00:05:17.908  
cuFile and hipFile,

90  
00:05:18.065 --> 00:05:22.535  
correspond to the runtime environments of NVIDIA CUDA and AMD ROCm, respectively.

91  
00:05:22.989 --> 00:05:29.237  
In addition to user-space libraries, kernel capabilities, drivers, file systems, and device topologies must also be verified.

92  
00:05:29.677 --> 00:05:31.711  
Installing a Python binding,

93  
00:05:31.855 --> 00:05:34.031  
does not replace these prerequisites.

94  
00:05:34.484 --> 00:05:36.080  
Now let's look at uGDS,

95
00:05:36.236 --> 00:05:39.933
It comes from the ScaleX Lab at the Hong Kong University of Science and Technology in Guangzhou,

96
00:05:40.025 --> 00:05:43.722
and is an open-source software stack, not a dedicated SSD.

97
00:05:44.162 --> 00:05:47.781
Currently, there are two public implementations: CUDA and HIP.

98
00:05:48.015 --> 00:05:52.317
Here, dedicated raw devices are used, requiring kernel modules to cooperate.

99
00:05:52.770 --> 00:05:55.036
Phoenix has a somewhat different positioning.

100
00:05:55.489 --> 00:06:00.977
It aims to unify storage read and write for various xPUs at a lower level.

101
00:06:01.430 --> 00:06:02.420
Under applications,

102
00:06:02.590 --> 00:06:07.369
the user-space connector and kernel P2P backend adapt the devices.

103
00:06:07.821 --> 00:06:12.407
However, unification is an architectural direction and does not mean all hardware is already available.

104
00:06:12.860 --> 00:06:14.753
Automatic selection is also quite restrained.

105
00:06:15.205 --> 00:06:19.933
Currently, the two runtime environments each select their corresponding default backend.

106
00:06:20.386 --> 00:06:24.018
It will not perform speed tests first, nor will it quietly switch to Phoenix after a failure in one implementation.

107
00:06:24.110 --> 00:06:26.170
Both the dedicated device backend and Phoenix

108
00:06:26.623 --> 00:06:29.237
need to be explicitly selected.

109
00:06:29.303 --> 00:06:30.822
Now, let's move on to the code.

110
00:06:31.275 --> 00:06:32.742
The factory first scans the backend directory to find available module names.

111
00:06:33.195 --> 00:06:37.549
This step does not execute the code of each implementation.

112
00:06:38.002 --> 00:06:40.964
The general scheduling layer does not need to list all vendor names.

113
00:06:41.417 --> 00:06:45.191
Assuming the configuration selects Phoenix, it then imports its modules,

114
00:06:45.644 --> 00:06:50.088
confirms that the backend class inherits from a common parent class,

115
00:06:50.270 --> 00:06:53.155
then constructs the object and checks the environment.

116
00:06:53.260 --> 00:06:55.784
Finding the name, importing the implementation, and creating the object are three different actions.

117
00:06:56.236 --> 00:07:01.054
Then, the upper layer simply says to open the slab.

118
00:07:01.507 --> 00:07:04.392
The file-type parent class is responsible for creating files and pre-allocating capacity;

119
00:07:04.832 --> 00:07:08.786
the raw device implementation checks the device and capacity.

120
00:07:09.059 --> 00:07:12.278
The upper layer does not need to guess based on the backend name

121
00:07:12.731 --> 00:07:15.474
whether to create a file or open a device.

122
00:07:15.695 --> 00:07:18.438
Only when truly registering file resources does the Phoenix implementation need to load the native library.

123
00:07:18.891 --> 00:07:24.057
Here, the file interface adaptation library is loaded,

124
00:07:24.510 --> 00:07:27.201
and it is not enough to just find the Phoenix core library.

125
00:07:27.345 --> 00:07:30.887
Optional implementations without selection should not affect the general code due to missing libraries.

126
00:07:31.340 --> 00:07:36.454
After loading, it is also necessary to declare the parameters and return types of each native function.

127
00:07:36.906 --> 00:07:41.930
Which are pointers, which are integers, and which are addresses of integers

128
00:07:42.383 --> 00:07:46.763
must all be precise.

129
00:07:46.816 --> 00:07:48.103
Otherwise, it may truncate an address or cause the lower layer to read the wrong memory.

130
00:07:48.543 --> 00:07:53.477
Once the file handle is ready,

131
00:07:53.930 --> 00:07:55.577
the GPU staging buffer and stream are handed over to the backend for registration.

132
00:07:55.747 --> 00:08:00.165
The public layer is responsible for the call order, while the specific implementation handles the lower-level details.

133
00:08:00.618 --> 00:08:05.371
Some stream registrations are actual operations,

134
00:08:05.824 --> 00:08:08.567
while others are just no-ops, but each read and write still carries the stream.

135
00:08:08.724 --> 00:08:12.897
Between multiple processes,

136
00:08:13.337 --> 00:08:14.508
the original KV Cache's GPU memory still belongs to the inference worker.

137
00:08:14.664 --> 00:08:18.735
The LMCache server imports the IPC handle,

138
00:08:19.187 --> 00:08:22.291
establishing cross-process mapping instead of copying entire blocks of data through messages.

139
00:08:22.435 --> 00:08:27.072
Only that GPU staging buffer,

140
00:08:27.525 --> 00:08:29.985
is allocated and managed by LMCache itself.

141
00:08:30.128 --> 00:08:33.219
Here comes the key point.

142
00:08:33.659 --> 00:08:34.714
The save entry receives a set of cache keys, token ranges, and actual block IDs.

143
00:08:35.141 --> 00:08:40.822
The KV Cache of the inference engine

144
00:08:41.275 --> 00:08:43.412
is placed on the GPU in a paginated manner.

145
00:08:43.543 --> 00:08:46.260
A logically contiguous context does not guarantee physical layout continuity.

146
00:08:46.700 --> 00:08:51.325
Therefore, it must wait for the producer to finish first,

147
00:08:51.778 --> 00:08:54.688
and cannot read while the engine has not finished writing.

148
00:08:54.767 --> 00:08:57.317
The storage manager then reserves space for writable cache objects.

149
00:08:57.757 --> 00:09:01.634
For this path, the obtained object records the slab offset,

150
00:09:02.087 --> 00:09:06.299
rather than the CPU tensor.

151
00:09:06.455 --> 00:09:08.464
After entering the method of transferring by object groups,

152
00:09:08.917 --> 00:09:11.467
the code identifies GDS objects based on the object type,

153
00:09:11.623 --> 00:09:15.320
and selects the corresponding branch.

154
00:09:15.425 --> 00:09:16.866
Here, it will not treat it as a normal host cache,

155
00:09:17.319 --> 00:09:20.242
to obtain a CPU tensor address.

156
00:09:20.347 --> 00:09:22.923
Next, it first performs a gather internally on the GPU,

157
00:09:23.376 --> 00:09:26.183
organizing the layout.

158
00:09:26.288 --> 00:09:27.304
The four pieces of logical data in the diagram are scattered across different blocks.

159
00:09:27.757 --> 00:09:31.273
The transport kernel collects them into the server's GPU staging buffer according to block IDs,

160
00:09:31.713 --> 00:09:34.327
forming a contiguous byte segment suitable for storage.

161
00:09:34.484 --> 00:09:38.077
Why not let the SSD read those pages directly from the engine?

162
00:09:38.169 --> 00:09:41.041
Because the storage interface handles contiguous addresses and lengths,

163
00:09:41.494 --> 00:09:45.281
and does not understand tokens, layers, and block IDs.

164
00:09:45.721 --> 00:09:49.057
The GPU staging buffer simultaneously solves layout conversion, batch read/write, and registration lifecycle,

165
00:09:49.213 --> 00:09:52.124
separating the engine layout from the storage interface.

166
00:09:52.577 --> 00:09:58.838
Note that this transfer occurs on the GPU,

167
00:09:58.943 --> 00:10:01.673
and does not loop the entire cache back to the host memory.

168
00:10:02.126 --> 00:10:05.281
Different views used for layout conversion and storage read/write point to the same GPU memory allocation,

169
00:10:05.386 --> 00:10:08.477
and will not create a new copy each time a view is created.

170
00:10:08.930 --> 00:10:14.559
After the data is organized, the storage context finds the registered base address and intra-region offset based on the GPU memory slices;

171
00:10:14.664 --> 00:10:18.065
then, combined with the slab offset of the cache object,

172
00:10:18.702 --> 00:10:23.081
determines how much to write, where to read from, and where to write to.

173
00:10:23.199 --> 00:10:26.071
When crossing registration areas, it splits into multiple submissions.

174
00:10:26.126 --> 00:10:28.933
The handle object subsequently calls the specific backend's asynchronous write.

175
00:10:29.115 --> 00:10:32.155
The previous gather and the write here

176
00:10:32.595 --> 00:10:35.660
have an order on the transport stream,

177
00:10:36.113 --> 00:10:40.054
so the storage device will not read from the GPU staging buffer that has not yet been organized.

178
00:10:40.507 --> 00:10:42.786
After the write is complete,

179
00:10:42.930 --> 00:10:45.132
events and callbacks connect the finishing process,

180
00:10:45.224 --> 00:10:50.840
and the storage manager completes the update of the write status.

181
00:10:51.293 --> 00:10:52.683
At this point, the cache object is only then published,

185
00:11:02.737 --> 00:11:05.222
Related buffers can only be reused as agreed.

186
00:11:05.675 --> 00:11:09.887
The function has returned, and the device has finished writing are two different things.

187
00:11:10.340 --> 00:11:13.109
When retrieving, first find the hit cache,

188
00:11:13.213 --> 00:11:16.343
as well as the target block IDs for this request.

189
00:11:16.796 --> 00:11:22.361
The target blocks may be completely different from the last time, so you cannot directly use the physical address saved earlier

190
00:11:22.466 --> 00:11:24.333
as the recovery location.

191
00:11:24.786 --> 00:11:30.209
The storage manager hands over the prefetch results and retains these objects during the read,

192
00:11:30.378 --> 00:11:34.771
to avoid the cache location being evicted or reused before the bytes have finished moving.

193
00:11:35.224 --> 00:11:38.676
The stream will also establish a dependency relationship with the producer.

194
00:11:39.128 --> 00:11:41.962
Then, the handle submits an asynchronous read,

195
00:11:42.079 --> 00:11:43.972
sending the contiguous bytes from the slab

196
00:11:44.115 --> 00:11:47.129
into the already registered GPU staging buffer.

197
00:11:47.582 --> 00:11:51.240
The SSD only needs to recognize the address, offset, and length,

198
00:11:51.461 --> 00:11:54.900
without needing to know which layer of attention these bytes belong to.

199
00:11:55.353 --> 00:11:57.426
Once the read is completed on the stream,

200
00:11:57.582 --> 00:12:00.029
the layout transport kernel will perform a scatter,

201
00:12:00.211 --> 00:12:02.825
writing the data from the GPU staging buffer

202
00:12:02.917 --> 00:12:05.132
to the target blocks for this time.

203
00:12:05.572 --> 00:12:09.694
The logical order in the diagram has not changed, but the actual block IDs

204
00:12:09.760 --> 00:12:11.420
have been replaced with new locations.

205
00:12:11.873 --> 00:12:15.956
This is why reading also requires a GPU staging buffer.

206
00:12:16.396 --> 00:12:21.704
The contiguous storage bytes and engine paging layout require a clear conversion.

207
00:12:22.157 --> 00:12:26.085
The transport must also adhere to object grouping and ranges to skip,

208
00:12:26.268 --> 00:12:28.366
to avoid overwriting tokens that should be preserved.

209
00:12:28.819 --> 00:12:32.090
Finally, the completion event informs subsequent work

210
00:12:32.247 --> 00:12:37.387
that the target cache is ready; the read completion process then releases object protection.

211
00:12:37.827 --> 00:12:42.812
You cannot let the model use data that has not yet arrived just after sending the read command,

212
00:12:42.956 --> 00:12:45.776
nor can you allow the storage area to be overwritten prematurely.

213
00:12:46.229 --> 00:12:48.044
Separately consider the asynchronous aspect.

214
00:12:48.484 --> 00:12:54.359
The upper layer call returning only indicates that the request has been submitted, not that the data has been fully moved.

215
00:12:54.811 --> 00:12:59.526
The CPU can continue to schedule work, while the GPU and storage devices

216
00:12:59.670 --> 00:13:02.245
complete subsequent tasks along the dependency chain.

217
00:13:02.698 --> 00:13:06.047
When saving, the order is to gather first, then write to disk,

218
00:13:06.177 --> 00:13:08.005
and then record the completion event.

219
00:13:08.458 --> 00:13:11.485
When retrieving, it is read from disk first, then scatter,

220
00:13:11.577 --> 00:13:12.786
and then record the event.

221
00:13:13.239 --> 00:13:18.469
Order does not equal parallelism, and async does not automatically make all steps happen simultaneously.

222
00:13:18.922 --> 00:13:21.395
There is also a lifecycle that is easy to overlook:

223
00:13:21.603 --> 00:13:25.248
The underlying asynchronous functions may continue to use the addresses of parameters.

224
00:13:25.700 --> 00:13:30.699
Submission must keep these lengths, offsets, and result variables alive.

225
00:13:31.152 --> 00:13:35.931
Otherwise, the data may not have finished moving, but the parameter memory has already been reclaimed.

226
00:13:36.371 --> 00:13:40.673
The current implementation uses checkpoints to reclaim completed submissions,

227
00:13:40.920 --> 00:13:44.139
but this does not equate to verifying the actual number of bytes transferred one by one.

228
00:13:44.592 --> 00:13:48.495
Completion of execution, successful read/write, and data persistence

229
00:13:48.600 --> 00:13:53.070
are different levels of issues and cannot be conflated based solely on one event.

230
00:13:53.523 --> 00:13:55.054
Finally, let's look at extensions.

231
00:13:55.507 --> 00:13:57.915
Taking the existing Phoenix implementation as an example,

232
00:13:58.084 --> 00:14:03.650
if we were to re-integrate it today, the core is not to add another check to the general scheduler,

233
00:14:03.742 --> 00:14:07.490
but to add a new implementation module in the backend directory.

234
00:14:07.943 --> 00:14:10.737
Its backend class inherits from the file-type parent class,

235
00:14:10.932 --> 00:14:13.160
reusing the preparation process for storage files;

236
00:14:13.368 --> 00:14:16.292
its asynchronous handle inherits from the common handle,

237
00:14:16.396 --> 00:14:17.954
taking on specific read/write tasks.

238
00:14:18.407 --> 00:14:22.670
Common driver states and resource releases can be placed in the parent class,

239
00:14:22.801 --> 00:14:24.938
allowing the required implementations to use them directly.

240
00:14:25.391 --> 00:14:27.606
Interfaces can be understood in three groups.

241
00:14:28.059 --> 00:14:31.163
The first group checks the environment, prepares storage and drivers;

242
00:14:31.396 --> 00:14:33.869
the second group registers buffers and streams;

243
00:14:34.064 --> 00:14:36.627
the third group is responsible for asynchronous read/write and release.

244
00:14:37.079 --> 00:14:41.047
The common layer calls these capabilities instead of guessing the backend names.

245
00:14:41.499 --> 00:14:45.312
Phoenix's own differences are handled within its own module:

246
00:14:45.507 --> 00:14:49.513
loading adaptation libraries upon first use, checking asynchronous symbols,

247
00:14:49.618 --> 00:14:53.637
binding function signatures, interpreting error codes and offset parameters.

248
00:14:54.090 --> 00:14:58.057
Adding a new backend does not require modifying all upper layer callers.

249
00:14:58.510 --> 00:15:01.317
Lifecycles do not need to be forced into uniformity.

250
00:15:01.770 --> 00:15:05.377
The current Phoenix encapsulation uses implicit driver initialization,

251
00:15:05.584 --> 00:15:09.887
stream registration does not receive flags and allows for no-op operations.

252
00:15:10.340 --> 00:15:14.642
It does not need to replicate numerical constants from other libraries, but each asynchronous read/write,

253
00:15:14.734 --> 00:15:16.188
still requires passing in the stream.

254
00:15:16.830 --> 00:15:19.482
In contrast to the initial changes made when integrating Phoenix,

255
00:15:19.690 --> 00:15:23.838
at that time, two central files needed to be modified to register configurations and scheduling.

256
00:15:24.292 --> 00:15:29.767
Now, under the premise of existing device platforms, new modules can be discovered through directories,

257
00:15:29.872 --> 00:15:31.572
without requiring changes to these two areas.

258
00:15:32.012 --> 00:15:35.850
The benefit is reduced interdependencies, not the elimination of implementation code.

259
00:15:36.303 --> 00:15:41.520
Native interface adaptation, targeted testing, dependency documentation, and hardware validation

260
00:15:41.651 --> 00:15:42.809
are still necessary.

261
00:15:43.249 --> 00:15:48.904
Especially for new devices that lack support at the lower level, you cannot claim usability just by adding a module.

262
00:15:49.357 --> 00:15:53.285
Software scalability and hardware verification must be discussed separately.

263
00:15:53.725 --> 00:15:58.479
Therefore, the value of GDS is not to turn SSDs into video memory,

264
00:15:58.609 --> 00:16:02.383
but to retain more reusable cache with larger capacity,

265
00:16:02.488 --> 00:16:05.476
and then use appropriate paths to send them back to computation.

266
00:16:05.929 --> 00:16:07.358
Ultimately, whether it is cost-effective,

267
00:16:07.501 --> 00:16:12.306
depends on hit rates, bandwidth, access granularity, and the cost of recalculation.

268
00:16:12.759 --> 00:16:17.036
Phoenix attempts to unify various accelerators at the device read/write layer,

269
00:16:17.205 --> 00:16:22.061
while the caching framework, at a higher level, unifies cache objects and transport processes.

270
00:16:22.514 --> 00:16:24.355
Both must maintain their boundaries,

271
00:16:24.550 --> 00:16:29.084
to have the opportunity for different devices to share the same clear caching logic.

272
00:16:29.537 --> 00:16:33.685
Looking back, the mystery actually comes from several concepts layered together.

273
00:16:34.138 --> 00:16:38.002
Separating data paths, resource ownership, and completion order,

274
00:16:38.249 --> 00:16:41.610
and then using inheritance and polymorphism to manage backend differences.

275
00:16:41.715 --> 00:16:45.283
This cache path from SSD to GPU

276
00:16:45.375 --> 00:16:47.925
becomes understandable and scalable.
