Tensors#
WholeMemory Tensor |
|
Create empty WholeMemory Tensor. Now only support dim = 1 or 2 :param comm: WholeMemoryCommunicator :param memory_type: WholeMemory type, should be continuous, chunked or distributed :param memory_location: WholeMemory location, should be cpu or cuda :param sizes: size of the tensor :param dtype: data type of the tensor :param strides: strides of the tensor :param tensor_entry_partition: rank partition based on entry; tensor_entry_partition[i] determines the entry count of rank i and shoud be a positive integer; the sum of tensor_entry_partition should equal to total entry count; entries will be equally partitioned if None :return: Allocated WholeMemoryTensor. |
|
|
Create WholeMemory Tensor from a list of files. :param comm: WholeMemoryCommunicator :param memory_type: WholeMemory type, should be continuous, chunked or distributed :param memory_location: WholeMemory location, should be cpu or cuda :param filelist: list of files :param dtype: data type of the tensor :param last_dim_size: 0 creates a 1-D array and a positive value creates a matrix with that column count. Required for binary input and inferred from Parquet metadata when omitted. :param last_dim_strides: stride of last_dim, -1 for same as size of last dim. :param tensor_entry_partition: rank partition based on entry; tensor_entry_partition[i] determines the entry count of rank i and shoud be a positive integer; the sum of tensor_entry_partition should equal to total entry count; entries will be equally partitioned if None :param file_format: file format, one of binary, parquet, or auto :param expected_entry_count: optional expected number of rows. An error is raised before allocation when the files contain a different row count. :param expected_shape: optional expected 1-D or 2-D shape. A one-column Parquet file defaults to 1-D unless |
Destroy allocated WholeMemory Tensor :param wm_tensor: WholeMemory Tensor :return: None |