python
class Graph(flow.nn.Graph):
def __init__(self):
super().__init__()
self.m_stage0 = flow.nn.Linear(8, 8, False).to_global(placement=P_0, sbp=B)
self.m_stage1 = flow.nn.Linear(8, 8, False).to_global(placement=P_1, sbp=B)
set_stage(stage_id, placement)
The Stage ID is numbered starting from 0 and increasing by 1.
The Placement is all tensors placement of this module.
self.m_stage0.config.set_stage(stage_id=0, placement=P_0)
self.m_stage1.config.set_stage(stage_id=1, placement=P_1)
self.config.set_gradient_accumulation_steps(4)
def build(self, x):
There will be automatically do tensor.to_global(placement) for all input tensor of this module.
So there is no need to write to_global() in/out of the module forward function.
y = self.m_stage0(x)
z = self.m_stage1(y)
return z
New Features
Graph
- Added new interfaces: `oneflow.env.init_rdma` and `oneflow.env.rdma_is_initialized` to delay turning on the RDMA, thus accelerating the network communications across multiple devices (Note: avoid using fork() after RDMA being turned on, for example, DataLoader’s `num_workers > 1` should be executed before `init rdma`). https://github.com/Oneflow-Inc/oneflow/pull/8415
- Graph provided new algorithm optimization interface: `graph.config.enable_straighten_algorithm` to optimize the execution order in computation graph, which maximizes the overlap between transferring and computation. With this interface, the data transfer speed witnesses a 0.6% rise in data parallelism mode and a 6% rise in model parallelism mode. (https://github.com/Oneflow-Inc/oneflow/pull/8347, https://github.com/Oneflow-Inc/oneflow/pull/8483, https://github.com/Oneflow-Inc/oneflow/pull/8495 )
- Optimized the implementation of clip grad in Graph to support `clip_grad_max_norm > 1.0` and provided configurable `clip_grad_norm_type`, which could only be set to `2` before but now can be set to `+/- inf`, `+/- 1`, `+/- 2`, `+/- 3`, and bigger p-norm values. See the reference from [here]( https://pytorch.org/docs/stable/generated/torch.nn.utils.clip_grad_norm_.html) (https://github.com/Oneflow-Inc/oneflow/pull/7548)
- Global tensor in Graph supported the `tensor.set_item` operation for invariable ops, for example, `mask[:, :len_keep] = 0` (https://github.com/Oneflow-Inc/oneflow/pull/7751)
- Graph exported `build_graph` and `compile_and_init_runtime` interfaces, allowing to compile the `pass` that was previously self-defined by users after building the graph, thus rewriting and optimizing the graph. The two interfaces also supported Graph to restore an external graph (job). (https://github.com/Oneflow-Inc/oneflow/pull/8168)
- Added the `RegisterJobPass` interface to support rewriting the self-defined external job pass graph. (https://github.com/Oneflow-Inc/oneflow/pull/8370)
- `oneflow.boxing.nccl.enable_use_compute_stream(True)` optimized supports for NCCL logical kernel:
- Added noncontiguous ReduceScatter kernel to support the conversion of `P -> S(i), (i > 0)` (https://github.com/Oneflow-Inc/oneflow/pull/8361)
- Supported the conversion of `B -> S` (https://github.com/Oneflow-Inc/oneflow/pull/8355)
- Enabled nccl send/recv primitives to support special SBP conversions (https://github.com/Oneflow-Inc/oneflow/pull/8318)
- Added the efficient fused kernel `oneflow.nn.FusedMLP`, which is controlled by `export ONEFLOW_FUNCTOR_DISABLE_FUSED_MLP = 0` (https://github.com/Oneflow-Inc/oneflow/pull/7391, https://github.com/Oneflow-Inc/oneflow/pull/8165, https://github.com/Oneflow-Inc/oneflow/pull/8217, https://github.com/Oneflow-Inc/oneflow/pull/8413)
Debug
- `Graph.debug` offered the new parameter: `max_stack_depth (default = 2)` to note the maximal stack depth of the Python stack where the op exists in Graph, making it convenient to locate the Python context for each op in Graph. (https://github.com/Oneflow-Inc/oneflow/pull/8028)
- Apart from supporting printing the input/output/variable info of modules in Graph, it also newly supported printing operator info constructed in module forward. (https://github.com/Oneflow-Inc/oneflow/pull/8135)
- Enabled `export ONEFLOW_DEBUG_MODE=true` and `export GLOG_v=3` to print the full memory log, which contains multi-level MemBlock info on each device (Total Memory-> Chunk -> MemBlock), Block that has exclusive memory, Eager Variable and other information. Besides, a lifecycle label was added in Regst to analyze each tensor's memory lifecycle.
- LightPlan provided a more simplified way to display Actor Graph, cutting down the cost of debug based on Plan. When `ONEFLOW_DEBUG_MODE = true `, a series of light plan files corresponding to each rank in Graph will be generated under the `log/local_rank_0/machine/` directory, containing simplified actor sub-graphs in each rank, and the filename is `GraphName_rank_i_light_plan`. (https://github.com/Oneflow-Inc/oneflow/pull/8396)
- The `print graph` method allowed to display the logic graph by Module, making the debugging more efficient in constructing graphs. (https://github.com/Oneflow-Inc/oneflow/pull/8131)
Eager
- Supported passing extra parameters when Optimizer ParamGroup is being built, meeting other special operation demands for LrScheduler. (https://github.com/Oneflow-Inc/oneflow/pull/7753)
- python
param_groups = [{"params": [model.parameters()], "excess_param": ...}]
optim = optim.Adam(param_groups, lr=0.1)
- Added the `oneflow.cuda.current_device` interface to return the device index of the current rank (https://github.com/Oneflow-Inc/oneflow/pull/7856)
- Added the `oneflow.utils.from_torch` interface to convert a PyTorch Tensor into an OneFlow Tensor (https://github.com/Oneflow-Inc/oneflow/pull/7851)
- Added the `oneflow.utils.to_torch` interface to convert an OneFlow Tensor into a PyTorch Tensor (https://github.com/Oneflow-Inc/oneflow/pull/7851)
- Added the `oneflow.cuda.empty_cache` interface to manually release memory https://github.com/Oneflow-Inc/oneflow/pull/8482)
- Added the `oneflow.roc_auc_score` interface on CPU, which is equivalent to `sklearn.metrics.roc_auc_score` (https://github.com/Oneflow-Inc/oneflow/pull/7951)
Tensor
- Provided the `Tensor.contiguous_` interface as the contiguous operation for the inplace version (https://github.com/Oneflow-Inc/oneflow/pull/8275)
- Added the `Tensor.local_to_global` and `Tensor.global_to_global` interfaces to separately implement different default check meta operations (https://github.com/Oneflow-Inc/oneflow/pull/8027)
- Global Tensor's Slice/SliceUpdate supported all nd_sbp inputs, and SliceUpdate fully supported the inplace operation and backpropagation (https://github.com/Oneflow-Inc/oneflow/pull/8313, https://github.com/Oneflow-Inc/oneflow/pull/8337, https://github.com/Oneflow-Inc/oneflow/pull/8344, https://github.com/Oneflow-Inc/oneflow/pull/8416)
Global Boxing
- Eager Global Tensor supported balanced spliter nd sbp eager boxing (https://github.com/Oneflow-Inc/oneflow/pull/7768)
- Supported executing Eager Slice Boxing on random devices, including non-CPU devices and non-CUDA-capable devices (https://github.com/Oneflow-Inc/oneflow/pull/8180)
OneEmbedding
For better recommendations, modern recommendation systems always rely on huge Embedding tables. Besides, frequent iterations of user data require model training to be fast enough.
OneEmbedding is a component designed for large-scale recommendation systems, and it's efficient, extensible, and highly flexible. The following are its advantages:
1. Hierarchical storage and dynamic capacity expansion: users can expand the capacity of the Embedding at much lower cost.
2. Mixed parallelism strategy: it supports easily extending the model to train it on multi-machine multi-GPU.
3. Embedding quantization for better communication: in the parallel scenario, communication data can be quantized to reduce the communication amount, thus accelerating the training.
4. Efficient data pipeline: the model parts that have no data dependency can be executed in advance, thus overlapping with other operations in time.
5. Automatic mixed precision training: data can be computed in FP16 to reduce the occupied memory, thus accelerating the training speed and ensuring high model convergence precision.
6. A collection of efficient CUDA ops for common operations in recommendation systems is available.
7. Flexible model building is supported.
See OneEmbedding API documentation from [here](https://oneflow.readthedocs.io/en/master/one_embedding.html).
PyTorch Compatibility
A collection of new functionalities and interfaces that are compatible with PyTorch 1.10.0 have been added.
Tensor
- Added the `Tensor.pin_memory` functionality, which supports changing the memory to pinned memory when the tensor is being created. (https://github.com/Oneflow-Inc/oneflow/pull/8073)
- Supported passing the `pin_memory` parameter when the tensor is being created. (https://github.com/Oneflow-Inc/oneflow/pull/8176)
- DataLoader supported `pin_memory` (https://github.com/Oneflow-Inc/oneflow/pull/8214)
- Added the`Tensor.is_pinned` attribute (https://github.com/Oneflow-Inc/oneflow/pull/8447)
- Added the `~Tensor` (invert) method to conduct logic NOT operation for each tensor with the dtype of .bool. (https://github.com/Oneflow-Inc/oneflow/pull/7899)
- Added the `Tensor.log2` method to get log<sub>2</sub> for each tensor. (https://github.com/Oneflow-Inc/oneflow/pull/7906)
- Added the `Tensor.new_zeros` method to generate a new tensor that has a shape of 0. (https://github.com/Oneflow-Inc/oneflow/pull/7937)
- Added the `oneflow.as_tensor` interface to convert the input data into a new tensor that shares data. (https://github.com/Oneflow-Inc/oneflow/pull/7855)
- Added the `Tensor.__array__` method. `np.array` supports to input oneflow tensor to construct `np.ndarry` object. (https://github.com/Oneflow-Inc/oneflow/pull/7970)
- Added the `Tensor.new_tensor` method to copy the input data to generate a new tensor. (https://github.com/Oneflow-Inc/oneflow/pull/7973)
- Added the `Tensor.half` method, which is equivalent to `tensor.to (oneflow.float16)` . (https://github.com/Oneflow-Inc/oneflow/pull/7971)
- Added the `Tensor.byte` method to generate a new uint8 tensor, and `tensor.byte()` is equivalent to `tensor.to(oneflow.uint8)`. (https://github.com/Oneflow-Inc/oneflow/pull/8053)
- Added the `Tensor.view_as` and `Tensor.new_empty` methods (https://github.com/Oneflow-Inc/oneflow/pull/8077)
- Added the `Tensor.type` method to implement corresponding cast and adding objects for `oneflow(.cuda).{Byte, Char, Short, Int, Long, Half, Float, Double}Tensor` (https://github.com/Oneflow-Inc/oneflow/pull/8129)
- Added the `Tensor.dot` method to compute the dot product of two 1D tensors, and this method is equivalent to `oneflow.dot`. (https://github.com/Oneflow-Inc/oneflow/pull/8520)
- Added the `oneflow.nn.init.orthogonal_` interface to initialize tensors (https://github.com/Oneflow-Inc/oneflow/pull/8009)
Operators
- Added the `oneflow.nn.Softshrink` op (https://github.com/Oneflow-Inc/oneflow/pull/7826)
- Added the `oneflow.nn.Threshold` op (https://github.com/Oneflow-Inc/oneflow/pull/7875)
- Added the `oneflow.nn.Hardshrink` activation function (https://github.com/Oneflow-Inc/oneflow/pull/7887)
- Added the `oneflow.isnan` and `oneflow.isinf` interfaces to decide the element in tensor is nan or inf (https://github.com/Oneflow-Inc/oneflow/pull/7943)
- The `oneflow.nn.functional.*` interface supported passing the `numpy scalar` parameter (https://github.com/Oneflow-Inc/oneflow/pull/7935)
- Added the `oneflow.nn.functional.cosine_similarity` op to calculate the cosine similarity of two tensors (https://github.com/Oneflow-Inc/oneflow/pull/8119)
- Added the `oneflow.nn.functional.conv_transpose1d`, the `oneflow.nn.functional.conv_transpose2d` op, and the`nn.functional.conv_transpose3d` op (https://github.com/Oneflow-Inc/oneflow/pull/7991)
- Added the `oneflow.unbind` interface to return a tuple of all slices along a given dimension (https://github.com/Oneflow-Inc/oneflow/pull/7730)
- Added the `oneflow.swapdims` interface to specify the swapping of two dimensions, and `oneflow.swapdims` is equivalent to NumPy’s `swapaxes`. (https://github.com/Oneflow-Inc/oneflow/pull/7659)
- Added the `oneflow.addcmul` op to execute an element-wise composite function: `out=input+value×tensor1×tensor2` (https://github.com/Oneflow-Inc/oneflow/pull/7282)
- Added the `oneflow.searchsorted` op (https://github.com/Oneflow-Inc/oneflow/pull/7949)
- Added the `oneflow.mm` op (https://github.com/Oneflow-Inc/oneflow/pull/8440)
- Added the `oneflow.tensordot` interface and offered a collection of cases of equivalent transformation operations (https://github.com/Oneflow-Inc/oneflow/pull/7968)
- Added the `oneflow.repeat_interleave` op to repeat the elements of the tensor, and this op is equivalent to `numpy.repeat` (https://github.com/Oneflow-Inc/oneflow/pull/8324)
- Added the `oneflow.amax` and `Tensor.amax` methods (https://github.com/Oneflow-Inc/oneflow/pull/7996)
- Added the `oneflow.median` and `Tensor.median` methods (https://github.com/Oneflow-Inc/oneflow/pull/8069)
- Added the `oneflow.normal` method and fixed the `Tensor.normal`method (https://github.com/Oneflow-Inc/oneflow/pull/7956)
- Added the `oneflow.amin` and `Tensor.amin` methods (https://github.com/Oneflow-Inc/oneflow/pull/8042)
- Added the `oneflow.mv` op and `Tensor.mv` method (https://github.com/Oneflow-Inc/oneflow/pull/8445)
Random
- Added new interfaces: `oneflow.cuda.manual_seed`, `oneflow.cuda.manual_seed_all`, `oneflow.seed`, `oneflow.manual_seed`, `oneflow.initial_seed`, `oneflow.get_rng_state`, `oneflow.set_rng_state` and improved the configuration of OneFlow random seed initialization. (https://github.com/Oneflow-Inc/oneflow/pull/7957 )
AutoGrad
- Added new interfaces: `oneflow.set_grad_enabled` and `oneflow.enable_grad` to enable or disable automatic gradient update for some of subgraphs. (https://github.com/Oneflow-Inc/oneflow/pull/8016)
- Supported the upstream gradient dtype of the autograd reverse operator is different from that of the input. (https://github.com/Oneflow-Inc/oneflow/pull/8233, https://github.com/Oneflow-Inc/oneflow/pull/8309)
- Supported the backward operator that does not capture any tensor to execute backward computation multiple times. (https://github.com/Oneflow-Inc/oneflow/pull/8031)
CUDA
- Added APIs for `oneflow.cuda.set_device` and `oneflow.cuda.synchronize`. (https://github.com/Oneflow-Inc/oneflow/pull/8322)
RNN
- Refactored the Module of RNN and migrated the implementation of Python layer splicing to C++, which greatly optimized the performance. Added modules related to RNNCell and modules aligned with the `torch.nn.utils.rnn` in functionality:
- Refactored modules: `RNN`, `LSTM`, and `GRU`
- Added modules: `RNNCell `, `LSTMCell`, `GRUCell`, and`oneflow.nn.utils.rnn`
- Supported and fixed RNN unit tests of local and global, and completed documentation.
Device
Supported heterogeneous equipment type: In order to cope with the complexity of different hardware, OneFlow, in line with the dependency inversion principle in software engineering, has introduced a hardware abstraction layer called **Execution Provider (EP)**. The hardware abstraction layer is composed of a series of interfaces, which are abstracted from the capabilities provided by the required hardware devices during the running of the framework. After the hardware abstraction layer is introduced, each modules can directly call the interface provided by the hardware abstraction layer, not the original hardware interface, to use the underlying hardware, so it's unneccessary to concern the specific details of the underlying hardware. When a new hardware device is introduced, because the hardware abstraction interface remains unchanged, all modules can adapt to the new hardware device without any modification. At the same time, when adapting new hardware for the framework, we do not need to pay attention to the specific implementation details of the framework. We only need to implement a series of interfaces according to the agreement of the hardware abstract interface and the actual situation of the hardware device, and then the hardware adaptation can be completed.
Execution Provider has defined a collection of runtime interfaces: device registration interface, device management interface, queue management interface, event management interface, and memory management interface.
Primitive
In addition to the runtime interfaces, the Execution Provider has also defined a set of computing interfaces called Primitive, which are used to describe the commonly-used computation in the deep learning framework, thus simplifying the development of operators in hardware adaptation. Compared with the runtime interfaces provided by the Execution Provider, the interfaces provided by Primitive are more loose and flexible. All interfaces are mutually independent, and each interface represents a specific computing capability provided by a certain hardware device. Similar to runtime interfaces, the abstraction of interfaces provided by Primitive is closer to the device side, and developers can carry out adaptation work without an in-depth understanding of OneFlow's mechanism. Developers need to implement all interfaces provided by Execution Provider when adapting runtime interfaces, but in the process of adapting Primitive, developers can selectively adapt according to the actual situation of the project.
- Added unit test of `ep::primitive` basic function (https://github.com/Oneflow-Inc/oneflow/pull/8099)
- Added `ep::primitive::constant_pad`, optimized performance, removed obsolete pad grad and used pad as the inverse of pad (https://github.com/Oneflow-Inc/oneflow/pull/8152)
- Used unary primitive interface instead of original implementation in Kernel (https://github.com/Oneflow-Inc/oneflow/pull/8270)
- Added environment variable ONEFLOW_EP_CUDA_CUBLAS_WORKSPACE_SIZE_MB to configure cublas workspace size (https://github.com/Oneflow-Inc/oneflow/pull/8478)
- Scalar logical kernel supported primitives (https://github.com/Oneflow-Inc/oneflow/pull/8531)
- Used primitives to implement logical not kernel (https://github.com/Oneflow-Inc/oneflow/pull/8544)
- Migrated all activation kernels to use primitive (https://github.com/Oneflow-Inc/oneflow/pull/8300)
- Bias add kernel supported primitive (https://github.com/Oneflow-Inc/oneflow/pull/8512)
- Decoupled OneDNN from `ep::primitive` CPU device and provided environment variable `ONEFLOW_ENABLE_ONEDNN_OPTS` to enable onednn to accelerate CPU primitive interface (https://github.com/Oneflow-Inc/oneflow/pull/8274)
Debug tools
- Saved the log independently for each rank to `log/local_rank_{i}` when launching multiple processes by launcher. (https://github.com/Oneflow-Inc/oneflow/pull/7825)
- Optimized the display of OF_PROFILER_RANGE_GUARD in nsys. (https://github.com/Oneflow-Inc/oneflow/pull/8121)
OneFlow-Profiler
OneFlow-Profiler is designed to collect various performance-related information during the execution flow of the framework. It can calculate the execution time of the operator or system components, the allocation of memory and DRAM, and can record the input and parameter information corresponding to the operator. This information can be used by developers to analyze which part brings the most overhead and implement some targeted optimizations.
- Added OneFlow-Profiler. (https://github.com/Oneflow-Inc/oneflow/pull/8047)
- Profiled the information of the CUDA operator. (https://github.com/Oneflow-Inc/oneflow/pull/8195)
- Profiled the bandwidth information of the operator. (https://github.com/Oneflow-Inc/oneflow/pull/8254)
- Added interfaces to collect bandwidth information and optimized code implementation. (https://github.com/Oneflow-Inc/oneflow/pull/8332)
- Refined Profiler. (https://github.com/Oneflow-Inc/oneflow/pull/8332)
- Used [Kineto](https://github.com/pytorch/kineto) and [CUPTI](https://docs.nvidia.com/cuda/cupti/index.html) to profile the information of CUDA operator. (https://github.com/Oneflow-Inc/oneflow/pull/8417)
Auto-Test
- When the value check fails, the value of the input tensor and Paramter will be automatically printed, and the pseudo-code segment of the output program will be highlighted for debugging (https://github.com/Oneflow-Inc/oneflow/pull/8383)
AutoProf
AutoProf is a framework designed to test the performance of OneFlow and PyTorch operators. It can automatically test the operator performance and print a comparison table under different CPU threads and GPUs. At present, it has been applied to the development of some existed operators and all new operators. Its effect is shown below:
<img width="1440" alt="image" src="https://user-images.githubusercontent.com/11607199/179392721-ae1d1f69-38cb-4894-92e7-bafdc06fa1c5.png">
- Added auto speed comparison framework of operator AutoProf to automatically run op to test: (https://github.com/Oneflow-Inc/oneflow/pull/8207)
- The speed of OneFlow and PyTorch.
- The speed of CPU/GPU Kernel under different numbers of threads.
- Total end-to-end time with CPU Kernel.
- Optimized the display of AutoProf to save testing time. (https://github.com/Oneflow-Inc/oneflow/pull/8303)
- Supported API tests without actual kernel execution, and the time would be end2end. (https://github.com/Oneflow-Inc/oneflow/pull/8320)
- Supported AutoProf to measure kernel bandwidth. (https://github.com/Oneflow-Inc/oneflow/pull/8367)
IR
- Used Cast to remove pass. (https://github.com/Oneflow-Inc/oneflow/pull/7837 )
- Used MLIR to complete constant folding, combined the composition optimization of Conv and BN. (https://github.com/Oneflow-Inc/oneflow/pull/7799)
- Optimized constant folding in OneFlow C++ API. (https://github.com/Oneflow-Inc/oneflow/pull/8124)
- Provided fault tolerance checking for parsed module. (https://github.com/Oneflow-Inc/oneflow/pull/8299)
- Fixed the BUG of constant folding unit test. (https://github.com/Oneflow-Inc/oneflow/pull/8340)
- Supported IREE. (https://github.com/Oneflow-Inc/oneflow/pull/8249)
- Added `oneflow_iree(python)` to CI. (https://github.com/Oneflow-Inc/oneflow/pull/8431)
- Removed redundant output_lbns in IR. (https://github.com/Oneflow-Inc/oneflow/pull/8409)
- Provided a conversion marker for Variable -> constant. (https://github.com/Oneflow-Inc/oneflow/pull/8412)
- Removed hardcoded properties in IR. (https://github.com/Oneflow-Inc/oneflow/pull/8420)
- Implemented AutoNHWC Pass and provided environment variable `ONEFLOW_MLIR_PREFER_NHWC`. Supported automatic conversion of common network data formats to channels last optimization and had a noticeable acceleration on NVIDIA graphics cards that support FP16. (https://github.com/Oneflow-Inc/oneflow/pull/7890)
Performance
Graph
- Optimized the speed and memory of GPT and BERT under 3-D parallelism:
- Performance optimization: `fused_scale_mask_softmax` operator supported broadcast input. Optimized the kernel implementation and performance of softmax under specific cols (1024). Optimized the incomplete GetSbp list of `fused_scale_mask_softmax` reverse operator. (https://github.com/Oneflow-Inc/oneflow/pull/8321)
- Communication optimization: Optimized the communication cost of SBP cost under `B->S`, `B->B`, `B->P`. (https://github.com/Oneflow-Inc/oneflow/pull/8378)
- Interface optimization: Optimized the inefficient edge connection problem caused by the misalignment of stage id and to_global sequence dependency when using pipeline stage. (https://github.com/Oneflow-Inc/oneflow/pull/8442)
- Communication optimization: `nccl_use_compute_stream` supported more comprehensive sbp conversions like `P -> S(i)`. (https://github.com/Oneflow-Inc/oneflow/pull/8361)
- Communication optimization: Parallel use of RDMA communication. (https://github.com/Oneflow-Inc/oneflow/pull/8415)
- Memory optimization: Eliminated the randomness of the memory multiplexing algorithm, so that the memory multiplexing effect of each rank is consistent when the subgraphs are the same. There will be no bad case. (https://github.com/Oneflow-Inc/oneflow/pull/8441)
- Memory optimization: Removed the extra buffer problem of Stage 0 CPU copy under Pipeline parallelism. (https://github.com/Oneflow-Inc/oneflow/pull/8484)
- Memory optimization: Under Checkpointing and Pipeline, the input identity of the module was de-duplicated to reduce additional Checkpointing tensor, and added the block name prefix of the module to the identity. (https://github.com/Oneflow-Inc/oneflow/pull/8509)
- Combination Optimization: ZeRO-DP supported using with Pipeline parallel and 3-D parallel. (https://github.com/Oneflow-Inc/oneflow/pull/8464)
- Memory optimization: Removed extra identity tensor in ZeRO optimization. (https://github.com/Oneflow-Inc/oneflow/pull/8407)
- Provided new environment variable optimization switches: `ONEFLOW_ENABLE_MULTI_TENSOR_MODEL_UPDATE ` and `ONEFLOW_FUSE_MODEL_UPDATE_CAST `. In the case of AMP, they supported the fusion of the Optimizer model update kernel and the next round of forward cast operators. (https://github.com/Oneflow-Inc/oneflow/pull/8373)
Eager
- Enabled `export ONEFLOW_EAGER_LOCAL_TO_GLOBAL_BALANCED_OVERRIDE =true` to accelerate the execution of Eager Global, which can save the synchronization of meta information on each rank of Global Tensor. (when users are confident that their code execution is symmetrical, SPMD)(https://github.com/Oneflow-Inc/oneflow/pull/7981)
> This environment variable is used to indicate whether the shape of the input data is the same when `local to global` is executed. If it is set to true, there is no need to synchronize the shape of each rank, and the logical shape is calculated locally.
- Used python c api to replace pybind11 to optimize the calling speed of tensor and functional.
- Optimized functional return types to save overhead and avoid reference copies. And solved the bug that the inplace tensor id may be inconsistent. (https://github.com/Oneflow-Inc/oneflow/pull/7985)
- Moved tensor API from pybind11 to c python API. Added tensor hash function. Resolves function naming conflict. (https://github.com/Oneflow-Inc/oneflow/pull/8258, https://github.com/Oneflow-Inc/oneflow/pull/8315, https://github.com/Oneflow-Inc/oneflow/pull/8342, https://github.com/Oneflow-Inc/oneflow/pull/8375)
- Performance optimization: Let vm worker threads concentrate on computing tasks, and decoupled memory tasks from computing tasks. (https://github.com/Oneflow-Inc/oneflow/pull/7976)
- Optimized the speed of operations in DataLoader, including `MakeLocalTensorFromData`, which is 20% faster under swin-T dataloader. (https://github.com/Oneflow-Inc/oneflow/pull/8066)
Operators & Tensor
- Optimized global `sparse_softmax_cross_entropy` kernel. (https://github.com/Oneflow-Inc/oneflow/pull/7298)
- Optimized and sped up CPU `permute` kernel with OneDNN. (https://github.com/Oneflow-Inc/oneflow/pull/7872)
- Optimized and sped up CPU `softmax` kernel with OneDNN. (https://github.com/Oneflow-Inc/oneflow/pull/8071 ， https://github.com/Oneflow-Inc/oneflow/pull/8075)
- Optimized the memory and speed required for the reverse calculation of the pooling kernel. (https://github.com/Oneflow-Inc/oneflow/pull/7980)
- Optimized Slice and Tensor getitem operations based on View to improve the speed of dataloader. (https://github.com/Oneflow-Inc/oneflow/pull/8148, https://github.com/Oneflow-Inc/oneflow/pull/8211, https://github.com/Oneflow-Inc/oneflow/pull/8243)
- Optimized the reverse composition logic of `flip` and `cumsum`, and remove some grad operators. When testing Grad diffs, used random value tests to increase test robustness. (https://github.com/Oneflow-Inc/oneflow/pull/8155)
- Optimized the memory usage of the `NormalizationAddReluGrad` operator and added versions that does not require addend_diff. (https://github.com/Oneflow-Inc/oneflow/pull/8213)
- Optimized and sped up the implementation of `tensor.reshape` and `tensor.reshape_as` from python implementation to c++ implementation. (https://github.com/Oneflow-Inc/oneflow/pull/8304)
- Converted `tensor.view`, `tensor.view_as`, `tensor.permute`, `tensor.transpose`, `tensor.contiguous_` from python implementation to c++ implementation. (https://github.com/Oneflow-Inc/oneflow/pull/8317)
- Greatly optimized the performance of `index_select` and `repeat_interleave` by using gather to replace dim gather. (https://github.com/Oneflow-Inc/oneflow/pull/8360)
- Optimized and removed temporary memory in cumprod cpu grad kernel. (https://github.com/Oneflow-Inc/oneflow/pull/8369)
- The `embedding` operator supported amp, improved the performance under normal path, and fixed the bug that the gather cpu kernel memory out of bounds. (https://github.com/Oneflow-Inc/oneflow/pull/8374)
- Optimized the performance of `Tensor.fill_`. (https://github.com/Oneflow-Inc/oneflow/pull/8283)
- Greatly optimized the performance of the broadcast element-wise binary family operators in reverse calculation. (https://github.com/Oneflow-Inc/oneflow/pull/8339)
- Added fusion operator BinaryCrossEntropyWithLogitsReduceMean. (https://github.com/Oneflow-Inc/oneflow/pull/8476)
- Added high-performance matrix multiplication Fused kernel based on cublasLt. (https://github.com/Oneflow-Inc/oneflow/pull/8462, https://github.com/Oneflow-Inc/oneflow/pull/8222, https://github.com/Oneflow-Inc/oneflow/pull/8063)
Primitive
- Lowered the elementwise.cuh template's requirement for pointer alignment.
Improvements
Graph
- Exported oneflow env to python and used python's objects to manage its lifecycle. (https://github.com/Oneflow-Inc/oneflow/pull/7792)
- Used Python's reference counting to control the life cycle of Graph and constructed strict and rich destruction test cases. (https://github.com/Oneflow-Inc/oneflow/pull/7857)
- Supported recycling independent threads that can no longer be reused when Graph is destructed. (https://github.com/Oneflow-Inc/oneflow/pull/7862)
- Changed the basic configuration of resource from one-time static effect to real-time effect. (https://github.com/Oneflow-Inc/oneflow/pull/8444)
- Consolidated the nccl_comm dynamically created by the Graph NCCL logical kernel into the runtime for initial creation to avoid the deadlock caused by the inconsistency between the creation order of each rank and the eager nccl comm creation order. (https://github.com/Oneflow-Inc/oneflow/pull/8263)
- Refactor optimization: Merged `nn.graph.util.IONode` , `nn.graph.util.IONodeType` into IOArgs. (https://github.com/Oneflow-Inc/oneflow/pull/8272)
- Refactor optimization: Renamed the global singleton Global object to the Singleton object. (https://github.com/Oneflow-Inc/oneflow/pull/8490)
- Refactor optimization: Removed gpu_device_num (https://github.com/Oneflow-Inc/oneflow/pull/8516)
- Refactor optimization: Removed outdated AvailableMemDesc concepts. (https://github.com/Oneflow-Inc/oneflow/pull/8145)
- Refactor optimization: Removed outdated Model IO Kernel logic. (https://github.com/Oneflow-Inc/oneflow/pull/8151)
- Refactor optimization: Replaced GpuDeviceNum with the actual number of devices to avoid coupling with specific device types. (https://github.com/Oneflow-Inc/oneflow/pull/8166)
Eager
- C++ is available now. You can manually trigger allocator gc on each stream (applicable in ZeRO)（https://github.com/Oneflow-Inc/oneflow/pull/8452）
- The execution of Eager VirtualMachine instruction is based on the execution of EP. (https://github.com/Oneflow-Inc/oneflow/pull/7923)
- Optimized and removed all redundant interfaces of ` Get(Ptr)OrThrow`. (https://github.com/Oneflow-Inc/oneflow/pull/7812)
- Added the validity check of `flow.save(global_dst_rank)`. (https://github.com/Oneflow-Inc/oneflow/pull/7964)
- Supported the backward function node to run multiple times if it does not capture any tensor. (https://github.com/Oneflow-Inc/oneflow/pull/8031)
- Added the `ThreadLocalCached` decorator to clear the cache in time to alleviate increasing memory. (https://github.com/Oneflow-Inc/oneflow/pull/7858)
- Added std for C++14::inclusive_scan/std::exclusive_scan implementations. (https://github.com/Oneflow-Inc/oneflow/pull/8128)
- Packaged the parameters required by the eager opkernel and pass them in each thread to solve some thread-unsafe problems. (https://github.com/Oneflow-Inc/oneflow/pull/7617)
- Eager Stream supports kernel computation on pinned memory. (https://github.com/Oneflow-Inc/oneflow/pull/8486)
- Introduced a tool class for dim range check to replace simplified Functor's various checking logic for dimensions. (https://github.com/Oneflow-Inc/oneflow/pull/8382)
- Refactoring and optimization: removed the Blob object in EagerBlobObject, which leads to redundant TensorView instructions. At the same time, in order to support ShapeView efficiently, the elem_cnt attribute has also been removed. (https://github.com/Oneflow-Inc/oneflow/pull/7895)
- Refactoring and optimization: extracted the algorithm used by BinAllocator to share dynamic memory pools
- Refactoring and optimization: `VectorAt` and `MapAt` functions uniformly use reference to pass parameters to solve the mixed use of reference interface and pointer interface. (https://github.com/Oneflow-Inc/oneflow/pull/8191)
- Refactoring and optimization: removed the cfg application on C++. (https://github.com/Oneflow-Inc/oneflow/pull/8158)
- Refactoring and optimization: removed the outdated code related to RemoteBlob in Single-Client. (https://github.com/Oneflow-Inc/oneflow/pull/8228)
- Refactoring and optimization: merged duplicate logic in eager boxing ccl and nccl boxing expr. (https://github.com/Oneflow-Inc/oneflow/pull/7930)
- Refactoring and optimization: removed cfg on Python and reduced the number of symbols to optimize the link speed of compilation.
- Refactoring and optimization: merged `symbol::IdCache` and `symbol::Storage`. (https://github.com/Oneflow-Inc/oneflow/pull/8331)
- Refactoring and optimization: introduced `llvm::SmallVetor` and used `oneflow::small_vector` instead of `fixed_vector`. Besides, we have optimized the implementation and usage of Shape and Stride. (https://github.com/Oneflow-Inc/oneflow/pull/8365 , https://github.com/Oneflow-Inc/oneflow/pull/8402)
- Refactoring and optimization: refactored ShapeView and Shape to eliminated duplication and inconsistencies. (https://github.com/Oneflow-Inc/oneflow/pull/8422)
- Refactoring and optimization: eager VirtualMachine has decoupled InstructionType's dependency on StreamType. (https://github.com/Oneflow-Inc/oneflow/pull/7607)
- Refactoring and optimization: removed the InstructionMsg class and merged all its functions and fields into the Instruction class. (https://github.com/Oneflow-Inc/oneflow/pull/7623)
Operators & Tensor
- Stride support:
- Tensor, UserOp and UserKernel in `user_op::` all supported stride attribute. (https://github.com/Oneflow-Inc/oneflow/pull/7829)
- `cast` supports stride. (https://github.com/Oneflow-Inc/oneflow/pull/8292)
- View support and optimization:
- Added a new input tensor to decide whether to support non-contiguous when making op definitions. Besides, we now support `transpose`, `permute`, `narrow`, `expand`, `expand_as`, `split`, `chunk`, `unfold_tensor`, `movedim`, `as_strided`, `select`, `swapaxes`, `T`, `t`, `hsplit`, `vsplit`, `tensor_split` none-contiguous view ops.(https://github.com/Oneflow-Inc/oneflow/pull/7813)
- Tensor slice used view operations by default.（https://github.com/Oneflow-Inc/oneflow/pull/8302）
- Automatically generated version status (Feature Stage) for OneFlow's API. (https://github.com/Oneflow-Inc/oneflow/pull/7945)
- Optimized CUDA memset to `cudaMemsetAsync`（https://github.com/Oneflow-Inc/oneflow/pull/7763）
- `LeakyReLU` supported inplace optimization. (https://github.com/Oneflow-Inc/oneflow/pull/8060)
- Added the following parameters to `nn.Embedding` interface: `padding_idx`, `max_norm`, `norm_type`, `scale_grad_by_freq`. (https://github.com/Oneflow-Inc/oneflow/pull/8110)
- Aligned PyTorch's `max_pool_1d`, `max_pool_2d`, `max_pool_3d`, `avg_pool_1d`, `avg_pool_2d`, `avg_pool_3d`, and distinguish old pooling kernel aligned with TensorFlow. (https://github.com/Oneflow-Inc/oneflow/pull/8111)
- VectorAt supported passing in non-const references: `JUST(VectorAt(vec, 1)) = 5;`. (https://github.com/Oneflow-Inc/oneflow/pull/8013)
- Reduced the uncommon kernel template specializations of layer norm. (https://github.com/Oneflow-Inc/oneflow/pull/8209)
- Modified the logic of `Tensor.numpy` to avoid extra memory growth when saving the model. (https://github.com/Oneflow-Inc/oneflow/pull/8449)
- Tensor str supported printing nd sbp. (https://github.com/Oneflow-Inc/oneflow/pull/8458)
- Slice supported SBP infer (S->P), and the semi-automatically deduced sbp was able to selecte the same sbp as expected in the reducible nd_sbp. (https://github.com/Oneflow-Inc/oneflow/pull/8536)
- When printing non-CPU and non-CUDA tensor, you must copy to cpu first and then print. (https://github.com/Oneflow-Inc/oneflow/pull/8548)
- Refactoring and optimization: decoupling user kernel and device tag. (https://github.com/Oneflow-Inc/oneflow/pull/8529)
- Refactoring and optimization: a series of kernels (`squeeze`, `reshape_like`, `flatten`, `expand_dims`, `reshape`, `amp_white_identity`, `identity`, `identity_buffer`, `parallel_cast`, `hierarchical_parallel_cast`, ` hierarchical_parallel_cast_like`) were refactored to CopyDataContentKernel https://github.com/Oneflow-Inc/oneflow/pull/8537
- Refactoring and optimization: removed obsolete `constant_pad1d` , `constant_pad2d` , `constant_pad3d` kernel. (https://github.com/Oneflow-Inc/oneflow/pull/8113)
- Refactoring and optimization: removed obsolete old lazy `upsample` kernel implementation.(https://github.com/Oneflow-Inc/oneflow/pull/8188)
- Refactoring and optimization: removed obsolete message in shape proto and used sequential to represent stride. (https://github.com/Oneflow-Inc/oneflow/pull/8220)
- Refactoring and optimization: removed obsolete multiply kernel, whick was included in `broadcast_mul`. (https://github.com/Oneflow-Inc/oneflow/pull/8359)
- Refactoring and optimization: Renamed the shape in UserOp/Kernel to shape_view interface. (https://github.com/Oneflow-Inc/oneflow/pull/8433)
- Refactoring and optimization: removed oneflow gemm. (https://github.com/Oneflow-Inc/oneflow/pull/8499)
- Optimized the Maybe return type of such interfaces as Scalar.As(). (https://github.com/Oneflow-Inc/oneflow/pull/8348)
Device
- Code refactoring `ep::CpuDevice` (https://github.com/Oneflow-Inc/oneflow/pull/7911)
- Code refactoring: removed hard-coded special decision for device type like "cpu", "cuda" from system code. (https://github.com/Oneflow-Inc/oneflow/pull/8201)
- Removed all dnn-related interfaces from the old version of KernelUtil (Primitive will be used to replace those interfaces). (https://github.com/Oneflow-Inc/oneflow/pull/8141)
- Removed all interfaces related to mathematical calculation in the old version of KernelUtil (Primitive will be used to replace those interfaces). (https://github.com/Oneflow-Inc/oneflow/pull/8157)
- Removed incomplete special decision for 'cuda 'device type in scope util. (https://github.com/Oneflow-Inc/oneflow/pull/8173)
- Achieved delayed capture of CUDA Graph(https://github.com/Oneflow-Inc/oneflow/pull/8474)
- Code refactoring: removed cuda_event. (https://github.com/Oneflow-Inc/oneflow/pull/8493)
- Code refactoring: removed useless WITH_CUDA macro. (https://github.com/Oneflow-Inc/oneflow/pull/8562)
Tests
Eager Global Module Tests：
In 0.8.0, we have completed the ability of all kernels to deal with global tensor in distributed situation, and fixed many known bugs related to sbp. The global tensor worked efficiently and correctly at the kernel level. No matter how the distributed topology structure changes, the same algorithm logic can efficiently get mathematically consistent results, which greatly reduced the trouble of verifying correctness in the complex, diverse and asymmetric distributed parallel training process.
| module/functional op | PR |
| -------------------------------- | ------------------------------------------------------------ |
| abs | [Oneflow-Inc/oneflow7540](https://github.com/Oneflow-Inc/oneflow/pull/7540) |
| 0_dim_tensor | [Oneflow-Inc/oneflow7540](https://github.com/Oneflow-Inc/oneflow/pull/7540) |
| activation | [Oneflow-Inc/oneflow7540](https://github.com/Oneflow-Inc/oneflow/pull/7540) |
| adaptive_pool | [Oneflow-Inc/oneflow7563](https://github.com/Oneflow-Inc/oneflow/pull/7563) |
| addmm | [Oneflow-Inc/oneflow7565](https://github.com/Oneflow-Inc/oneflow/pull/7565) |
| add | [Oneflow-Inc/oneflow7204](https://github.com/Oneflow-Inc/oneflow/pull/7204) |
| affine_grid | [Oneflow-Inc/oneflow7578](https://github.com/Oneflow-Inc/oneflow/pull/7578) |
| arange | [Oneflow-Inc/oneflow7576](https://github.com/Oneflow-Inc/oneflow/pull/7576) |
| argmax | [Oneflow-Inc/oneflow7579](https://github.com/Oneflow-Inc/oneflow/pull/7579) |
| argmin | [Oneflow-Inc/oneflow7581](https://github.com/Oneflow-Inc/oneflow/pull/7581) |
| argsort | [Oneflow-Inc/oneflow7582](https://github.com/Oneflow-Inc/oneflow/pull/7582) |
| argwhere | [Oneflow-Inc/oneflow7584](https://github.com/Oneflow-Inc/oneflow/pull/7584) |
| avgpool | [Oneflow-Inc/oneflow7585](https://github.com/Oneflow-Inc/oneflow/pull/7585) |
| batch_gather | [Oneflow-Inc/oneflow7590](https://github.com/Oneflow-Inc/oneflow/pull/7590) |
| bernoulli | [Oneflow-Inc/oneflow7732](https://github.com/Oneflow-Inc/oneflow/pull/7732) |
| bmm | [Oneflow-Inc/oneflow7741](https://github.com/Oneflow-Inc/oneflow/pull/7741) |
| broadcast_like | [Oneflow-Inc/oneflow7742](https://github.com/Oneflow-Inc/oneflow/pull/7742) |
| cast | [Oneflow-Inc/oneflow7773](https://github.com/Oneflow-Inc/oneflow/pull/7773) |
| ceil | [Oneflow-Inc/oneflow7744](https://github.com/Oneflow-Inc/oneflow/pull/7744) |
| chunk | [Oneflow-Inc/oneflow7750](https://github.com/Oneflow-Inc/oneflow/pull/7750) |
| clamp | [Oneflow-Inc/oneflow7752](https://github.com/Oneflow-Inc/oneflow/pull/7752) |
| clip_grad | [Oneflow-Inc/oneflow7757](https://github.com/Oneflow-Inc/oneflow/pull/7757) |
| concat | [Oneflow-Inc/oneflow7204](https://github.com/Oneflow-Inc/oneflow/pull/7204) |
| conv1d | [Oneflow-Inc/oneflow7769](https://github.com/Oneflow-Inc/oneflow/pull/7769) |
| conv2d | [Oneflow-Inc/oneflow7771](https://github.com/Oneflow-Inc/oneflow/pull/7771) |
| conv3d | [Oneflow-Inc/oneflow7771](https://github.com/Oneflow-Inc/oneflow/pull/7771) |
| cumsum | [Oneflow-Inc/oneflow7772](https://github.com/Oneflow-Inc/oneflow/pull/7772) |
| deconv2d | [Oneflow-Inc/oneflow7772](https://github.com/Oneflow-Inc/oneflow/pull/7772) |
| diagonal | [Oneflow-Inc/oneflow7772](https://github.com/Oneflow-Inc/oneflow/pull/7772) |
| diag | [Oneflow-Inc/oneflow7421](https://github.com/Oneflow-Inc/oneflow/pull/7421) |
| div | [Oneflow-Inc/oneflow7421](https://github.com/Oneflow-Inc/oneflow/pull/7421) |
| dot | [Oneflow-Inc/oneflow7421](https://github.com/Oneflow-Inc/oneflow/pull/7421) |
| dropout | [Oneflow-Inc/oneflow7772](https://github.com/Oneflow-Inc/oneflow/pull/7772) |
| empty | [Oneflow-Inc/oneflow7508](https://github.com/Oneflow-Inc/oneflow/pull/7508) |
| eq | [Oneflow-Inc/oneflow7421](https://github.com/Oneflow-Inc/oneflow/pull/7421) |
| erfc | [Oneflow-Inc/oneflow7421](https://github.com/Oneflow-Inc/oneflow/pull/7421) |
| erf | [Oneflow-Inc/oneflow7421](https://github.com/Oneflow-Inc/oneflow/pull/7421) |
| expand | [Oneflow-Inc/oneflow7772](https://github.com/Oneflow-Inc/oneflow/pull/7772) |
| expm1 | [Oneflow-Inc/oneflow7421](https://github.com/Oneflow-Inc/oneflow/pull/7421) |
| eye | [Oneflow-Inc/oneflow7421](https://github.com/Oneflow-Inc/oneflow/pull/7421) |
| flatten | [Oneflow-Inc/oneflow7421](https://github.com/Oneflow-Inc/oneflow/pull/7421) |
| flip | [Oneflow-Inc/oneflow7496](https://github.com/Oneflow-Inc/oneflow/pull/7496) |
| floor | [Oneflow-Inc/oneflow7421](https://github.com/Oneflow-Inc/oneflow/pull/7421) |
| fmod | [Oneflow-Inc/oneflow7421](https://github.com/Oneflow-Inc/oneflow/pull/7421) |
| fold | [Oneflow-Inc/oneflow7772](https://github.com/Oneflow-Inc/oneflow/pull/7772) |
| greater_equal | [Oneflow-Inc/oneflow7421](https://github.com/Oneflow-Inc/oneflow/pull/7421) |
| greater | [Oneflow-Inc/oneflow7366](https://github.com/Oneflow-Inc/oneflow/pull/7366) |
| fused_bias_add_dropout | [Oneflow-Inc/oneflow7867](https://github.com/Oneflow-Inc/oneflow/pull/7867) |
| fused_bias_add_gelu | [Oneflow-Inc/oneflow7867](https://github.com/Oneflow-Inc/oneflow/pull/7867) |
| fused_scale_mask_softmax_dropout | [Oneflow-Inc/oneflow7867](https://github.com/Oneflow-Inc/oneflow/pull/7867) |
| fused_scale_mask_softmax | [Oneflow-Inc/oneflow7867](https://github.com/Oneflow-Inc/oneflow/pull/7867) |
| fused_scale_tril | [Oneflow-Inc/oneflow7867](https://github.com/Oneflow-Inc/oneflow/pull/7867) |
| fused_self_attention | [Oneflow-Inc/oneflow7867](https://github.com/Oneflow-Inc/oneflow/pull/7867) |
| fused_tril_softmax_mask_scale | [Oneflow-Inc/oneflow7867](https://github.com/Oneflow-Inc/oneflow/pull/7867) |
| gather_nd | [Oneflow-Inc/oneflow7880](https://github.com/Oneflow-Inc/oneflow/pull/7880) |
| gather | [Oneflow-Inc/oneflow7880](https://github.com/Oneflow-Inc/oneflow/pull/7880) |
| glu | [Oneflow-Inc/oneflow7880](https://github.com/Oneflow-Inc/oneflow/pull/7880) |
| grid_sample | [Oneflow-Inc/oneflow7881](https://github.com/Oneflow-Inc/oneflow/pull/7881) |
| groupnorm | [Oneflow-Inc/oneflow7885](https://github.com/Oneflow-Inc/oneflow/pull/7885) |
| masked_fill | [Oneflow-Inc/oneflow7457](https://github.com/Oneflow-Inc/oneflow/pull/7457) |
| masked_select | [Oneflow-Inc/oneflow7492](https://github.com/Oneflow-Inc/oneflow/pull/7492) |
| math_ops | [Oneflow-Inc/oneflow7461](https://github.com/Oneflow-Inc/oneflow/pull/7461) |
| matmul | [Oneflow-Inc/oneflow7465](https://github.com/Oneflow-Inc/oneflow/pull/7465) |
| maxpool | [Oneflow-Inc/oneflow7683](https://github.com/Oneflow-Inc/oneflow/pull/7683) |
| max | [Oneflow-Inc/oneflow7450](https://github.com/Oneflow-Inc/oneflow/pull/7450) |
| mean | [Oneflow-Inc/oneflow7650](https://github.com/Oneflow-Inc/oneflow/pull/7650) |
| meshgrid | [Oneflow-Inc/oneflow7533](https://github.com/Oneflow-Inc/oneflow/pull/7533) |
| min_max_observer | [Oneflow-Inc/oneflow7725](https://github.com/Oneflow-Inc/oneflow/pull/7725) |
| min | [Oneflow-Inc/oneflow7450](https://github.com/Oneflow-Inc/oneflow/pull/7450) |
| movedim | [Oneflow-Inc/oneflow7679](https://github.com/Oneflow-Inc/oneflow/pull/7679) |
| moving_average_min_max_observer | [Oneflow-Inc/oneflow7726](https://github.com/Oneflow-Inc/oneflow/pull/7726) |
| mul | [Oneflow-Inc/oneflow7717](https://github.com/Oneflow-Inc/oneflow/pull/7717) |
| narrow | [Oneflow-Inc/oneflow7647](https://github.com/Oneflow-Inc/oneflow/pull/7647) |
| negative | [Oneflow-Inc/oneflow7644](https://github.com/Oneflow-Inc/oneflow/pull/7644) |
| ne | [Oneflow-Inc/oneflow7642](https://github.com/Oneflow-Inc/oneflow/pull/7642) |
| nms | [Oneflow-Inc/oneflow7536](https://github.com/Oneflow-Inc/oneflow/pull/7536) |
| nonzero | [Oneflow-Inc/oneflow7645](https://github.com/Oneflow-Inc/oneflow/pull/7645) |
| normalize | [Oneflow-Inc/oneflow7635](https://github.com/Oneflow-Inc/oneflow/pull/7635) |
| ones_like | [Oneflow-Inc/oneflow7635](https://github.com/Oneflow-Inc/oneflow/pull/7635) |
| parital_fc | [Oneflow-Inc/oneflow7534](https://github.com/Oneflow-Inc/oneflow/pull/7534) |
| permute | [Oneflow-Inc/oneflow7635](https://github.com/Oneflow-Inc/oneflow/pull/7635) |
| prod | [Oneflow-Inc/oneflow7635](https://github.com/Oneflow-Inc/oneflow/pull/7635) |
| randint | [Oneflow-Inc/oneflow7508](https://github.com/Oneflow-Inc/oneflow/pull/7508) |
| rand | [Oneflow-Inc/oneflow7508](https://github.com/Oneflow-Inc/oneflow/pull/7508) |
| reshape | [Oneflow-Inc/oneflow7472](https://github.com/Oneflow-Inc/oneflow/pull/7472) |
| roi_align | [Oneflow-Inc/oneflow7794](https://github.com/Oneflow-Inc/oneflow/pull/7794) |
| scatter_nd | [Oneflow-Inc/oneflow7807](https://github.com/Oneflow-Inc/oneflow/pull/7807) |
| scatter_ops | [Oneflow-Inc/oneflow7807](https://github.com/Oneflow-Inc/oneflow/pull/7807) |
| sign | [Oneflow-Inc/oneflow7818](https://github.com/Oneflow-Inc/oneflow/pull/7818) |
| slice | [Oneflow-Inc/oneflow7818](https://github.com/Oneflow-Inc/oneflow/pull/7818) |
| softplus | [Oneflow-Inc/oneflow7818](https://github.com/Oneflow-Inc/oneflow/pull/7818) |
| sparse_softmax_cross_entr | [Oneflow-Inc/oneflow7298](https://github.com/Oneflow-Inc/oneflow/pull/7298) |
| split | [Oneflow-Inc/oneflow7277](https://github.com/Oneflow-Inc/oneflow/pull/7277) |
| sqrt_square_sum | [Oneflow-Inc/oneflow7277](https://github.com/Oneflow-Inc/oneflow/pull/7277) |
| squeeze | [Oneflow-Inc/oneflow7289](https://github.com/Oneflow-Inc/oneflow/pull/7289) |
| stack | [Oneflow-Inc/oneflow7289](https://github.com/Oneflow-Inc/oneflow/pull/7289) |
| stateful_kernel_with_cache | [Oneflow-Inc/oneflow7289](https://github.com/Oneflow-Inc/oneflow/pull/7289) |
| std | [Oneflow-Inc/oneflow7303](https://github.com/Oneflow-Inc/oneflow/pull/7303) |
| sub | [Oneflow-Inc/oneflow7303](https://github.com/Oneflow-Inc/oneflow/pull/7303) |
| sum | [Oneflow-Inc/oneflow7303](https://github.com/Oneflow-Inc/oneflow/pull/7303) |
| tensor_ops | [Oneflow-Inc/oneflow7307](https://github.com/Oneflow-Inc/oneflow/pull/7307) |
| tensor_scatter_nd_update | [Oneflow-Inc/oneflow7308](https://github.com/Oneflow-Inc/oneflow/pull/7308) |
| tile | [Oneflow-Inc/oneflow7322](https://github.com/Oneflow-Inc/oneflow/pull/7322) |
| transpose | [Oneflow-Inc/oneflow7332](https://github.com/Oneflow-Inc/oneflow/pull/7332) |
| tril | [Oneflow-Inc/oneflow7322](https://github.com/Oneflow-Inc/oneflow/pull/7322) |
| TripletMarginLoss | [Oneflow-Inc/oneflow7332](https://github.com/Oneflow-Inc/oneflow/pull/7332) |
| triu | [Oneflow-Inc/oneflow7882](https://github.com/Oneflow-Inc/oneflow/pull/7882) |
| unfold | [Oneflow-Inc/oneflow7883](https://github.com/Oneflow-Inc/oneflow/pull/7883) |
| unfold_tensor | [Oneflow-Inc/oneflow7883](https://github.com/Oneflow-Inc/oneflow/pull/7883) |
| unsqueeze | [Oneflow-Inc/oneflow7882](https://github.com/Oneflow-Inc/oneflow/pull/7882) |
| upsample | [Oneflow-Inc/oneflow7884](https://github.com/Oneflow-Inc/oneflow/pull/7884) |
| var | [Oneflow-Inc/oneflow7891](https://github.com/Oneflow-Inc/oneflow/pull/7891) |
| view | [Oneflow-Inc/oneflow7886](https://github.com/Oneflow-Inc/oneflow/pull/7886) |
| weight_norm | [Oneflow-Inc/oneflow7886](https://github.com/Oneflow-Inc/oneflow/pull/7886) |
| where | [Oneflow-Inc/oneflow7886](https://github.com/Oneflow-Inc/oneflow/pull/7886) |
| zeropad2d | [Oneflow-Inc/oneflow7886](https://github.com/Oneflow-Inc/oneflow/pull/7886) |
EP::Primitive
Completed some unit tests of Primitive `log_softmax`, `softmax`, `copynd`, `Memset`, `Memcpy`, `matmul`, `add`, binary, unary, `matmul`, `batch_matmul`, fill etc. (https://github.com/Oneflow-Inc/oneflow/pull/8132, https://github.com/Oneflow-Inc/oneflow/pull/8139, https://github.com/Oneflow-Inc/oneflow/pull/8137, https://github.com/Oneflow-Inc/oneflow/pull/8109, https://github.com/Oneflow-Inc/oneflow/pull/8143, https://github.com/Oneflow-Inc/oneflow/pull/8108, https://github.com/Oneflow-Inc/oneflow/pull/8154, https://github.com/Oneflow-Inc/oneflow/pull/8154, https://github.com/Oneflow-Inc/oneflow/pull/8118 ， https://github.com/Oneflow-Inc/oneflow/pull/8291)
Exception
Improve exception error handling
- Added `reshape` exception handling. (https://github.com/Oneflow-Inc/oneflow/pull/7847)
- Improved the error message of module when the input information does not match. (https://github.com/Oneflow-Inc/oneflow/pull/7918)
- Added the `MAYBE_NEED_ERROR_MSG_CHECK` environment variable to check whether the CHECK function of Maybe contains oneflow:: Error message. It is used to prompt developers to add error prompt message. (https://github.com/Oneflow-Inc/oneflow/pull/7955)
- Improved the exception error message of `gather` op.(https://github.com/Oneflow-Inc/oneflow/pull/7979)
- Improved `LayerNorm` error message. (https://github.com/Oneflow-Inc/oneflow/pull/8090)
- Optimized the error message when Eager and Graph encounter multiple inconsistent input placement in op. (https://github.com/Oneflow-Inc/oneflow/pull/8054)
- Improved the error message checking in activation-related kernel processing logic.(https://github.com/Oneflow-Inc/oneflow/pull/8080)
- Improved the error message in `tensor.to_global` and `tensor.to_local`. (https://github.com/Oneflow-Inc/oneflow/pull/8067)
- Improved the exception error message in the `dot` kernel. (https://github.com/Oneflow-Inc/oneflow/pull/8051)
- Rewrited the exception check in `batch_matmul` kernel. (https://github.com/Oneflow-Inc/oneflow/pull/8186)
- Fixed the problem of exception error checking when Python parses arg. (https://github.com/Oneflow-Inc/oneflow/pull/8205)
- Improved the exception error checking logic of all array functor. (https://github.com/Oneflow-Inc/oneflow/pull/8116)
- Improved the exception error checking logic of all binary functor. (https://github.com/Oneflow-Inc/oneflow/pull/8161)
- Improved the exception error reporting logic in nn grad functor. (https://github.com/Oneflow-Inc/oneflow/pull/8210)
- Added error message when Graph.build is not reloaded. (https://github.com/Oneflow-Inc/oneflow/pull/8250)
- Added TypeError type and device-related error message. (https://github.com/Oneflow-Inc/oneflow/pull/8057)
- Improved the error message of Eager SliceBoxing. (https://github.com/Oneflow-Inc/oneflow/pull/8232)
- Improved the error message of broadcast op. (Improve the error message of broadcast op)
- Improved the error message of Eager Boxing when it is at runtime. (https://github.com/Oneflow-Inc/oneflow/pull/7926)
- Improved the error message of Tensor index. (https://github.com/Oneflow-Inc/oneflow/pull/8234)
- Improved the error message in nn.functor. (https://github.com/Oneflow-Inc/oneflow/pull/7910)
- Added check for Physical Shape when Graph compiles exec_graph. (https://github.com/Oneflow-Inc/oneflow/pull/8002)
- Added default error message for CUDA check. (https://github.com/Oneflow-Inc/oneflow/pull/8427)
- Added similar error checking information to add n calculation. (https://github.com/Oneflow-Inc/oneflow/pull/8495)
- Improved the error message of arg sort. (https://github.com/Oneflow-Inc/oneflow/pull/8513)
- Improved the error message of bias add. (https://github.com/Oneflow-Inc/oneflow/pull/8524)
- Improved the error message in autograd function. (https://github.com/Oneflow-Inc/oneflow/pull/8496)
- Improved the error message of batch gather. (https://github.com/Oneflow-Inc/oneflow/pull/8533)
- Improved the error message prompt of defense code in autograd. (https://github.com/Oneflow-Inc/oneflow/pull/8525 ， https://github.com/Oneflow-Inc/oneflow/pull/8541)
Build
- Supported CUDA 11.5, 11.6. (ttps://github.com/Oneflow-Inc/oneflow/pull/7852 , https://github.com/Oneflow-Inc/oneflow/pull/8423)
- Fixed the version of click at 8.0.0. (https://github.com/Oneflow-Inc/oneflow/pull/7967)
- Updated nccl version to 2.12.10. (https://github.com/Oneflow-Inc/oneflow/pull/7822)
- Default alignment pytorch version 1.10.0. (https://github.com/Oneflow-Inc/oneflow/pull/7019)
- Updated tvm oneflow frontend dependencies. (https://github.com/Oneflow-Inc/oneflow/pull/8048)
- Updated the version of LLVM/MLIR to support IREE. (https://github.com/Oneflow-Inc/oneflow/pull/8068 ， https://github.com/Oneflow-Inc/oneflow/pull/8461)
- Fixed the version of protobuf between 3.9.2 to 4.0. (https://github.com/Oneflow-Inc/oneflow/pull/8198)
- Removed the cfg tool in cmake. (https://github.com/Oneflow-Inc/oneflow/pull/8218)
- The environment variable of CMAKE INTERPROCEDURAL OPTIMIZATION was enabled by default. (https://github.com/Oneflow-Inc/oneflow/pull/8237)
- Removed the XRT part in the OneFlow source code, and the OneFlow-XRT will be used as a third-party plugin for oneflow. (https://github.com/Oneflow-Inc/oneflow/pull/8273 ，https://github.com/Oneflow-Inc/oneflow/pull/8288)
- read more: https://github.com/Oneflow-Inc/oneflow-xrt
- Changed Liboneflow to dynamic library. (https://github.com/Oneflow-Inc/oneflow/pull/8312)
- Updated the version of clang-tidy to 14.0.4. Supports the following syntax now: NOLINT, NOLINTNEXTLINE, NOLINTBEGIN & NOLINTEND. (https://github.com/Oneflow-Inc/oneflow/pull/8306)
- Removed `EXTERNAL_INCLUDE_DIRS` , only builds with target. (https://github.com/Oneflow-Inc/oneflow/pull/8421)
- Removed obsolete linkages in cmake. (https://github.com/Oneflow-Inc/oneflow/pull/8426)
CI
Improve the running speed and stability of CI
- Supported CI to automatically upload built docs.(https://github.com/Oneflow-Inc/oneflow/pull/7894 https://github.com/Oneflow-Inc/oneflow/pull/7917)
- Added CI test for IREE. (https://github.com/Oneflow-Inc/oneflow/pull/8419)
- Printed the pip package in the container used to test in order to query version information easily. (https://github.com/Oneflow-Inc/oneflow/pull/7952)
- Optimized the old version of SpeedTest. (https://github.com/Oneflow-Inc/oneflow/pull/7871 https://github.com/Oneflow-Inc/oneflow/pull/7990 https://github.com/Oneflow-Inc/oneflow/pull/8035)
- Optimized the memory used by AutoTest. (https://github.com/Oneflow-Inc/oneflow/pull/7988)
- Adjusted the threshold of benchmark. (https://github.com/Oneflow-Inc/oneflow/pull/8043)
- Adjusted the timeout threshold. (https://github.com/Oneflow-Inc/oneflow/pull/8103)
- Optimized the warning output related to `__del__` in CI. (https://github.com/Oneflow-Inc/oneflow/pull/8049)
- Optimized the interval of gc to improve the test speed. (https://github.com/Oneflow-Inc/oneflow/pull/8138)
- Optimized the use of super Tensor in CI unit test to avoid gc too slow and slow down the running speed of CI. (https://github.com/Oneflow-Inc/oneflow/pull/8177)
- Optimized the number of CI build to improve the speed of build. (https://github.com/Oneflow-Inc/oneflow/pull/8229)
- Optimized CI workflow, stops all workflows when a job fails. (https://github.com/Oneflow-Inc/oneflow/pull/8255)
- Increased maximum parallelism 5 -> 10. (https://github.com/Oneflow-Inc/oneflow/pull/8259)
- Strict CI timeout-minutes. (https://github.com/Oneflow-Inc/oneflow/pull/8266)
- Supported optional multi-machine testing via the `need-test-distributed` tag. (https://github.com/Oneflow-Inc/oneflow/pull/8372)
- Tried to use a distributed test cache when testing on multiple machines. (https://github.com/Oneflow-Inc/oneflow/pull/8387/files)
- Optimized the test time of global test. (https://github.com/Oneflow-Inc/oneflow/pull/8468)
- Optimized the execution time of test_math_ops, test_loss, test_activation, test_tensor_part1, test_tensor_part2 and other eager test. (https://github.com/Oneflow-Inc/oneflow/pull/8494)
- Optimized test_convtranspose, test_einsum, test_sqrt_square_sum in expensive eager test. (https://github.com/Oneflow-Inc/oneflow/pull/8504)
Models
- Added the test of LiBai in CI. (https://github.com/Oneflow-Inc/oneflow/pull/7537, https://github.com/Oneflow-Inc/oneflow/pull/7929)
- Fixed the speed test for Swin-Transformer. (https://github.com/Oneflow-Inc/oneflow/pull/7840)
- Added the benchmark test for flow-vision.(https://github.com/Oneflow-Inc/oneflow/pull/7806, https://github.com/Oneflow-Inc/oneflow/pull/8024)
- Added compatibility tests for `conv_mixer`, `densenet`, `ghostnet`, `googlenet`, `inception_v3`, `mnasnet`, `rexnet`, `rexnet_lite`, `res2net`, `shufflenet_v2`, `squeezenet`, `convnext`, `crossformer`, `efficientnet`, `levit`, `mlp_mixer`, `poolformer`, `pvt`, `res_mlp`, `uniformer`, `swin_transformer`, `senet` and other models. Fixes such compatibility issues as conv2d module padding parameter does not support string; the parameter list of functional.layer_norm is not aligned; meshgrid does not support the input of list[tensor]; adds a interface for tensor.reshape_as. (https://github.com/Oneflow-Inc/oneflow/pull/7942)
- Fixed the bug of Swin-Transformer dataloader. (https://github.com/Oneflow-Inc/oneflow/pull/8037)
- Added single-node 4-Gpus tests for models such as InsightFace in oneflow_face repository. (https://github.com/Oneflow-Inc/oneflow/pull/8130)
Bug fixes
Graph
- Fixed the bug of nccl deadlock caused by CUDA kernel asynchronous launch limit for nccl logical kernel in 3-D parallelism. (https://github.com/Oneflow-Inc/oneflow/pull/7924)
- Fixed cycle import of scope and session. (https://github.com/Oneflow-Inc/oneflow/pull/7993)
- Used log_softmax + nll to make sparse_softmax_cross_entropy ms more stable numerically for calculating subgraphs. (https://github.com/Oneflow-Inc/oneflow/pull/7987)
- Fixed the bug that B2P boxing misses TaskEdge lbi. (https://github.com/Oneflow-Inc/oneflow/pull/8052)
- Fixed the problem that compilation fails due to eager free tensor is not in nn.Graph's job. (https://github.com/Oneflow-Inc/oneflow/pull/8114)
- Fixed the possible problem of SegmentFault caused by BlobDesc. (https://github.com/Oneflow-Inc/oneflow/pull/8252)
- Solved the bug of circular import in python 3.6. (https://github.com/Oneflow-Inc/oneflow/pull/8268)
- Solved the problem that Graph's input and parameter/buffer tensors fail to handle non-contiguous tensors.(https://github.com/Oneflow-Inc/oneflow/pull/8281)
- Solved the potential deadlock caused by inconsistent partial order execution of multiple ranks in 3-D parallelism. （https://github.com/Oneflow-Inc/oneflow/pull/8226）
- Fixed the bug that Ibverbs failed to start the environment due to incorrect mtu value in special network environment. (https://github.com/Oneflow-Inc/oneflow/pull/8451)
- Solved the potential deadlock caused by the partial order execution of each rank when the subsequent subgraph of GradAcc is inserted into the NCCL logical op; at the same time, traverse the subsequent subgraph of GradAcc more comprehensively to solve the problem of missing NCCL op. (https://github.com/Oneflow-Inc/oneflow/pull/8459)
- Fixed the bug that NCCL logical kernels does not support bool type. (https://github.com/Oneflow-Inc/oneflow/pull/8455)
- Fixed the bug of tensor detach and clone in Graph. (https://github.com/Oneflow-Inc/oneflow/pull/8498)
Eager
- Aligned `DataLoader.__next__` interface (https://github.com/Oneflow-Inc/oneflow/pull/7835)
- Fixed backtracking failure when calculating higher-order derivatives, which is caused by the capturing of forward detached tensors via ` AutoGrad`
- Fixed inadequate execution of the semantics of sync by Barrier Instruction (https://github.com/Oneflow-Inc/oneflow/pull/7702)
- Fixed memory leak caused by imperfect management of VM instruction count
- Fixed `getitem` when tensor device id is not in the current rank
- Fixed `global norm` error on gradient calculation for various placements when calling clip grad in pipeline parallelism in eager global mode (https://github.com/Oneflow-Inc/oneflow/pull/7879)
- Fixed possible int32 arithmetic overflow caused by `Shape.elem_cnt` (https://github.com/Oneflow-Inc/oneflow/pull/8178)
- Fixed incorrect results produced by `Module.to_global` when introducing parameters (https://github.com/Oneflow-Inc/oneflow/pull/8187)
- Fixed extra GPU memory usage in `flow.load` and `module.load_state_dict` (https://github.com/Oneflow-Inc/oneflow/pull/8301)
- Fixed extra GPU memory usage when Optimizer loads models (https://github.com/Oneflow-Inc/oneflow/pull/8310)
- Fixed the error occurs when loading models via `flow.load` in multi nodes (https://github.com/Oneflow-Inc/oneflow/pull/8314)
- Fixed instability of eager caused by the introduction of callback thread (https://github.com/Oneflow-Inc/oneflow/pull/8193)
- Fixed `tensor.from_numpy` interface to avoid memory leak when the input of numpy is non-contiguous tensor (https://github.com/Oneflow-Inc/oneflow/pull/8391)
- Fixed stack overflow when destructing the deep backward computational graph after recursion (https://github.com/Oneflow-Inc/oneflow/pull/8056)
Operators & Tensor
Global Tensor
- Fixed global SBP inference of `unfold` (https://github.com/Oneflow-Inc/oneflow/pull/7883)
- Fixed global SBP inference of `grid_sample` (https://github.com/Oneflow-Inc/oneflow/pull/7881)
- Fixed incorrect pass of values in slice boxing kernel in certain cases (https://github.com/Oneflow-Inc/oneflow/pull/7893)
- Fixed eager global inplace (https://github.com/Oneflow-Inc/oneflow/pull/7903)
- Fixed SBP inference of `upsample` op (https://github.com/Oneflow-Inc/oneflow/pull/7884)
- Fixed SBP inference of `ScatterAdd`, `ScatterUpdate`, and `ScatterScalarUpdate` (https://github.com/Oneflow-Inc/oneflow/pull/7807)
- Fixed backward memory error of `partial_fc` with Global Tensor (https://github.com/Oneflow-Inc/oneflow/pull/8041)
- Added support for S0 in `randperm` and fixed equal local tensors across all ranks in random op in Split (https://github.com/Oneflow-Inc/oneflow/pull/7571)
- Fixed tensor getitem index error in global (https://github.com/Oneflow-Inc/oneflow/pull/8153)
- Fixed SBP inference of `RoiAlign` and added global unit test (https://github.com/Oneflow-Inc/oneflow/pull/7794)
- Fixed SBP inference of `stack` op (https://github.com/Oneflow-Inc/oneflow/pull/8181)
- Fixed random initialization in median under CPU global (https://github.com/Oneflow-Inc/oneflow/pull/8245)
- Fixed SBP inference of `narrow` op and added global unit test for `narrow` and `chunk` (https://github.com/Oneflow-Inc/oneflow/pull/7750)
- Improved legal SBP list of `batch_matmul` (https://github.com/Oneflow-Inc/oneflow/pull/8385)
- Fixed NLLLoss’ failure to support model parallelism (https://github.com/Oneflow-Inc/oneflow/pull/8380)
- Fixed S->S and S->P inference in Slice Op SBP infer (https://github.com/Oneflow-Inc/oneflow/pull/8521)
Tensor
- Fixed the bug occurs when Tensor dim is set to -1
- Fixed failure for Tensor type to be directly transferred to int and float in Python (https://github.com/Oneflow-Inc/oneflow/pull/7927)
- Fixed the bug in `Tensor.is_contiguous` that skips initialization when caching and executes random initialization when getting values (https://github.com/Oneflow-Inc/oneflow/pull/7785)
- Fixed the bug in Tensor slice view under 1d contiguous (https://github.com/Oneflow-Inc/oneflow/pull/7898)
- Fixed incorrect processing of None value by `Tensor.__eq__` (https://github.com/Oneflow-Inc/oneflow/pull/7938)
- Fixed unaligned memory size in `from_numpy` interface (https://github.com/Oneflow-Inc/oneflow/pull/7963)
- Fixed incorrect initialization of random seed in Tensor (https://github.com/Oneflow-Inc/oneflow/pull/7904)
- Fixed failure of ` oneflow.Size` to create Tensor with a specified shape (https://github.com/Oneflow-Inc/oneflow/pull/8429)
- Aligned `alpha` parameter in `Tensor.add` (https://github.com/Oneflow-Inc/oneflow/pull/8140)
Scalar Tensor
- Fixed failure of `add` to support Scalar Tensor (https://github.com/Oneflow-Inc/oneflow/pull/7827)
- Fixed failure of `reduce_sum` to support Scalar Tensor (https://github.com/Oneflow-Inc/oneflow/pull/7866)
- Fixed failure of `one_hot` to support Scalar Tensor (https://github.com/Oneflow-Inc/oneflow/pull/7975)
Fixed failure of `gather` to support Scalar Tensor (https://github.com/Oneflow-Inc/oneflow/pull/8376)
- Fixed “memory access out of bounds” error in `dim_scatter` kernel under Scalar Tensor (https://github.com/Oneflow-Inc/oneflow/pull/8418)
- Fixed failure of start and end parameters in `arrange` op to support Scalar Tensor (https://github.com/Oneflow-Inc/oneflow/pull/8522)
- Fixed failure of `all` to support Scalar Tensor and 0-Size Tensor (https://github.com/Oneflow-Inc/oneflow/pull/8547)
0-Size Tensor
- Fixed failure of `conv` and `deconv` to support 0-Size Tensor (https://github.com/Oneflow-Inc/oneflow/pull/8001)
- Fixed failure of `cuda_check_numerics` to support 0-Size Tensor (https://github.com/Oneflow-Inc/oneflow/pull/8050)
- Fixed failure of `expand` and `advanced_index` to support 0-Size Tensor (https://github.com/Oneflow-Inc/oneflow/pull/8094)
- Fixed the bug occurs when processing 0-Size Tensor in `repeat_interleave` kernel and removed relevant special judge in `gather` (https://github.com/Oneflow-Inc/oneflow/pull/8414)
- Fixed failure of `diag` to support 0-Size Tensor (https://github.com/Oneflow-Inc/oneflow/pull/8557)
Operators
- Fixed sorting in `nms` unit test (https://github.com/Oneflow-Inc/oneflow/pull/7831)
- Fixed torch alignment of beta and threshold interfaces of `softplus` op (https://github.com/Oneflow-Inc/oneflow/pull/7888)
- Fixed failure of `expand` to support passing tuples as parameters (https://github.com/Oneflow-Inc/oneflow/pull/7913)
- Fixed computation failure in `randperm` when n is too large (https://github.com/Oneflow-Inc/oneflow/pull/7908)
- Fixed failure relative to list or tuple in parameter passing in `meshgrid` (https://github.com/Oneflow-Inc/oneflow/pull/7933)
- Fixed `nn.functional.conv2d` bug that all parameters must be specified (https://github.com/Oneflow-Inc/oneflow/pull/7892)
- Fixed failure of `rand` and `randn` to support tuple as an input (https://github.com/Oneflow-Inc/oneflow/pull/7914)
- Fixed the bug occurs in `concat` when inputs are of inconsistent data types (https://github.com/Oneflow-Inc/oneflow/pull/7921)
- Fixed wrong device id got by generator in certain cases in `randn`,`dropout`, `randint`, `rand`, `random_mask_like`, and `randperm` (https://github.com/Oneflow-Inc/oneflow/pull/7896)
- Fixed inconsistent behaviors of `__shfl_sync` under `sm_61` in `layernorm` (https://github.com/Oneflow-Inc/oneflow/pull/7978)
- Fixed failure of `scatter` op to support negative dim (https://github.com/Oneflow-Inc/oneflow/pull/7934)
- Fixed the bug in `scatter` op nd update value(https://github.com/Oneflow-Inc/oneflow/pull/7953)
- Fixed failure of `masked_select` to support certain Broadcast operations in eager mode (https://github.com/Oneflow-Inc/oneflow/pull/7984)
- Fixed the bug in `PReLU` op when dispatching num_blocks (https://github.com/Oneflow-Inc/oneflow/pull/8004)
- Fixed misused numpy forced synchronization logic in `index_select` python and transferred the logic to functor for implementation (https://github.com/Oneflow-Inc/oneflow/pull/7965)
- Aligned dtype parameter in `prod` (https://github.com/Oneflow-Inc/oneflow/pull/7932)
- Fixed the bug occurs when `ord = 0` in `linalg.vector_norm` op; Fixed check on nan/inf by clip_grad (https://github.com/Oneflow-Inc/oneflow/pull/8007)
- Fixed failure of `min` and `max` to operate on inconsistent dtypes (https://github.com/Oneflow-Inc/oneflow/pull/8021)
- Added `num_batches_tracked ` buffer to `batch_norm` to facilitate transfer of ResNet-18, a torch pretrained model, to OneFlow (https://github.com/Oneflow-Inc/oneflow/pull/7920)
- Fixed the misuse of `logf`, `expf`, and `powf` in math kernel (https://github.com/Oneflow-Inc/oneflow/pull/8038)
- Fixed exclusion of dtype parameters in `cumsum` and `cumprod` and provided `Tensor.cumsum` and `Tensor.cumprod` methods (https://github.com/Oneflow-Inc/oneflow/pull/8065)
- Fixed possible overflow when dtype is not int64 in `non_zero` op (https://github.com/Oneflow-Inc/oneflow/pull/7907)
- Aligned `sum`, `mean`, `all`, `any`, and `prod` operations in `reduce` (https://github.com/Oneflow-Inc/oneflow/pull/8085)
- Fixed incorrect backward computation in `cumprod` (https://github.com/Oneflow-Inc/oneflow/pull/8136)
- Aligned `alpha` parameter in `sub` operation (https://github.com/Oneflow-Inc/oneflow/pull/8026)
- Fixed shape inference in `upsample` op (https://github.com/Oneflow-Inc/oneflow/pull/8105)
- Fixed failure of `addn` inplace operation on CPU tensor (https://github.com/Oneflow-Inc/oneflow/pull/8280)
- Fixed limit on tensor size in `cum` backward op based on the size of shared memory (https://github.com/Oneflow-Inc/oneflow/pull/8289)
- Improved the logic of dtype inference for `arange` op (https://github.com/Oneflow-Inc/oneflow/pull/8338)
- Fixed NaN propagation of UnaryFunctor (https://github.com/Oneflow-Inc/oneflow/pull/8346)
- Fixed ndim check of `pad` (https://github.com/Oneflow-Inc/oneflow/pull/8354)
- Fixed vector check in `broadcast_min` and `broadcast_max` backward computations (https://github.com/Oneflow-Inc/oneflow/pull/8379)
- Fixed the bug relative to index computation logic in `cumprod` op (https://github.com/Oneflow-Inc/oneflow/pull/8388)
- Fixed possible int32 overflow in `softmax` and math unary / binary cuda kernel; for kernels that operate integer division on `i` in `CUDA_1D_KERNEL_LOOP`, provided `if` statement to branch computations to prevent performance loss in most cases when int32 works (https://github.com/Oneflow-Inc/oneflow/pull/8472)
- Fixed failure to pass size via `size=(...)` in random ops (`normal`, `rand`, `randn`, `randint`, and `randperm`) (https://github.com/Oneflow-Inc/oneflow/pull/8506)
Device
- Fixed error in `cudaGetDeviceCount` when CUDA device count=0 (https://github.com/Oneflow-Inc/oneflow/pull/8184)
- Fixed possible unregistration of devices caused by `hob.ToString` method; Used static local variables to establish dependency between static variables of device registration and the static code for device registration (https://github.com/Oneflow-Inc/oneflow/pull/8235)
- Fixed `cudaErrorNoDevice` caused by drive errors (https://github.com/Oneflow-Inc/oneflow/pull/8262)
- Fixed memory leak caused by realpath (https://github.com/Oneflow-Inc/oneflow/pull/8540)
Higher order derivative
- Introduced AutogradCapturedTensor in backward computation to avoid circular reference and allow correct backtracking to the input gradient node in higher order derivative graph (https://github.com/Oneflow-Inc/oneflow/pull/7808)
- Added higher order derivative of `sin/cos` op; Fixed `autograd` bugs relative to higher order derivative (https://github.com/Oneflow-Inc/oneflow/pull/8163)
- Fixed bugs in backward computation in `concat` and `split_like` to support higher order derivative (https://github.com/Oneflow-Inc/oneflow/pull/8208)
Build
- Fixed RTD [sphinx] failure to build docstr (https://github.com/Oneflow-Inc/oneflow/pull/7901)
- Fixed compilation failure caused by opencv copy header failure (https://github.com/Oneflow-Inc/oneflow/pull/7944)
- Fixed failure to generate a new `.so` in compilation when `CMAKE_LINK_DEPENDS_NO_SHARED=YES` (https://github.com/Oneflow-Inc/oneflow/pull/7868)
- Fixed Eigen url in cmake third party (https://github.com/Oneflow-Inc/oneflow/pull/8223)
- Fixed the bug caused by multi-time linking to libof_protoobj in XRT (https://github.com/Oneflow-Inc/oneflow/pull/8326)
- Made libproto a dynamic library to avoid collision between static global variables (https://github.com/Oneflow-Inc/oneflow/pull/8345)
- Made `of_pyext_obj` static only when there is one Python extension dynamic library that has Python symbols (https://github.com/Oneflow-Inc/oneflow/pull/8393)
- Fixed the bug in `undefined symbol: del_curterm` in source code compilation (https://github.com/Oneflow-Inc/oneflow/issues/8398)
- Fixed false positive warning in gcc11 compilation (https://github.com/Oneflow-Inc/oneflow/pull/8401)
- Fixed SegFault that occurs when unzipping dataset in the container by making zlib a dynamic library (https://github.com/Oneflow-Inc/oneflow/pull/8481)
- Fixed undefined reference of culibosTlsSetValue (https://github.com/Oneflow-Inc/oneflow/pull/8479)
- Fixed stringop-truncation compilation error for gcc9 (https://github.com/Oneflow-Inc/oneflow/pull/8532)
CI
- Disabled static link of Simple CI and enabled debug build to avoid too many symbols (https://github.com/Oneflow-Inc/oneflow/pull/7940)
- Fixed the bug in AutoTest fake program; Fixed print error in AutoTest (https://github.com/Oneflow-Inc/oneflow/pull/8279; https://github.com/Oneflow-Inc/oneflow/pull/8290)
Module
- Disabled conv3d test temporarily for its relatively large error of random values (https://github.com/Oneflow-Inc/oneflow/pull/7969)
- Reduced test error in nn.LayerNorm (https://github.com/Oneflow-Inc/oneflow/pull/7941)
- Optimized input data range of certain math op tests (https://github.com/Oneflow-Inc/oneflow/pull/8010)
- Fixed incorrect unit test case in `permute` (https://github.com/Oneflow-Inc/oneflow/pull/8083)
- Aligned error message of chunk to torch (https://github.com/Oneflow-Inc/oneflow/pull/8096)
- Fixed incorrect use of `permute` in tensor tests (https://github.com/Oneflow-Inc/oneflow/pull/8144)
- Fixed omission of test cases in `instancenorm` (https://github.com/Oneflow-Inc/oneflow/pull/8215)
- Adjusted unit test threshold for `leaky_relu` (https://github.com/Oneflow-Inc/oneflow/pull/8242)
- Annotated cpu bn grad method that tests with random values (https://github.com/Oneflow-Inc/oneflow/pull/8257)
- Skipped test cases of `global argmax` and `median` in multi-GPU scenarios (https://github.com/Oneflow-Inc/oneflow/pull/8264)
- Adjusted unit test threshold for `fused_dot_feature_interaction` (https://github.com/Oneflow-Inc/oneflow/pull/8293)
- Disabled unit tests for `conv_transpose1d`, `conv_transpose2d`, and `conv_transpose3d` (https://github.com/Oneflow-Inc/oneflow/pull/8319)
- Adjusted tolerance setting in embedding_renorm unit test (https://github.com/Oneflow-Inc/oneflow/pull/8394)
- Removed test cases with excessive accumulated elements in `test_fused_dot_feature_interaction_pooling_sum` to avoid overly large sum error (https://github.com/Oneflow-Inc/oneflow/pull/8425)
Documentation
- Ensured that all PyTorch references in OneFlow API documentation belong to the same PyTorch version (1.10.0) (https://github.com/Oneflow-Inc/oneflow/pull/8058)
- Added "copy" button for code in API docs to facilitate trial runs of sample code (https://github.com/Oneflow-Inc/oneflow/pull/7997)
- Refined script that automatically generates version status for OneFlow APIs and fixed bugs in docs (https://github.com/Oneflow-Inc/oneflow/pull/8546)
- Refined interface documentation of Tensor and Module (https://github.com/Oneflow-Inc/oneflow/pull/7823)
- Refined `Tensor.to_global` interface documentation and added descriptions of ` gard_sbp`
- Refined ` Tensor.to_local` interface documentation
- Added Tensor Attributes docs for `oneflow.placement`, `oneflow.env.all_device_placement`, and `oneflow.sbp.sbp`
- Added interface documentation for `Module.to_consistent` (outdated) and `Module.to_global`
- Fixed invalid links in Tensor docs and updated `consistent` to `global` (https://github.com/Oneflow-Inc/oneflow/pull/7821)
- Added docstr for `Tensor.sqrt`, `Tensor.square`, `Tensor.addmm`, `Tensor.cosh`, `Tensor.diagonal`, `Tensor.log`, `Tensor.ndim`, and `Tensor.rsqrt` (https://github.com/Oneflow-Inc/oneflow/pull/7841)
- Enabled derived classes of pybind11 to add documentation for non-overriding methods and added interface documentation related to Tensor and autograd (https://github.com/Oneflow-Inc/oneflow/pull/7849)
- Refined documentation of `oneflow.argsort` (https://github.com/Oneflow-Inc/oneflow/pull/7844)
- Refined documentation of `Tensor.zero_`, `Tensor.is_contiguous`, `Tensor.is_cuda`, and `oneflow.nn.functional.layer_norm` op (https://github.com/Oneflow-Inc/oneflow/pull/7839)
- Refined interface documentation of `support_sparse` and `step` in `oneflow.optim.Adamw`, `oneflow.optim.SGD` (https://github.com/Oneflow-Inc/oneflow/pull/7848)
- Refined interface documentation of `LambdaLR.step`, `ReduceLROnPlateau.in_cooldown`, and `ReduceLROnPlateau.is_better` (https://github.com/Oneflow-Inc/oneflow/pull/7848)
- Refined interface documentation of `nn.Module` (https://github.com/Oneflow-Inc/oneflow/pull/8190)
- Refined interface documentation of `oneflow.optim.lr_scheduler.PolynomialLR` (https://github.com/Oneflow-Inc/oneflow/pull/8430)
- Refined docs and formula illustrations for `oneflow.nn.CombinedMarginLoss` (https://github.com/Oneflow-Inc/oneflow/pull/8206)
- Refined documentation of `oneflow.logical_and`, `oneflow.logical_or`, `oneflow.logical_xor`, and `oneflow.logical_not` (https://github.com/Oneflow-Inc/oneflow/pull/8297)
- Fixed the bug in the documentation of quantization ops (https://github.com/Oneflow-Inc/oneflow/pull/8333)
- Updated solution in Troubleshooting for the case when `libunwind.h` is not found (https://github.com/Oneflow-Inc/oneflow/pull/8336)
- Restructured API documentation based on features; added and refined docs of features that are unique to OneFlow (https://github.com/Oneflow-Inc/oneflow/pull/8392)