Bigdl

Highlights
* Add MKL-DNN Int8 support, especially for VNNI acceleration support. Low precision inference accelerates both latency and throughput significantly
* Add support for runnning MKL-BLAS models under MKL-DNN. We leverage MKL-DNN to speed up both training and inference for MKL-BLAS models
* Add Spark 2.4 support. Our examples and APIs are fully compatible with Spark 2.4, we released the binary for Spark 2.4 together with other Spark versions

Details
* [New Feature] Add MKL-DNN Int8 support, especially for VNNI support
* [New Feature] Add support for runnning MKL-BLAS models under MKL-DNN
* [New Feature] Add Spark 2.4 support
* [New Feature] Add auto fusion to speed up model inference
* [New Feature] Memoery reorder support for low precision inference
* [New Feature] Add bytes support for DNN Tensor
* [New Feature] Add SAME padding in MKL-DNN layers
* [New Feature] Add combined (add/or) triggers for training completion
* [Enhancement] Inception-V1 python training support enhancement
* [Enhancement] Distributed Optimizer enhancement to support customized optimizer
* [Enhancement] Add compute output shape for DNN supported layers
* [Enhancement] New MKL-DNN computing thread pool
* [Enhancement] Add MKL-DNN support for Predictor
* [Enhancement] Documentation enhancement for Sparse Tensor, MKL-DNN support, etc
* [Enhancement] Add ceilm mode for AvgPooling and MaxPooling layers
* [Enhacement] Add binary classification support for DLClassifierModel
* [Enhacement] Improvement to support conversion between NHWC and NCHW for memory reoder
* [Bug Fix] Fix SoftMax layer with narrowed input
* [Bug Fix] TensorFlow loader to support checking all data types
* [Bug Fix] Fix Add operation bug to support double type when loading TensorFlow graph
* [Bug Fix] Fix one-step weight update missing issue in validation during training
* [Bug Fix] Fix scala compiler security issue in 2.10 & 2.11
* [Bug Fix] Fix model broadcast cache UUID issue
* [Bug Fix] Fix predictor issue for batch size == 1

Highlights
* MKL-DNN support enhancement, which includes training optimization, more models training support and model serialization support
* A new distributed optimizer for models powered by MKL-DNN. This optimizer can overlap training and communication during the distributed training, which lead to a better scalability on multi-nodes
Details
* [New Feature] A new optim method ParallelAdam which leverages the multi-thread capacity
* [New Feature] Add new validation methods HitRate, which is widely used in recommendation
* [New Feature] Add new validation methods NDCG, which is widely used in recommendation
* [New Feature] Support communication priority when synchronize parameter in the distributed training
* [New Feature] Support ModelBroadcast customization
* [New Feature] Add a new distributed optimizer for models powered by MKL-DNN. This optimizer can overlap training and communication during the distributed training, which lead to a better scalability on multi-nodes
* [API Change] Add batch size into the Python model.predict API
* [Enhancement] Add MKL-DNN training example for LeNet
* [Enhancement] Improve the training performance by getting rid of narrowing gradients and zero gradients for model powered by MKL-DNN
* [Enhancement] Add training example for VGG-16 based on MKL-DNN
* [Enhancement] Support nested table in Graph output
* [Enhancement] Enhancement on thread pool to make it compatible with MKL-DNN engine
* [Enhancement] MKL-DNN model serialization support
* [Enhancement] Add VGG-16 validation example
* [Bug Fix] Fix JoinTable throwing exception during backward if batch size is changed
* [Bug Fix] Change Reshape to InferReShape in ReshapeLoadTF
* [Bug Fix] Fix splitBatch issue in Predictor, where the model has multiple Graph and each Graph outputs a table
* [Bug Fix] Fix MDL-DNN inference performance issue not to copy weights at inference
* [Bug Fix] Fix the issue that the training will crash if there are unlabeled data
* [Bug Fix] Fix the issue that the input is grey image while the model needs 3 channels input
* [Bug Fix] Correct the style check job to make both input and output file format to UTF-8 format
* [Bug Fix] Load the relevant library only if MKL-DNN engine specified
* [Bug Fix] Shade org.tensorflow.framework to avoid conflict
* [Bug Fix] Fix dlframes not packaged in pip issue
* [Bug Fix] Fix LocalPredictor cannot be serialized because of nested logger variable
* [Bug Fix] Need to clear Recurrent preTopology's output while cloneCells
* [Bug Fix] MM layer output different output for same input if ran multiple times
* [Bug Fix] Distribute predictor will send model twice when do `mapPartition`
* [Document] Kubernetes programming guide to spark2.3
* [Document] Add document for wrap preprocessor and model in one graph and add its python API

Highlights
* We integrate [MKL-DNN](https://github.com/intel/mkl-dnn) as an alternative execution engine for CNN models. MKL-DNN provides better training/inference performance and less memory consuming. On some CNN models, we find there’s 2x throughput improvement in our experiment.
* Support using different optimization methods to optimize different parts of the model. This is necessary when train some models.
* Spark 2.3 support. We have tested our code and examples on Spark 2.3. We release the binary for Spark 2.3, and Spark 1.5 will not be supported.

Details
* [New Feature] MKL-DNN integration. We integrate [MKL-DNN](https://github.com/intel/mkl-dnn) as an alternative execution engine for CNN models. It supports speedup layers like: AvgPooling, MaxPooling, CAddTable, LRN, JoinTable, Linear, ReLU, SpatialConvolution, SpatialBatchnormalization, Softmax. MKL-DNN provides better training/inference performance and less memory consuming.
* [New Feature] Layer fusion. Support layer fusion on conv + relu, batchnorm + relu, conv + batchnorm and conv + sum(some of the fusion can only be applied in the inference). Layer fusion provides better performance especially on inference. Currently layer fusion are only available for MKL-DNN related layers.
* [New Feature] Multiple optimization method support in optimizer. Support using different optimization methods to optimize different parts of the model.
* [New Feature] Add a new optimization method Ftrl, which is often used in recommendation model training.
* [New Feature] Add a new example: Training Resnet50 on ImageNet dataset.
* [New Feature] Add new OpenCV based image preprocessing transformer ChannelScaledNormalizer.
* [New Feature] Add new OpenCV based image preprocessing transformer RandomAlterAspect.
* [New Feature] Add new OpenCV based image preprocessing transformer RandomCropper.
* [New Feature] Add new OpenCV based image preprocessing transformer RandomResize.
* [New Feature] Support loading Tensorflow Max operation.
* [New Feature] Allow user to specify input port when loading Tensorflow model. If the input operation accepts multiple tensors as input, user can specify which to feed data to instead of feed all tensors.
* [New Feature] Support loading Tensorflow Gather operation.
* [New Feature] Add random split for ImageFrame
* [New Feature] Add setLabel and getURI API into ImageFrame
* [API Change] Add batch size into the Python model.predict API.
* [API Change] Add generateBackward into load Tensorflow model API, which allows user choose whether to generate backward path when load Tensorflow model.
* [API Change] Add feature() and label() to the Sample.
* [API Change] Deprecate the DLClassifier/DLEstimator in org.apache.spark.ml. Prefer using DLClassifier/DLEstimator under com.intel.analytics.bigdl.dlframes.
* [Enhancement] Refine StridedSlice. Support begin/end/shrinkAxis mask just like Tensorflow.
* [Enhancement] Add layer sync to SpatialBatchNormalization. SpatialBatchNormalization can calculate mean/std on a larger batch size. The model with SpatialBatchNormalization layer can converge to a better accuracy even the local batch size is small.
* [Enhancement] Code refactor in DistriOptimizer for advanced parameter operations, e.g. global gradient clipping.
* [Enhancement] Add more models into the LoadModel example.
* [Enhancement] Share Const values when broadcast the model. The Const value will not be changed and we can share it when use multiple model for inference on a same node, which will reduce memory usage.
* [Enhancement] Refine the getTime and time counting implementation.
* [Enhancement] Support group serializer so that layers of the same hierarchy could share the same serializer.
* [Enhancement] Dockerfile use Python 2.7.
* [Bug Fix] Fix memory leak problem when using quantized model in predictor.
* [Bug Fix] Fix PY4J Java gateway not compatible in Spark local mode for Spark 2.3.
* [Bug Fix] Fix a bug in python inception example.
* [Bug Fix] Fix a bug when run Tensorflow model using loop.
* [Bug Fix] Fix a bug in the Squeeze layer.
* [Bug Fix] Fix python API for random split.
* [Bug Fix] Using parameters() instead of getParameterTable() to get weight and bias in serialization.
* [Document] Fix incorrectness in Quantized model document.
* [Document] Fix incorrect instructions when generate Sequence files for ImageNet 2012 dataset in the document.
* [Document] Move bigdl-core build document into a separated page and refine the format.
* [Document] Fix incorrect command in Tensorflow load and transfer learning examples.

Highlights
* Bring in a Keras-like API(Scala and Python). User can easily run their Keras code (training and inference) on Apache Spark through BigDL. For more details, see [this link](https://bigdl-project.github.io/0.5.0/#KerasStyleAPIGuide/keras-api-python/).
* Support load Tensorflow dynamic models(e.g. LSTM, RNN) in BigDL and support more Tensorflow operations, see [this page](https://bigdl-project.github.io/0.5.0/#APIGuide/tensorflow_ops_list/ ).
* Support combining data preprocessing and neural network layers in the same model (to make model deployment easy )
* Speedup various modules in BigDL (BCECriterion, rmsprop, LeakyRelu, etc.)
* Add DataFrame-based image reader and transformer

New Features
* Tensor can be converted to OpenCVMat
* Bring in a new Keras-like API for scala and python
* Support load Tensorflow dynamic models(e.g. LSTM, RNN)
* Support load more Tensorflow operations(InvertPermutation, ConcatOffset, Exit, NextIteration, Enter, RefEnter, LoopCond, ControlTrigger, TensorArrayV3,TensorArrayGradV3, TensorArrayGatherV3, TensorArrayScatterV3, TensorArrayConcatV3, TensorArraySplitV3, TensorArrayReadV3, TensorArrayWriteV3, TensorArraySizeV3, StackPopV2, StackPop, StackPushV2, StackPush, StackV2, Stack)
* ResizeBilinear support NCHW
* ImageFrame support load Hadoop sequence file
* ImageFrame support gray image
* Add Kv2Tensor Operation(Scala)
* Add PGCriterion to compute the negative policy gradient given action distribution, sampled action and reward
* Support gradual increase learning rate in LearningrateScheduler
* Add FixExpand and add more options to AspectScale for image preprocessing
* Add RowTransformer(Scala)
* Support to add preprocessors to Graph, which allows user combine preprocessing and trainable model into one model
* Resnet on cifar-10 example support load images from HDFS
* Add CategoricalColHashBucket operation(Scala)
* Predictor support Table as output
* Add BucketizedCol operation(Scala)
* Support using DenseTensor and SparseTensor together to create Sample
* Add CrossProduct Layer (Scala)
* Provide an option to allow user bypass the exception in transformer
* DenseToSparse layer support disable backward propagation
* Add CategoricalColVocaList Operation(Scala)
* Support imageframe in python optimizer
* Support get executor number and executor cores in python
* Add IndicatorCol Operation(Scala)
* Add TensorOp, which is an operation with Tensor[T]-formatted input and output, and provides shortcuts to build Operations for tensor transformation by closures. (Scala)
* Provide a docker file to make it easily to setup testing environment of BigDL
* Add CrossCol Operation(Scala)
* Add MkString Operation(Scala)
* Add a prediction service interface for concurrent calls and accept bytes input
* Add SparseTensor.cast & SparseTensor.applyFun
* Add DataFrame-based image reader and transformer
* Support load tensoflow model files saved by tf.saved_model API
* SparseMiniBatch supporting multiple TensorDataTypes

Enhancement
* ImageFrame support serialization
* A default implementation of zeroGradParameter is added to AbstractModule
* Improve the style of the document website
* Models in different threads share weights in model training
* Speed up leaky relu
* Speed up Rmsprop
* Speed up BCECriterion
* Support Calling Java Function in Python Executor and ModelBroadcast in Python
* Add detail instructions to run-on-ec2
* Optimize padding mechanism
* Fix maven compiling warnings
* Check duplicate layers in the container
* Refine the document which introduce how to automatically Deploy BigDL on Dataproc cluster
* Refactor adding extra jars/python packages for python user. Now only need to set env variable BIGDL_JARS & BIGDL_PACKAGES
* Implement appendColumn and avoid the error caused by API mismatch between different Spark version
* Add python inception training on ImageNet example
* Update "can't find locality partition for partition ..." to warning message

API change
* Move DataFrame-based API to dlframe package
* Refine the Container hierarchy. The add method(used in Sequential, Concat…) is moved to a subclass DynamicContainer
* Refine the serialization code hierarchy
* Dynamic Graph has been an internal class which is only used to run tensorflow models
* Operation is not allowed to use outside Graph
* The getParamter method as final and private[bigdl], which should be only used in model training
* remove the updateParameter method, which is only used in internal test
* Some Tensorflow related operations are marked as internal, which should be only used when running Tensorflow models

Bug Fix
* Fix Sparse sample batch bug. It should add another dimension instead of concat the original tensor
* Fix some activation or layers don’t work in TimeDistributed and RnnCell
* Fix a bug in SparseTensor resize method
* Fix a bug when convert SparseTensor to DenseTensor
* Fix a bug in SpatialFullConvolution
* Fix a bug in Cosine equal method
* Fix optimization state mess up when call optimizer.optimize() multiple times
* Fix a bug in Recurrent forward after invoking reset
* Fix a bug in inplace leakyrelu
* Fix a bug when save/load bi-rnn layers
* Fix getParameters() in submodule will create new storage when parameters has been shared by parent module
* Fix some incompatible syntax between python 2.7 and 3.6
* Fix save/load graph will loss stop gradient information
* Fix a bug in SReLU
* Fix a bug in DLModel
* Fix sparse tensor dot product bug
* Fix Maxout ser issue
* Fix some serialization issue in some customized faster rcnn model
* Fix and refine some example document instructions
* Fix a bug in export_tf_checkpoint.py script
* Fix a bug in set up python package.
* Fix picklers initialization issues
* Fix some race condition issue in Spark 1.6 when broadcasting model
* Fix Model.load in python return type is wrong
* Fix a bug when use pyspark-with-bigdl.sh to run jobs on Yarn
* Fix empty tensor call size and stride not throw null exception

Highlights
* Supported all Keras layers, and support Keras 1.2.2 model loading. See [keras-support](https://bigdl-project.github.io/0.4.0/#ProgrammingGuide/keras-support/) for detail
* Python 3.6 support
* OpenCV support, and add a dozen of image transformer based on OpenCV
* More layers/operations

New Features
* Models & Layers & Operations & Loss function
+ Add layers for Keras: Cropping2D, Cropping3D, UpSampling1D, UpSampling2D, UpSampling3D, masking,Maxout,HighWay,GaussianDropout, GaussianNoise, CAveTable, VolumetricAveragePooling, HardSigmoidSReLU, LocallyConnected1D, LocallyConnected2D, SpatialSeparableConvolution, ActivityRegularization, SpatialDropout1D, SpatialDropout2D, SpatialDropout3D
+ Add Criterion for keras: PoissonCriterion, KullbackLeiblerDivergenceCriterion, MeanAbsolutePercentageCriterion, MeanSquaredLogarithmicCriterion, CosineProximityCriterion
+ Support NHWC for LRN and BatchNormalization
+ Add LookupTableSparse (lookup table for multivalue)
+ Add activation argument for recurrent layers
+ Add MultiRNNCell
+ Add SpatialSeparableConvolution
+ Add MSRA filler
+ Support SAME padding in 3d conv and allows user config padding size in convlstm and convlstm3d
+ TF opteration: SegmentSum, conv3d related operations, Dilation2D, Dilation2DBackpropFilter, Dilation2DBackpropInput, Digamma, Erf, Erfc, Lgamma, TanhGrad, depthwise, Rint, All, Any, Range, Exp, Expm1, Round, FloorDiv, TruncateDiv, Mod, FloorMod, TruncateMod, IntopK, Round, Maximum, Minimum, BatchMatMu, Sqrt, SqrtGrad, Square, RsqrtGrad, AvgPool, AvgPoolGrad, BiasAddV1, SigmoidGrad, Relu6, Relu6Grad, Elu, EluGrad, Softplus, SoftplusGrad, LogSoftmax, Softsign, SoftsignGrad, Abs, LessEqual, GreaterEqual, ApproximateEqual, Log, LogGrad, Log1p, Log1pGrad, SquaredDifference, Div, Ceil, Inv, InvGrad, IsFinite, IsInf, IsNan, Sign, TopK. See details at [tensorflow_ops_list](https://bigdl-project.github.io/0.4.0/#APIGuide/tensorflow_ops_list/))
+ Add object detection related layers: PriorBox, NormalizeScale, Proposal, DetectionOutputSSD, DetectionOutputFrcnn, Anchor
* Transformer
+ Add image Transformer based on OpenCV: Resize, Brightness, ChannelOrder, Contrast, Saturation, Hue, ChannelNormalize, PixelNormalize, RandomCrop, CenterCrop, FixedCrop, DetectionCrop, Expand, Filler, ColorJitter, RandomSampler, MatToFloats, AspectScale, RandomAspectScale, BytesToMat
+ Add Transformer: RandomTransformer, RoiProject, RoiHFlip, RoiResize, RoiNormalize
* API change
+ Add predictImage function in LocalPredictor
+ Add partition number option for ImageFrame read
+ Add an API to get node from graph model with given name
+ Support List of JTensors for label in Python API
+ Expose local optimizer and predictor in Python API
* Install & Deploy
+ Support BigDL on [Spark on k8s](https://github.com/apache-spark-on-k8s/spark)
* Model Save/Load
+ Support big-sized model (parameter exceed > 2.1G) for both java and protobuffer
+ Support keras model loading
* Training
+ Allow user to set new train data or new criterion for optimizer reusing
+ Support gradient clipping (constant clip and clip by L2-norm)

Enhancement
* Speed up BatchNormalization.
* Speed up MSECriterion
* Speed up Adam
* Speed up static graph execution
* Support reading TFRecord files from HDFS
* Support reading raw binary files from HDFS
* Check input size in concat layer
* Add proper exception handling for CaffeLoader&Persister
* Add serialization support for multiple tensor numeric
* Add an Activity wrapper for Python to simplify the returning value
* Override joda-time in hadoop-aws to reduce compile time
* LocalOptimizer-use modelbroadcast-like method to clone module
* Time counting for paralleltable's forward/backward
* Use shade to package jar-with-dependencies to manage some package conflict
* Support loading bigdl_conf_file in multiple python zip files

Bug Fix
* Fix getModel failed in DistriOptimizer when model parameters exceed 2.1G
* Fix core number is 0 where there's only one core in system
* Fix SparseJoinTable throw exception if input’s nElement changed.
* Fix some issues found when save bigdl model to tensorflow format file
* Fix return object type error of DLClassifier.transform in Python
* Fix graph generatebackward is lost in serialization
* Fix resizing tensor to empty tensor doesn’t work properly
* Fix Adapter layer does not support different batch size at runtime
* Fix Adaper layer cannot be serialized directly
* Fix calling wrong function when set user-defined mkl threads
* Fix SmoothL1Criterion and SoftmaxWithCriterion doesn’t deal with input’s offset.
* Fix L1Regularization throw NullPointerException while broadcasting model.
* Fix CMul layer will crash for certain configure

Highlights
* New protobuf-based model storage format
* Support model quantization
* Support sparse tensor and model
* Easier and broader Tensorflow model load support
* More layers/operations
* Apache Spark 2.2 support

New Features
* Models & Layers & Operations & Loss function
+ Support convlstm3D model
+ Support Variational Auto Encoder
+ Support Unet
+ Support PTB model
+ Add SpatialWithinChannelLRN layer
+ Add 3D-deconv layer
+ Add BifurcateSplitTable layer
+ Add KLD criterion
+ Add Gaussian layer
+ Add Sampler layer
+ Add RNN decoder layer
+ Support NHWC data format in 2D-conv, 2D-pooling layers
+ Support same/valid padding type in 2D-conv and 2D-pooling layers
+ Support dynamic execution flow in Graph
+ Graph node can pass nested tensors
+ Layer/Operation can support different input and output numeric tensor
+ Start to support operations in BigDL, add following operations: LogicalNot, LogicalOr, LogicalAnd, 1D Max Pooling, Squeeze, Prod, Sum, Reshape, Identity, ReLU, Equals, Greater, Less, Switch, Merge, Floor, L2Loss, RandomUniform, Rank, MatMul, SoftMax, Conv2d, Add, Assert, Onehot, Assign, Cast, ExpandDims, MaxPool, Realdiv, BiasAdd, Pad, Tile, StridedSlice, Transpose, Negative, AssignGrad, BiasAddGrad, Deconv2D, Conv2DBackFilter CrossEntropy, MaxPoolGrad, NoOp, RandomUniform, ReluGrad, Select, Sum, Pow, BroadcastGradientArgs, Control Dependency
+ Start to support sparse layers in BigDL, add following sparse layers: SparseLinear, SparseJoinTable, DenseToSparse
* Tensor
+ Support sparse tensor
+ Support scalar (0-D tensor)
+ Tensor support more numeric type: boolean, short, int, long, string, char, bytestring
+ Tensor don’t display full content in toString when there’re too many elements
* API change
+ Expose evaluate API to python
+ Add a predictClass API to model to simplify the code when user want to use model in classification
+ Change model.test to model.evaluate in Python
+ Refine Recurrent, BiRecurrent and RnnCell API
+ Sample.features from ndarray to JTensor/List[JTensor]
+ Sample.label from ndarray to JTensor
* Install & Deploy
+ Support Apache Spark 2.2
+ Add script to run BigDL on Google DataProc platform
+ Refine run-example.sh scripts to run bigdl examples on AWS with build-in Spark
+ Pip install will now auto install spark-2.2
+ Add a docker file
* Model Save/Load
+ New model persistent format(protobuf based) to provide a better user experience when save/load bigdl models
+ Support load more operations from Tensorflow
+ Support read tensor content from Tensorflow checkpoint
+ Support load a subset of Tensorflow graph
+ Support load Tensorflow preprocessing graph(read/parse tfrecord data, image decoders and queues)
+ Automatically convert data in Tensorflow queue to RDD and feeding model training in BigDL
+ Support load deconv layer from caffe and Tensorflow
+ Support save/load SpatialCrossLRN torch module
* Training
+ Allow user to modify the optimization algorithm status when resuming the training in Python
+ Allow user to specify optimization algorithms, learning rate and learning rate decay when use BigDL in Spark * ML pipeline
+ Allow user to stop gradient on some layers in backpropagation
+ Allow user to freeze layer parameters in training
+ Add ML pipeline python API, user can use BigDL with ML pipeline in python code

Enhancement
1. Support model quantization. User can speed up model inference by quantize the model
2. Display bigdl model in Tensorboard
3. User can easily convert a sequential model to graph model by invoking new added toGraph method
4. Remove unnecessary contiguous check in 3D conv
5. Support global average pooling
6. Support regularizer in 3D convolution layer
7. Add regularizer for convlstmpeephole3d
8. Throw more meaningful messages in layers and criterions
9. Migrate GRU/LSTM/RNN/LSTM-Peehole definition from sequence to graph
10. Switch to pytest for python unit tests
11. Speed up tanh layer
12. Speed up sigmoid layer
13. Speed up recurrent layer
14. Support batch normalization in recurrent
15. Speedup Python ndarray to scala tensor convertion
16. Improve gradient sync performance in distributed training
17. Speedup tensor dot operation with mkl dot
18. Speedup copy operation in recurrent container
19. Speedup logsoftmax
20. Move classes.lst and img_class.lst to the model example folder, so user can easier to find them.
21. Ensure spark.speculation is set to false to get a better performance in training
22. Easier to turn on performance data in distributed training log
23. Optimize memory usage when broadcasting the model
24. Support mllib vector as feature for BigDL
25. Support create multiple tensors Sample in python
26. Support resizing in BytesToBGRImg

Bug Fix
1. Fix TemporalConv layer cannot return parameter table
2. Fix some bugs when loading dilated group convolution from caffe
3. Fix some bugs when loading caffe v1 layers
4. Fix a bug in TimeDistributed layer
5. Fix get incorrect execution time in recurrent layers
6. Fix inplace layer clear state bug
7. Fix incorrect training data sample count under some input
8. Remove label check in BytesToGreyImg
9. Fix a bug in concat table when it contains no layer
10. Fix a bug in maptable
11. Fix some typos in document
12. Use newInstance method to obtain FileSystem