Smooth Pinball Neural Network for Probabilistic Forecasting of Wind Power

This paper introduces a novel approach for nonparametric probabilistic forecasting of wind power, termed the smooth pinball neural network (SPNN), which combines a smooth approximation of the pinball loss function with a neural network architecture and a weighting initialization scheme to prevent the quantile cross-over problem.

The paper states: "This paper analyzes the effectiveness of a novel approach for nonparametric probabilistic forecasting of wind power that combines a smooth approximation of the pinball loss function with a neural network architecture and a weighting initialization scheme to prevent the quantile cross over problem."

The authors motivate the work by noting that due to the chaotic nature of weather, variable and uncertain wind power production poses planning and operational challenges unseen in conventional generation. They highlight that point forecasting can result in unavoidable errors which can be significant and they also lack information on uncertainty, thus probabilistic forecasts are needed.

The main contributions are summarized as follows:

1. "We use a new and simple objective function which is a logistic based smooth approximation of the pinball loss function for multiple quantile regression and show how to apply it to train a neural network with standard gradient descent back-propagation."

2. We introduce an initialization weighting scheme to prevent the quantile cross over problem and improve estimation.

3. We are the first to showcase a quantile based neural network for probabilistic forecasting of wind with a sliding window of training data.

4. We design experiments to validate our model using publicly available data from 10 wind farms from the Global Energy Forecasting Competition 2014 and benchmark performance with common and advanced methods.

The paper provides mathematical background on probabilistic forecasting, defining that "Given a random variable Yt such as wind power at time t, its probability density function is defined as ft and its the cumulative distribution function as Ft. If Ft is strictly increasing, the quantile qt(τ) of the random variable Yt is uniquely defined as the value x such that P(Yt < x) = τ or equivalently as the inverse of the distribution function qt(τ) = Ft−1(τ). The paper notes that In probabilistic forecasting, we are trying to predict one of two classes of density functions, either parametric or nonparametric, and for wind power, the wind density may fluctuate therefore making nonparametric forecasting more ideal then fitting a parametric density."

The paper reviews quantile regression, stating Quantile regression is a popular approach for nonparametric probabilistic forecasting. It was introduced by [35] for estimating conditional quantiles and is closely related to models for the conditional median. The pinball loss function is defined as ρτ(u) = τu if u ≥ 0 and (τ − 1)u if u < 0, where 0 < τ < 1 is the tilting parameter.

For evaluation, the paper uses the quantile score (QS), which is found to be a proper scoring rule; it is related to the continuous rank probability score; and it is also the main evaluation criteria in the 2014 Global Energy Forecasting Competition (GEFCOM 2014). The QS is defined as QS = (1/N) Σ t=1 N Σ m=1 M ρτm(yt − q̂t(τm)). Additionally, the paper uses reliability measures (PICP and ACE) and sharpness measures (interval score, IS).

The core of the paper is the SPNN model. The authors note that "the pinball function ρ employed by the original linear quantile regression model in Eq. (1) is not differentiable at the origin, x = 0. The non-differentiability of ρ makes it difficult to apply gradient based optimization methods in fitting the quantile regression model." Therefore, they use a smooth approximation proposed by Zheng: Sτ,α(u) = τu + α log(1 + exp(−u/α)), where α > 0 is a smoothing parameter. The paper states: Zheng proved in [41] that in the limit as α → 0+ that Sτ,α(u) = ρτ(u).

The SPNN architecture is described: "It has an input layer, output layer, and one hidden layer. The input layer consists of nx number of input nodes and takes vector Xt of input features at time t. The hidden layer consists of nh number of hidden neurons and the output layer consists of M number of output nodes corresponding to the estimated quantiles." The hidden layer uses the tanh activation function, and the output layer uses the identity activation function.

The objective function for SPNN is given by:

E = (λ1/(2NM))W[1]2 F + (λ2/(2NM))W[2]2 F + (1/(NM)) Σ t=1 N Σ m=1 M [τm(yt − q̂t(τm)) + α log(1 + exp(−(yt − q̂t(τm))/α))]

The paper explains: To train our model, we use standard gradient descent with backpropagation. The gradients are derived in detail, showing the chain rule application for both the hidden-to-output weights and the input-to-hidden weights.

To address the quantile cross-over problem, the paper introduces a novel weight initialization scheme: "Our scheme prevents quantiles from crossing over by initializing estimates of weights to fixed quantile values. In our approach we start by assuming all past observations of wind power correspond to a uniform distribution such that given a predictor Xt for all training observations t = 1... N and M number of quantiles we're interested in estimating we get the following quantile output matrix. The output weights are then calculated as W[2] = H† · Q, where H = tanh(X · W[1]) and H† is the Moore-Penrose generalized inverse. The paper notes: This scheme does not guarantee there are no cross-overs, but it does greatly reduce them as that it is seen in section IV."

The case study uses data from GEFCom2014, with the 12 months of 2013 from all 10 zones for testing. Training is done using a sliding window of two previous months to forecast the third month. The features used are: "derived wind speeds at 10m and 100m, wind direction at 10m and 100m, wind energy at 10m and 100m, hour of day, day of the year, and we also include in training the four raw wind speeds at 10m and 100m for U and V directions."

The benchmark methods include "the persistence model that corresponds to the normal distribution and is formed by the last 24 hours of observations, the climatology model that is based on all past wind power, and the uniform distribution that assumes all observations occur with equal probability. For our advanced benchmarks we then use multiple quantile regression (MQR) with L2 regularization, and support vector quantile regression (SVQR) with a radial basis function kernel."

The hyperparameters chosen were: 10000 training iterations, 20 hidden nodes, 0.3 for the learning rates, 0.01 for the smoothing rate, and 0.1 for each of the weight regularization terms.

Results show that "SPNN-w results in the lowest quantile scores across all ten zones by a significant amount. This provides evidence of the value of our proposed method in providing full predictive densities. We see that between SPNN-wo and SPNN-w there is a decrease in the drop of the QS meaning that smart weight initialization also leads to better performance. For the ACE score, SPNN overall has the lowest or second lowest ACE for most of the zones. For the interval score, SPNN-w has the lowest across all zones. This means it produced the sharpest intervals."

Regarding quantile cross-overs, the paper states: "We see in Fig. 2 that cross overs for MQR and SVQR range from 200 to 900 for each zone and SPNN-wo is over 1000 for some zones. However, with the weight initialization scheme applied in SPNN-w we see a drop of almost two orders of magnitude in cross-overs."

The paper concludes: "Wind power forecasting is crucial for many decision making problems in power systems operations, and is a vital component in integrating more wind into the power grid. Due to the chaotic nature of the wind it is often difficult to forecast. Uncertainty analysis in the form of probabilistic wind prediction can provide a better picture of future wind coverage. This paper introduces a novel approach for probabilistic wind forecasting using a neural network with smooth approximation to the pinball ball loss function in estimating quantiles. We develop a novel weight initialization scheme where weights are fixed to a squared error approximation of the wind power corresponding to the uniform distribution, to ensure multiple quantiles can be estimated simultaneously without overlapping each other. We verify the effectiveness of our SPNN model with the dataset of the Global Energy Forecasting Competition 2014. We compare forecasts to common and advanced benchmarks and are evaluated using the quantile score, reliability, and sharpness metrics. Our results show superior performance across the prediction horizons, which verify effectiveness of the model for forecasting while preventing estimated quantiles from overlapping."

Future work is mentioned: "Future work will look into applying SPNN to forecast ocean wave and solar power, to test its effectiveness across different renewable energies, and on electricity pricing and load demand for smart grid applications. In this study we trained our model using NWP data. Another problem to study is very short term probabilistic forecasting using only past wind power data. Future work can also then look into expanding the SPNN model for providing full predictive densities given lagged past data of power only. New extensions need to be explored to ensure that past data is retained in memory by using deep or recurrent neural network architectures."

Based on the paper, here are the specific improvements I can make to AI systems and what the improved system can do:

1. Replace non-differentiable loss functions with smooth approximations in neural network training

• Implement the logistic-based smooth pinball loss (Eq. 3) instead of the standard pinball loss (Eq. 1) for quantile regression tasks. This enables gradient descent backpropagation without subgradient methods or linear programming solvers.

2. Add a weight initialization scheme to prevent quantile crossover in multi-quantile neural networks

• Initialize output weights using the Moore-Penrose pseudoinverse of the hidden layer activations multiplied by a uniform quantile matrix (Section III-C). This ensures initial quantile estimates are monotonically ordered and equidistant, reducing crossover by nearly two orders of magnitude (from >1000 to <20 per zone).

3. Train a single neural network to output multiple quantiles simultaneously

• Use an output layer with M nodes (one per quantile level) and a shared hidden layer. The objective function sums the smooth pinball loss across all quantiles (Eq. 5). This avoids training M separate models and reduces computational cost.

4. Apply sliding-window training with L2 regularization for non-stationary time series

• Use a two-month sliding window for training and forecast one month ahead. Include L2 regularization terms (λ1, λ2) on both weight matrices to prevent overfitting on limited historical data.

1. Probabilistic wind power forecasting with 18 quantiles (5% to 95%)

• Output a full predictive density for each hour, enabling construction of 10% to 90% prediction intervals. Achieves quantile scores 5–15% lower than multiple quantile regression and support vector quantile regression across all 10 wind farm zones.

2. Guarantee monotonic quantile estimates

• The initialization scheme ensures that lower quantiles (e.g., 5%) never exceed higher quantiles (e.g., 95%) during training and inference. This eliminates the need for post-hoc reordering heuristics.

3. Handle nonlinear, bounded, and skewed data distributions

• The neural network with a tanh hidden layer captures nonlinear relationships between numerical weather predictions (wind speed, direction, energy) and wind power. The smooth loss handles asymmetric errors typical of wind power (bounded between 0 and 1).

4. Provide reliable and sharp prediction intervals

• Achieves average coverage error (ACE) below 5% (compared to 6–8% for benchmarks) and interval scores (IS) that are 10–20% lower, meaning narrower intervals without sacrificing coverage.

5. Scale to multiple zones or assets with shared architecture

• The same network structure and hyperparameters (20 hidden nodes, 10,000 iterations, learning rate 0.3, smoothing α=0.01) work across all 10 wind farms without per-zone tuning, indicating robustness and transferability.

6. Train efficiently on standard hardware

• With 2 months of hourly data (1,440 samples) and 12 input features, training completes in minutes on a standard CPU (Intel i7, 16GB RAM), making it suitable for operational forecasting where models are retrained monthly.