(717f) Control Policies for Energy Storage Systems Under Uncertainty

The use of energy storage is expected to increase towards 2030 and beyond to handle variable and uncertain energy sources such as solar and wind in the power grid, as well as solar thermal energy and industrial waste heat in thermal systems [1]. From a control perspective, we need more research efforts in designing policies for operating energy systems in scenarios of high variability and uncertainty.

A policy (sometimes called decision function, rule or feedback control law) is a function that determines a feasible control action given what we know at a point in time. There are two fundamental strategies for designing policies [2]:

• Policy search. Here we use an objective function to search within a family of functions to find a function that works best.
• Lookahead approximations. Alternatively, we can construct policies by approximating the impact of a decision now on the future.

Either of these approaches can produce optimal policies, although this is rare. The reason is computational, a stochastic dynamic program can rarely be solved to optimality. However, these two strategies are the basis for four universal approximations (policy function approximations, cost function approximations, value function approximations, and direct lookahead approximations) that cover all of the approaches that have ever been used in the literature [2]. When considering the problem of designing a policy for a specific application, it is useful to screen within different types of approximations, because each strategy can work best depending on the problem characteristics [3].

In this work, we analyze a district heating network in the city of Trondheim, where the heat is produced in a waste incineration plant. The network has also back-up electric boilers that can be used on demand by purchasing electricity at the price of the spot electricity market, which is highly stochastic. The aggregated heat demand is time-dependent, but relatively predictable. We consider also an energy storage system (a hot water tank). The objective is to design a policy for controlling the heat flows in the network such that we satisfy the energy demand and minimize the price paid for the electricity over time. We design two different policies:

• The first policy is an analytical function with two tunable parameters related to the price of the electricity. The idea is to charge the tank when the electricity is cheap (below some parameter) and discharge the tank when the price of the electricity is high (above some other parameter). These two parameters are tuned using stochastic gradient methods. This policy belongs to the policy function approximation class, and it is widely used in industry due to its simplicity.
• The second policy is a scenario-based model predictive control, which uses a stochastic model of the future to make decisions over time. This type of policy belongs to the direct lookahead approximation class.

Both policies are benchmarked against the posterior optimal solution (solution to the deterministic problem when we consider a perfect forecast of the future). We show that simple policy function approximations properly tuned can be very effective for systems where we have an idea about the structure of the solution ("buy low, sell high").

REFERENCES
[1] A. M. Annaswamy and M. Amin, IEEE vision for smart grid controls: 2030 and beyond, IEEE, 2013.
[2] W. B. Powell, A unied framework for stochastic optimization, European Journal of Operational Research, 275 (2019), pp. 795-821.
[3] W. B. Powell and S. Meisel, Tutorial on stochastic optimization in energy|part ii: An energy storage illustration, IEEE Transactions on Power Systems, 31 (2015), pp. 1468-1475.