Details about the data in the “RenewEconomy Widget”


This glossary still a work in progress – so if you see something awry below, please let us know.

Thanks for your interest in understanding more about the “Live Supply & Demand Widget” – located here on the NEM-watch portal, on the RenewEconomy site and (in smaller version) on a growing number of other websites across the internet (no charge to embed).

On this page, we intend to go into significant detail about the data conveyed in the widget in order to answer most of the commonly asked questions we’ve received over the years (including some triggered by our inclusion of the consumption side of the equation in November 2017).

 

(A)  The Unit of Measure is “megawatts”, which is a Rate

All data is shown in megawatts (i.e. MW) – with 1MW = 1,000kW = 1,000,000 W.

Remember that a rate of energy generation/transfer/consumption is measured in MW whereas a volume of energy generation/transfer/consumption is measured in MWh (i.e. a rate over a period of time).

 

(B)  A “point in time” view

The widget shows data aggregated together from several different data providers – so the data points shown in the widget are of slightly different cadence, and timestamps.
1.  We have designed the widget to show the most current data available
2.  You can mouse-over a particular data point to understand more.

If you’d like to trend data over time, you might find our NEMreview product to be of use in that respect – all of the data sets in the widget (and 10,000’s more) are available for licensed users.

 

(C)  About the main grids

It should be noted that the widget does not provide visibility of all of the electricity supplied across the country – however it does provide visibility of the lion’s share.  This is the case because the NEMwatch widget shows the supply and demand balance across Australia’s two main electricity grids:
1)   In the east, Australia’s National Electricity Market (NEM) covers only ½ of the Australian land-mass but accounts for roughly 90% of electricity consumed across the country.
2)   The second major grid is the South-West Interconnected System (SWIS) that supports Wholesale Electricity Market (WEM) which supplies the cities of Perth and Fremantle, and surrounding areas of south-western WA.

(C1)  The SWIS, in Western Australia

In the widget we show data from the SWIS as representing the whole of Western Australia – which is not strictly correct, but which is an approximation that is of a reasonable level of accuracy (i.e. given the total electricity consumption in other parts of Western Australia (including in the missing grids below) is relatively low).

The WEM operates in local Perth time, which is UTC + 8 hours.  This means that clocks in the WEM are 2 hours behind “NEM time” clocks – however all data in the widget (e.g. when you mouse over) is translated to NEM time to reduce confusion (though we know this will not totally alleviate all confusion, unfortunately).

With respect to the data shown as Supply-and-Demand for the whole of Western Australia (but in most cases representing just the SWIS), please note the following:

Supply-side Data
in Western Australia
In response to questions we’ve fielded on this widget since it was first launched in early 2015, we’ve provided some details below about the sources of the data used to represent Western Australia.

Black Coal, Gas, Liquid, Wind, Large Solar Data for these fuel types is supplied by the AEMO West (formerly the IMO) and is of a 30-minute cadence.

A focus of the widget is providing a high-level picture, so rather than distracting with unit-level data (which is available as a time-series in NEMreview v7) in the widget we take the raw data published by AEMO West, and aggregate up to fuel type using mapping functions we have established in our Generator Catalog.

 

Other The primary objective of the widget is to assist in raising the general level of awareness of the big picture of the current fuel mix providing for electricity supplies across Australia.  Hence on some occasions simplifications are used.

One of these simplifications has been to omit specific fuel types where they represent only a tiny proportion of the electricity generated across Australia in order to ensure the widget is as clear, and least crowded, as possible.  In cases such as this, generation from these other fuel types are aggregated together under an “Other” category.

Our Generator Catalog is also coded such that, whenever new Generators are added, any data pertaining to these new units are aggregated under the “Other” category until such time that the new units have been correctly assigned to their proper fuel type.  Hence keen observers might observe some short periods after new stations begin operation when the “Other” category jumps for an interim period..

Small Solar In March 2015, this widget was updated to include “APVI Small Solar”.  This is the estimated production from small-scale (i.e. predominantly rooftop) solar PV derived by the Australian PV Institute (APVI).

The method used by the APVI has been progressively evolved, over time, with the objective being to improve the accuracy of the estimate – however the key points to understand are that:
1)  This particular data set is an estimate, as distinct from actual metered data for the larger-scale plant;
2)  At its root, the estimation method is a “sample and extrapolation” method, whereby real output for a sample number of smaller installs is then extrapolated up to an estimated aggregate based on the total installed capacity of small-scale solar in a particular location.  It’s not perfect but (currently) is the best available method.  The challenges of the estimation method were also discussed in this Wattclarity article from June 2016.

This estimate is actually supplied for the whole of Western Australia, but we have made the assumption (because of population density) that this is almost equivalent to production in the SWIS.

Medium Solar Given that the small-scale solar figure relies on an installed capacity derived from STC registrations at the Clean Energy Regulator, and that there are a growing number of solar installations across Australia qualifying for LGCs instead (i.e. being larger than 100kW), but not being big enough to warrant the AEMO accessing live production from SCADA systems, there will be a growing volume of “Medium Solar” production not accounted for in the Small Solar or Large Solar numbers.

This was discussed further in this article from November 2017 on WattClarity.

(Large) Battery Discharging There are no large-scale battery facilities currently operating in the SWIS.  See below for the regions of the NEM.
(Small) Battery Discharging There currently are already a number of small-sized battery systems operation in the SWIS (and more broadly across WA).

However currently there is no visibility about how many battery systems are currently installed, and of what type – let alone any visibility of how these are operating.  This presents a number of growing challenges for the grid in this multi-faceted energy transition, and was the reason why we started our process of developing a national Energy Storage Register.

In the absence of data such as that, we cannot provide any visibility in this widget of when these smaller systems are discharging (i.e. acting like generators).

Other missing data We’re missing other embedded generation, for instance

If this does not answer the questions you had about Supply-Side Data for WA, please provide your feedback here.

Demand-side Data
in Western Australia
In this widget, we start with the principle that (because the SWIS is an isolated grid) instantaneous supply must always equals instantaneous demand.

Demand (approx AEMO Operational) Given that starting point, we have taken the approach of setting the “Operational Demand” number for Western Australia to be the simple sum of all of the generation supplied in the region at that same point in time – with the exception of the Small-Scale Solar (which is needed to be represented as below.
Demand (Pumping Hydro) There are no pumped storage hydro facilities currently operating in the SWIS.  See below for the regions of the NEM.
Demand (Large Battery Charging) There are no large-scale battery facilities currently operating in the SWIS.  See below for the regions of the NEM.
Demand (Small Battery Charging) As noted above, there currently are already a number of small-sized battery systems operation in the SWIS (and more broadly across WA).

However in the absence of visibility of these battery systems (which we’re seeking to address with  a national Energy Storage Register) we cannot provide visibility in this widget of when these smaller systems are charging (i.e. acting like loads).

Demand (the AEMO doesn’t see) We treat the APVI small-scale solar production data for WA the same as we do for each of the regions of the NEM on the demand side, labelling it “Demand the AEMO doesn’t see” in real time.

If this does not answer the questions you had about Demand-Side Data for WA, please provide your feedback here.

We’ll refine the above description of the data used to represent WA based on your feedback you can provide here.

(C2)  The National Electricity Market (NEM) on the east

In the widget we show data for each of the states that combine together to comprise the NEM (noting that there are pockets of some states not connected up to the NEM and hence overlooked in the data sets provided in the widget).

The NEM operates in Eastern Standard Time, which is UTC + 10 hours and does not adjust to daylight savings in NSW, VIC and TAS (nor does it adjust to the time-zone difference to South Australia).  People who work in the wholesale market are familiar in thinking in “NEM time” (but we appreciate that this concept will be foreign to some of the broader range of users of this widget).

With respect to the data shown as Supply-and-Demand for the NEM regions (Queensland, New South Wales, Victoria, Tasmania and South Australia), please note the following:

Supply-side Data
in QLD, NSW, VIC, SA, TAS 

(see here about why there’s no ACT shown)

In response to questions we’ve fielded on this widget since it was first launched in early 2015, we’ve provided some details below about the sources of the data used to represent the NEM.

Black Coal, Brown Coal, Gas, Liquid, Wind, Large Solar Data for these fuel types is supplied by the AEMO East (formerly NEMMCO) and is of a 5-minute cadence.

It is metered data at an “as generated” level (so not equivalent to the “sent-out” level used later for settlement), sampled from SCADA just prior to the start of a (5-minute) Dispatch Interval, but shown at the time point for the end of the Dispatch Interval – i.e. offset by a little over 5 minutes from the time at which it was actually metered.

A focus of the widget is providing a high-level picture, so rather than distracting with unit-level data (which is available as a time-series in NEMreview v7 or in real-time in ez2view) in the widget we take the raw data published by AEMO East, and aggregate up to fuel type using mapping functions we have established in our Generator Catalog.

Hydro Production This number (a single number for each region) shows the aggregate of each of the hydro production across a particular region – regardless of the type of hydro facility.

It is not possible to separately identify (in pumped storage hydro) if the water molecules had been previously pumped uphill, or had accumulated in the upper pond due to rainfall (or snowfall).

Other The primary objective of the widget is to assist in raising the general level of awareness of the big picture of the current fuel mix providing for electricity supplies across Australia.  Hence on some occasions simplifications are used.

One of these simplifications has been to omit specific fuel types where they represent only a tiny proportion of the electricity generated across Australia in order to ensure the widget is as clear, and least crowded, as possible.  In cases such as this, generation from these other fuel types are aggregated together under an “Other” category.

An example here is for the three sugar mills in QLD that have data accessible to, and published by, the AEMO in real time – Invicta, Pioneer and Rocky Point.  There are a number of other sugar mills (each with power stations, though smaller still), but these have not data accessible to AEMO in real time.

Our Generator Catalog is also coded such that, whenever new Generators are added, any data pertaining to these new units are aggregated under the “Other” category until such time that the new units have been correctly assigned to their proper fuel type.  Hence keen observers might observe some short periods after new stations begin operation when the “Other” category jumps for an interim period..

Small Solar In March 2015, this widget was updated to include “APVI Small Solar”.  This is the estimated production from small-scale (i.e. predominantly rooftop) solar PV derived by the Australian PV Institute (APVI).

The method used by the APVI has been progressively evolved, over time, with the objective being to improve the accuracy of the estimate – however the key points to understand are that:
1)  This particular data set is an estimate, as distinct from actual metered data for the larger-scale plant;
2)  At its root, the estimation method is a “sample and extrapolation” method, whereby real output for a sample number of smaller installs is then extrapolated up to an estimated aggregate based on the total installed capacity of small-scale solar in a particular location.  It’s not perfect but (currently) is the best available method.  The challenges of the estimation method were also discussed in this Wattclarity article from June 2016.

Medium Solar Given that the small-scale solar figure relies on an installed capacity derived from STC registrations at the Clean Energy Regulator, and that there are a growing number of solar installations across Australia qualifying for LGCs instead (i.e. being larger than 100kW), but not being big enough to warrant the AEMO accessing live production from SCADA systems, there will be a growing volume of “Medium Solar” production not accounted for in the Small Solar or Large Solar numbers.

This was discussed further in this article from November 2017 on WattClarity.

(Large) Battery Discharging As noted on Wattclarity in November 2017, the Telsa/Neoen battery in South Australia named the “Hornsdale Power Reserve” began charging on Saturday 25th November.  Discharge was also shown on the same day.

Other large batteries will follow in relatively short time, and will be visualised in the widget.

When discharging (like is the case for hydro), these  batteries need to be accounted for by the AEMO in the scheduling process.  Hence the AEMO accesses, and publishes, metered rate of production for these batteries.  In our widget we aggregate these by region to highlight operational pattern.

(Small) Battery Discharging There currently are already a number of small-sized battery systems operation across each of the NEM regions.

However currently there is no visibility about how many battery systems are currently installed, and of what type – let alone any visibility of how these are operating.  This presents a number of growing challenges for the grid in this multi-faceted energy transition, and was the reason why we started our process of developing a national Energy Storage Register.

In the absence of data such as that, we cannot provide any visibility in this widget of when these smaller systems are discharging (i.e. acting like generators).

Other missing data We’re missing other embedded generation, for instance

If this does not answer the questions you had about Supply-Side Data for any of the NEM regions, please provide your feedback here.

Demand-side Data
in QLD, NSW, VIC, SA, TAS(see here about why there’s no ACT shown)
In this widget, we start with the principle that (because the SWIS is an isolated grid) instantaneous supply must always equals instantaneous demand.

Demand (approx AEMO Operational) In the real-time operation of the NEM, it is not possible for the AEMO to aggregate a live reading of all of the millions of consumption meters scattered across the NEM.  Hence, in order to “measure” demand in real time, the AEMO adds up all of the electricity supplied by the generators it can see and makes adjustment based on interconnector flows to derive a region-level demand number.

In the widget, given that we want to explicitly show pumping at pumped storage hydro facilities, we are unable to use any of the AEMO’s own aggregated demand figures (there are a few).

Instead, we have to adopt our own approach by:
#1)  Add up supplies for all generators in the region;
#2)  Add onto this any imports from neighbouring region (as they are supplying consumption in the region in question); whilst
#3)  Subtract off exports to neighbouring region (for the same reason).
This gives us our estimate for what the AEMO refers to as “Operational Demand”.

A little later on we will link through to an even greater description of Operational Demand.

Demand (Pumping Hydro) In the NEM there are currently 3 pumped storage hydro facilities:
#1) the upper reservoir at Tumut in the Snowy Scheme
#2) the Wivenhoe pumped storage system in South-East Queensland, where the hydro reservoir sits above the large Wivenhoe flood mitigation and water supply dam; and
#3) The Shoalhaven scheme in NSW.The pumping loads associated with each of these schemes are Scheduled Loads, meaning that the owners of these facilities bid into the NEM for prices below which they will be happy to consume (i.e. pump water uphill to charge their “battery”).  Because of this, the AEMO accesses, and publishes in real time, metered data for the current rate of consumption at each of these facilities.  These charging rates are aggregated up for the relevant region, and shown in the widget.Note that these demands are integrated into the AEMO’s “Demand and Non-Sched Generation” data set, which is why we can’t use that directly in the widget and hence have to compile our own estimate of Operational Demand.
Demand (Large Battery Charging) As noted on Wattclarity in November 2017, the Telsa/Neoen battery in South Australia named the “Hornsdale Power Reserve” began charging on Saturday 25th November.

Like the pumping loads in pumped hydro, the “pumping” load used to charge the battery is also a Scheduled Load. Hence the AEMO accesses, and publishes, metered rate of consumption for these batteries.  In our widget we aggregate these by region to highlight operational pattern.

Demand (Small Battery Charging) As noted above, there currently are already a number of small-sized battery systems operation in each region of the NEM.

However in the absence of visibility of these battery systems (which we’re seeking to address with  a national Energy Storage Register) we cannot provide visibility in this widget of when these smaller systems are charging (i.e. acting like loads).

Demand (the AEMO doesn’t see)  

Because of the approach the AEMO takes to calculating a real-time view of Operational Demand, they effectively do not (cannot currently) “see” the share of consumption that small-scale solar production is supplying.

It’s irrelevant, from the perspective of this widget, whether an individual’s small-scale solar is exported to the grid, or consumed at home (though we’ve noted here on WattClarity how the incentives vary for each owner of a system).

The key point, from the perspective of this widget is that the AEMO does not see, in real time, the share of consumption supplied by small-scale PV.  Hence it is listed as such on the widget.

 

If this does not answer the questions you had about Demand-Side Data for any of the NEM regions, please provide your feedback here.

We’ll refine the above description of the data used to represent each region of the NEM based on your feedback you can provide here.

 

(C3)  Notable grids (and areas) not currently covered by the NEMwatch Widget

The NEMwatch widget does provide visibility of the vast majority of electricity supplied and consumed across Australia – however we are continuing to look at how we can expand the scope of coverage to include some of the smaller grids, including the following:

The Mt Isa grid in north-west QLD Discuss
The Darwin grid in NT Discuss
The Alice grid in NT Discuss
The Katherine grid in NT Discuss
The North-West Interconnected System (NWIS) in WA Discuss
Off-grid, in remote parts of Australia Discuss

If you can help us gain access to data for any of the above, please let us know how (by using this feedback form)

 

SPECIFIC NOTES ABOUT DIFFERENT DATA SETS

The following table provides a few pointers about different data sets – with more detail provided here on this page:

In Western Australia
(The WEM operates in local Perth time, which is UTC + 8 hours)
Supply Data for most of the fuel types is supplied by the AEMO West (formerly the IMO) and is of a 30-minute cadence.

This widget was updated (March 2015) to include “APVI Small Solar”, which is the estimated production from small-scale (i.e. predominantly rooftop) solar PV – described here .  This estimate is actually supplied for the whole of Western Australia, but we have made the assumption (because of population density) that this is almost equivalent to production in the SWIS.

Demand Data Being an isolated grid, supply always equals demand in the SWIS.
Across Queensland, NSW, Victoria, South Australia and Tasmania (the NEM).
(The NEM operates in Eastern Standard Time (UTC + 10 hours) and does not shift with daylight savings.  Hence for clarity it is known as “NEM time”)
Supply Data for most of the fuel types is supplied by the AEMO East and is of 5 minute cadence.

1)  It comes from the AEMO’s “Initial MW” SCADA readings taken from most of the larger generators that operate in the NEM and are allocated to respective Fuel Types using information in our Generator Catalog.

2)  In some cases where generation by a particular fuel type is relatively small (such as the case of the few bagasse-fired power stations in QLD that have data reported by AEMO) we have taken the step of leaving them classified as “Other” to save space in the legend.

3)  Where a new station appears with fuel type that is not yet set, it will appear as fuel type “Other” for an interim period.

This widget was updated (March 2015) to include “APVI Small Solar”, which is the estimated production from small-scale (i.e. predominantly rooftop) solar PV – described here .

Demand What is shown as “AEMO Operational Demand” is a number we calculate from unit level data in order to ensure that we can correctly provide visibility to consumption for charging of storage (either battery charging, or pumping to pumped storage hydro).  This means that the figure for AEMO Operational Demand does not match to a single raw data field published by AEMO.

Pumping to pumped storage hydro, being a scheduled load, is reported explicitly by AEMO.

Similarly, charging of large-scale batteries (starting with Hornsdale Power Reserve) is reported explicitly by AEMO.

Finally, there are a number of points of consumption that the AEMO can’t see (especially in real time) – such as where it is supplied by generators that operate “behind the meter”.  Small-scale PV is one example of this, but there are others (such as embedded generation on industrial sites).  As we are able to access (or estimate) other aspects of this, we will evolve our representation.

Inter-Regional Flows The NEM operates as 5 interconnected “regions” approximating state boundaries (with the ACT part of the NSW region).

By seeing the difference between supply and demand in a particular region, you are able to understand the extent to which the region is importing (or exporting) at that time.

 

(D)  Specific notes about data providers

More here…

About the APVI For the night owls, or those who like to use electronic forms, provide us your feedback on this form
… or just give us a call You can, of course, just give us a call -> Brisbane business hours on +61 7 3368 4064

More

 

(E)  Can’t find what you’re looking for above?

We’ve tried, in the above, to provide a fairly comprehensive explanation of the approach taken to the data provided in

(E1)  OTHER FREQUENTLY ASKED QUESTIONS ABOUT THE WIDGET

If your questions are not answered above, you might find answers on this Frequently Asked Questions page, which we will continue to update as new questions come to hand.

If this is not enough to help explain what we’ve done with the data on the “Live Supply & Demand Widget”, don’t hesitate to provide us your feedback directly (or you can just call us on +61 7 3368 4064).

(E2)  DO YOU HAVE FEEDBACK YOU’D LIKE TO PROVIDE?

Given that the widget has been operating since 2015, we’ve fielded a number of questions about it since that time.  We’ve tried to clearly answer the most common questions here on a Frequently Asked Questions page.

If you have specific points of feedback about our “Live Supply & Demand Widget”, we’re keen to hear from you:

Use our feedback form For the night owls, or those who like to use electronic forms, provide us your feedback on this form
… or just give us a call You can, of course, just give us a call -> Brisbane business hours on +61 7 3368 4064