Utility Data Sources
There are numerous different types of data sources available. Similarly, there are different methods for acquiring those. All of these have to be considered when you're deciding how to set your program, how to inform your system owners, and how to prepare your organization. What we'll do now is just review a sampling of the choices that are available to different organizations and enterprises, and we will start by discussing the landscape of billing data. Billing or invoicing information is usually the first piece of the puzzle. It's usually where you begin when you're establishing some sort of energy management or analytics platform. The billing data is usually sourced from the monthly statements themselves. File types when we're talking about billing data can range from the paper bills, literally the traditional mail service bills in their envelopes, account by account, arriving usually every month. On the other end of the spectrum will be things like the larger investor-owned utilities offering web services, maybe true APIs. Aside from the sources themselves, methods of acquisition, again, are going to range from the mailroom and its staff, possibly a secondary scanning operation that may be staffed all the way through to the choice or the decision to maybe just use a service provider or a data provider to help acquire these types of data sets. We'll dig a little deeper into this list of usual suspects and talk about some of the strengths and weaknesses of each.Paper
We'll start, obviously, with paper. This is the traditional method, the invoices themselves, communication between the utility and the customers. There are some strengths that need to be pointed out in this. This modality is going to offer what we consider a high level of detail. Paired with that is the form of the document. This medium itself is going to include terminology and vocabulary that most everyone's familiar with. These are the established methods that have been in place for years or decades. Your analyst, your accountants, are going to be familiar with the way the tariffs are described, the way the charges are described in this format. We're also going to walk through weaknesses with this approach:- Your method of transition between this source and your database relies on manual data entry.
- You're potentially compromising data integrity.
- Factoring at scale always raises a risk.
The next method we'll discuss really for most intents and purposes – it's paper again. What you're doing when you're trying to create a scanning step in the process is just introduce a link in the chain that gives you the ability to have a lighter, more mobile format, but really this is just paper again. Again, the strengths remain. You're going to have a format that's detail rich, and you're going to have, again, taxonomy, vocabulary that everyone who's familiar with energy is going to be familiar with. Here, you're going to get an advantage in terms of at scale it's going to be easier to organize and file these types of sources away. They're going to be mobile. They're going to be portable, you can move them from department to department internally. You can share them with your customers and other vendors more readily. Again, the weakness here is going to be tied to data entry assuming that that is still the method that you're using to move from source into data system or data platform. There's a secondary possible alternative here in that character recognition technology can be an option assuming you want to do that in house. Obviously, there are going to be costs associated with a specialized service like that, but that is a possible alternative once leaving paper and moving to scan-based sources. Because this is really just paper, you still have the concerns about the environmental impact of relying on traditional mail.
- Line item detail
- Familiar vocabulary
- Portability
- Speed advantage of bill availability
- Environmentally friendly format
Again, because it's purely electronic it's much more environmentally friendly as an alternative. Similar to the scanning method where we mentioned OCR as a possible innovative response, these are structured files. There are parsing opportunities. There are software solutions that can allow systems to ingest and parse these technologies. There are going to be considerations there about what it will cost to build that expertise, or if you want to find a vendor that could help you with that.
Electronic Data Interchange (EDI)
Still sort of in the mode of discussing true electronic sources, we'll talk about EDI now. This is specifically designed, and when we say EDI we mean most likely the EDI 810 standard which is most commonly used for utilities. This is really designed specifically for business-to-business transactions, another form of e-billing that utilities support. By a lot of assessments, this option might be, in terms of speed and reliability of scale, the most efficient method. There are no opportunities for error in the functional transition between source to data system here. It's going to be similar also in bringing the bill availability speed to bear, the time between meter read and the publication of this format, is going to be much faster than any operation that's relying on traditional mail. There are some unique, we would say, cons or weaknesses to this approach:- By truly being just a system-to-system transaction, there may be a need to map some form terminology. Frequently, the EDI standard will create acronyms or abbreviations for things that would've appeared in a paper bill. There may be a bit of a learning curve or a mapping exercise as it moves from the source into your data system.
- Similarly, there's really no receipt when these transactions are complete. When I say receipt, I just mean an image of the bill that you could hand to an auditor or to an analyst to justify that month's transaction. There's a lot of value for a lot of operations in always having some form of the bill to go back to and justify what occurred, and that is largely absent in EDI 810 arrangements because the utility is using this as a complete alternative to the paper channel.
Text File
Another type of electronic format that may be available would be a comma delimited file type, we list here as just a text file. Some of the larger utilities may offer through a MyEnergy portfolio or MyAccount program some way for you take all of your currently enrolled accounts and just make data requests of some kind. Obviously here, there is a big speed and scalability advantage. You're also likely to still benefit from the delta between meter read and this product being available. You're going to be ahead of traditional mail and paper services again in this mode. Often because this is, I guess, similar to an ad hoc reporting system, you can make much more customized requests. Maybe in your operation, you only want a few data points related to usage things like peak demand only, and maybe you don't need full line item on charges. There are a few charges that interest you and it's very possible that with these types of interfaces you can make specific or let's custom requests of the utility. Now paired with that a possible weakness or a con to those strengths are that just by virtue of the fact that there's more of a dynamic nature of the data configuration in these systems they are also likely limiting the overall detail. This is probably not going to be a perfect one-to-one of every data point that's available on the eventual statement. With the freedom to order up the data a la carte you're also probably being a little restricted in total amount of available detail. Again, here you're probably not going to receive an actual bill image that would've looked like the mailed or e-bill file. That may or may not be a significant sacrifice in your system.API
The next and possibly the final electronic alternative is probably also the most advanced. This would be utilities that offer more of a true web service or an API exchange, an electronic format that's just completely independent of the billing system. Similar to EDI 810 itself, this probably brings the most speed and reliability at scale. Still going to have the bill availability advantages, and when compared to a delimited ad hoc reporting interface this format will be less likely to exhibit any data limitations. This is probably going to be much more a full one-to-one representation, but by being more complex, by being more complete it's also going to have a more significant information services or information technology resource requirement. You're probably going to have to involve development, a few engineers, somewhere in the process to make sure you're incorporating and integrating these types of REST requests and file types correctly. Also, likely in these cases, no bill image, no bill receipt. It bears mentioning that this is sort of leading edge. They're going to be just a really small minority of utilities that offer a true API or XML type service.Data Service
The final thing we just want to mention is it's also possible that instead of developing your own internal methods for receiving these types of data sources and incorporating all the possible methods for reading them and moving them from source into your database, you may just decide to work with a partner. Working with a data provider is going to mean someone has already answered the question about which of the modalities are most appropriate in which place, and they've likely also devised a certain standard way of delivering to you so that the end-to-end process is a little less complicated from your desk or your chair. An organization that is, let's say, regionally and functionally complicated, with different commodities, different territories, they may decide that utility data collection is a project that is best suited for in-house development, the need for total control, and that the time and effort required for that is acceptable. Of course, there will always be situations where you decide that working with a partner might make more strategic sense. Working with a partner is not without some challenges:- You're going to have to be careful in your selection of which service provider is going to fit you best.
- You're going to need to understand that service provider's internal methods of collection, what modalities they prefer and what that means to you.
- Coordinating with the utilities by way of a third party is going to inherently add one more level of complexity, and so there's probably a certain amount of project management resources that will be needed as well.
Speak with a data expert today to learn how the Urjanet Utility Data Platform can help your business.
Check out the scorecard in these slides to get a snapshot of each of these data source's pros and cons:
Interval Data Access
Now we want to just shift gears a little bit and talk about the other side of the equation, the metering data. When gathering AMI, advanced metering or smart metering information, this is really normally a natural evolution for an energy management platform and energy management team. Once you've made some decision about how to source and how to place the invoice information, your next step is going to be going after the metered source.Integrating interval data from smart meters can give your energy management program a strategic boost. Learn more in our eBook.
Here the primary sources are going to range from initiatives, I'm sure you've all heard of Green Button, that's a great example, all the way the other end of the spectrum to hardware installation and network infrastructures themselves. Managing those different source types, again, with Green Button on the one hand, we're talking just about some way to interface with data download capabilities or using APIs all the way out to maybe once again using a service partner or provider if you're dealing with maybe a more complicated integration.Green Button
Here too, we'll just quickly look at the landscape of these source types and talk in a little more detail about each. Let's start with Green Button. This is easily the most publicized source of meter-based information. It was officially launched in 2012, started in the West Coast. I think as of now there's somewhere between 50 and 100 utilities and suppliers that made or have made an announcement that they will participate in this initiative. The first thing we have to note here is this was generally advertised as a value added service, free of charge to most users, and it was also designed to be vendor friendly, so not just a way for you to see your own meter information but also a way that assuming that you had a sizeable portfolio and a complicated organization you could also share that information with partners and vendors without too much hassle. Because it is a published standard that means that there are commercially available front ends being developed all the time, and that in house development, if that's the method you choose, is also going to be a more viable option. You'll hear the industry term Green Button Connect used a lot, and that's really just a signal for pieces of software or services that have already taken the steps to integrate with that standard. Now we just have to mention as a possible negative or at least worth noting that most of this data is going to be day behind at best. It's 15-minute or interval, but generally if you compare it to some sort of real time system you are most likely going to be dealing with somewhere between 24 to 48-hour delay in receiving this data in most cases. We also just have to mention that, in general, overall adoption of Green Button has been slower than most people hoped.Utility Proprietary System
An alternative here would be that prior to 2012 a lot of the larger utilities have a legacy system or proprietary system that they either built in total or in some part on their own as a platform to provide their customers, both commercial and possibly residential, with smart metering information. A good example of this would be San Diego Gas and Electric. They actually offer both, a Green Button avenue as well as continuing to support a legacy system. I believe their internal system is known as Quick View. Advantages to a proprietary system as opposed to Green Button are:- Sometimes the data sets are a little more free for you to customize or adjust.
- There may be things like recorders at the monthly level and maybe you can get closer to the real time into the spectrum than you could on the Green Button.
- There's usually a reporting tool set that sits on top, and there may be an extended set of data points when compared to Green Button.
It's not unusual, frankly, to have customers that have a choice with certain utilities of using Green Button or the legacy system. A lot of times the better system may look to be going with the legacy system in place as opposed to adopting Green Button. The biggest weakness that has to be mentioned here is the price barrier. Most of the non-Green Button data sources do have a subscription fee associated, and we also need to mention that right now there's a certain, a significant amount I would say, of pricing inconsistency in the market. This is a service that really the industry just hasn't made up its mind about. There are orders of magnitude differences between some utilities and others. It is generally priced by meter, but you'll see examples out there of utilities and providers that think this is somewhere south of $10 per meter per month, and you will see examples of service providers that think it's somewhere north of $100 per meter per month. Very similar to this example is just a slight tweak which are utilities as a variation on this theme that have just decided to purchase commercially available packaging as their platform and use that to provide the advanced metering information to their customers. Here, again, you're more likely to have control over setting more custom interval ranges and trying to get closer to real time into the spectrum. The fact that these services are sold to utilities produces a number of advantages:
- Having been productized, it's more likely that they've been well tested and vetted in the market.
- They're sold to different utilities; therefore, they are cross-regional, they're cross-utility, and so that means on their own they've created a bit of a standard.
- There's likely going to be reporting tool sets to sit on top of this more so than there would be for Green Button.
Here, however, the pricing barrier and all the same concerns are going to exist as well.