Scraping energy prices

Is your energy provider not giving easy access to energy prices? Costly, difficult or impossible to get access to an API? Tired of checking web sites for updated information, or missing out on changes silently published? Change Detection might be your free, self hosted solution.

My gas meter gives a dry contact impulse for every 10 litres I use. With the help of an esp32 I can transfer this information to Home Assistant. My Natural Gas provider does not have an API that gives access to updated prices (consumption and subscription), but does give access to this information on a web page I must sign in to. I’m using Change Detection to read and publish this data over MQTT just after midnight each day. I also get access to 2 days old consumption data.

Techno Tim has made a good video about ChangeDetection:

This is how I use it:

Run stack in Docker

version: "3.9"
services:
  changedetection:
    container_name: ChangeDetection
    image: ghcr.io/dgtlmoon/changedetection.io:latest
    hostname: changedetection
    mem_limit: 4g
    cpu_shares: 768
    security_opt:
      - no-new-privileges:true
    restart: unless-stopped
    ports:
      - 5054:5000
    volumes:
      - /docker/changedetection:/datastore:rw
    environment:
     - PLAYWRIGHT_DRIVER_URL=ws://playwright-chrome:3000/?stealth=1&--disable-web-security=true&blockAds=true
    depends_on:
      playwright-chrome:
        condition: service_started
  playwright-chrome:
    hostname: playwright-chrome
    container_name: playwright-chrome
    image: ghcr.io/browserless/chromium:latest
    restart: unless-stopped

Use “Browser steps” to navigate to the web-page you wish to scrape. Use “Visual Filter Selector” to indicate data you wish to observe for changes. Use “Notifications” to publish changes. For me it looks like this:

Filters and triggers:

/html/body/app-root/app-navbar/mat-sidenav-container/mat-sidenav-content/app-pce-detail/div/div/div[1]/div[1]/div/div[2]/div/span[2]
/html/body/app-root/app-navbar/mat-sidenav-container/mat-sidenav-content/app-pce-detail/div/div/div[1]/div[1]/div/div[1]/div/span[2]
/html/body/app-root/app-navbar/mat-sidenav-container/mat-sidenav-content/app-pce-detail/div/div/div[2]/div[2]/mat-tab-group/div/mat-tab-body[1]/div/app-marker-price/div[3]/div/span[1]/span[2]
/html/body/app-root/app-navbar/mat-sidenav-container/mat-sidenav-content/app-pce-detail/div/div/div[2]/div[2]/mat-tab-group/div/mat-tab-body[1]/div/app-marker-price/div[3]/div/span[2]/span[2]

Notifications:

Notification URL List:

mqtt://mqtt-user:[email protected]:1883/gascounter/changed/detection?retain=yes

Notification Title:

data

Notification Body:

{{current_snapshot}}

In MQTT Explorer:

This data is from January, hence the identical monthly and yearly values.

MQTT sensors:

- name: "Yearly gas consumption"
  unique_id: "d5229283-9a68-4831-b137-aa985a27005f"
  state_topic: "gascounter/changed/detection"
  value_template: "{{ value.split('data')[1][2:252].split(' kWh')[0][:252]| replace(',', '.',1)| replace(' ', '',1) }}"
  device_class: energy
  unit_of_measurement: "kWh"
  state_class: measurement
  qos: 0
  force_update: true

- name: "Monthly gas consumption"
  unique_id: "2a8d90af-3154-4c54-af78-03fa37be9275"
  state_topic: "gascounter/changed/detection"
  value_template: "{{ value.split('data')[1][2:252].split(' kWh')[1]| replace(',', '.',1)| replace(' ', '') }}"
  device_class: energy
  unit_of_measurement: "kWh"
  state_class: measurement
  qos: 0
  force_update: true

- name: "Gas price"
  unique_id: "fb0a5710-2536-4b49-958b-7e5111817daf"
  state_topic: "gascounter/changed/detection"
  value_template: "{{ value.split('data')[1][2:252].split('kWh ')[2].split(' ')[0]| replace(',', '.',1)| replace(' ', '',1)| float }}"
  unit_of_measurement: "€/kWh"
  state_class: measurement
  qos: 0
  force_update: true

- name: "Subscription price gas"
  unique_id: "b9587edb-4e81-4192-b0fd-8723818cde33"
  state_topic: "gascounter/changed/detection"
  value_template: "{{ value.split('data')[1][2:252].split('kWh ')[2].split(' ')[1]| replace(',', '.',1)| replace(' ', '',1)| float }}"
  unit_of_measurement: "€/an"
  state_class: measurement
  qos: 0
  force_update: true

Please let me know if you are using ChangeDetection for other things.