Building the reference systems

tl;dr Create our own testbed to measure the power consumption for running the test suite of common open source libraries. The testbed is CI/CD pipeline in which the Runners are on a Kubernetes cluster dedicated server (for now) that has access to the environmental data agent API or is monitored by a tool such as Scaphandre so that we can access or track the underlying power consumption. Each test run should report on power consumption so it can be further analyzed by the research team. The configuration of each test-run/test-subject will also be conducted by the research team.

Roles

• The researcher is configuring the various open-source libraries as test cases that can be run on the test infrastructure. The aim of the researcher is to review the different research subjects in regards to their power consumption. For the researcher, being able to get the data from each test run is the most important so it can be analyzed and further visualized.
• The infrastructure operator is the person responsible for configuring and running the CI/CD infrastructure, most likely within a Kubernetes cluster. This person wants to have it easy to expose the power consumption to the test environment for researchers and developers to use. It should be easy to set up, easy to monitor and to identify inaccuracies in the actually reported power consumption.
• The developer is the person building or maintaining an open-source library. For them, it should be very visible which increase or decrease in power consumption each new test execution is creating so that they can determine if their code is becoming more or less efficient. The resulting metrics should be reliable, consistent, and investigatable by the developer. Historic analysis from test-run to test-run should also be easy to do, so that the developer may understand the correlation between code changes and changes in power consumption.

Definition

• Test infrastructure: What is being defined and built within the scope of this project is a test infrastructure for open source libraries. This can be Jenkins or GitLab to give an example.
• Test run: The execution of a test suite within the test infrastructure
• Open-source library: As part of the project we have selected a few open source libraries that we would like to analyze in regards to their power consumption, examples can be Drupal or React
• Test suite: An integration or unit test suite, is a grouping of many tests in one suite, often with the aim to test the entire functionality of the library in one test run

User stories

As a researcher

• I want to be able to take an existing open-source library, with the unit or integration tests and easily configure a test run for it on the test infrastructure
• Optional: I want to be able to configure test runs without having to create a dedicated repository, without coding my own test setup
• I want to be able to trigger test runs manually to test the power consumption of a specific open-source library
• I want to be able to trigger a period or batch-based test run (e.g. running the same test for 24 hours 500 times)
• I want to be able to download the power consumption metrics & measurements for each test run in a computer-readable format (JSON, CSV, ...)
• I want to be able to see the power consumption metrics for each test within a test suite in Watt/hours
• I want to be able to see the total power consumption in Watt/hours for the entire test run

As an infrastructure operator

<aside> 👉 In the first iteration, this will not be needed, we will only run on our own infrastructure, however, as the project progresses, we do want to enable the owners and maintainers of open source libraries to install the ability to measure power consumption into their own test environments (Gitlab, Github, ...)

</aside>

• I want to be able to install the power consumption measurement components (’plugins’) into my infrastructure with a few lines of commands
  • Gitlab Plugin
  • Kubernetes ‘Sidecar’
  • Github Actions
  • Jenkins Plugin
• I want to be able to still measure the power consumption even if the underlying or required APIs are not available in my infrastructure
  • e.g. if the environmental data agent can not collect either data from the operating system (e.g. using RAPL) and there is no power consumption data exposed within the underlying virtualized infrastructure/data center, we will work with some constants/hardware specific power consumption data instead
• I want the additional measuring infrastructure to have a minimal additional resource footprint in my infrastructure

As a developer

<aside> 👉 In the first iteration, it will only be the researcher role that has access to the infrastructure, developers will only be let on the test infrastructure later

</aside>

• I want to be able to see the power consumption of each test run of my code or changes to my code
• I want to be able to compare the power consumption of the latest test run with previous test runs
• I want to be able to see the power consumption per test in my test suite
• Optional: I want to receive an alert when there are significant changes in power consumption from one test run to another (e.g. 10% increase)

Resolver of power consumption

see Github

Resource allocation solver

see Github

Initial ideas

Messmethode Mit welcher Methode/Vorgehensweise und mit welchen Tools messen wir? (bspw. pyRAPL für Python Code oder Birkenfeld Methode für IDEs) Methodik für die IDEs: Birkenfeld-Methode Methodik für die Software-Bibliotheken: Wo gehen wir messen: Compiler, ...

Absolute Zahl per kWh - Indikator Total tests: 14164 (14160 Passed, 4 Failed, 0 Others) Failed tests: 4 (4 New, 0 Existing) Pass percentage: 99.97% Run duration: 30m 14s Tests not reported: 0 Library review: Was gibt es alles an tools, die versuchen solche Messungen zu machen (GitHub)

Entweder es ist VERGLEICHBAR oder WIEDERHOLBAR aber nicht beides. (z.B. Bild/Text Bibliotheken müssten alle die gleiche Funktion Ausführen für Vergleichbarkeit)

Messaufbau Infrastruktur von Green Lab nutzen? Oder eigene Racks dahin bringen Wiederholbarkeit: Tests durchführen, was drum herum bauen? Non Stop Messung

Beispiel Messaufbau Starten der Software Messung der Energiekennzahlen über die DCIM Schnittstelle von der Hardware Messung der Energiekennzahlen über die Strommesser (externe Validierungsdaten für die DCIM Daten) Erfassen von Laufzeitinformationen (Auslastung CPU, Speicher, etc.) Stoppen der Software

Ergebnis: Ein wiederholbarer Messaufbau, der eine reibungslose Messung gewährleisten