S-CASE Blog | S-CASE & ETICS: Infrastructure Pilot Application

By using S-CASE tools, Engineering will develop ETICS Build and Test Environment Manager (BTE Manager).

ETICS BTE is one of the three pilot applications, foreseen by the project, which will help partners validate and evaluate the S-CASE paradigm. ETICS (etics.res.eng.it) is a system which automates and improves the execution of builds and tests and verifies the quality of the software produced. It is especially conceived for distributed, multi-language and multi-platform software.

A build/test lifecycle cannot exclude the management of the environment where builds and tests are performed. In ETICS, at the moment, this is done manually by the developer or tester. The BTE manager, developed by using S-CASE, will be able to create/modify/delete, accordingly to users’ needs, the Virtual Machines of a certain Virtual Infrastructure. It will also optimize the resource consumption by reusing inactive virtual machines for specific build sessions and by quickly disposing useless machines. User’s will no longer need to manually create the needed virtual machines, profile them (adding, for example, Application Servers, Containers etc), tune the environment (e.g. managing the IP addresses, DHCP Servers etc) and, after the development or tests, dispose the whole environment.

Introduction & Overview to ETICS

ETICS (E-infrastructure for Testing, Integration and Configuration of Software) is a system which automates and improves the execution of builds and tests. It is especially conceived for distributed, multi-language and multi-platform software and provides meaningful measurements of the overall software quality. ETICS consists of a build and test execution system, offered as web-services, able to exploit distributed computational resources and a plug-in mechanism for integrating engineering tools, to design, maintain and monitor build and multi-node testing scenarios.

A complete build/test lifecycle cannot exclude the management of the environment where builds and tests are performed. Currently ETICS don’t bother of this aspect, which is manually performed by the developer or tester. In particular he/she has to create the needed virtual machines, profile them (adding, for example, Application Servers, Containers etc), tune the environment (e.g. managing the IP addresses, DHCP Servers etc) and, after the development or tests, dispose the whole environment.

The automation and the optimization of that process would be a very good improvement for Engineering’s development process. For this purpose a good solution may be a web service that receives as input the features of the requested environment and creates/modifies/deletes according the Virtual Machines of a certain Virtual Infrastructure. It may optimize the resource consumption by reusing inactive
virtual machines for specific build sessions and quickly dispose useless machines.

BTE Manager defines the needed VMs on the basis of the request received from ETICS
and of the current status of the requested environment (e.g. is it already present? Do we need new Vms? Do we need to destroy some of the VMs?). In some cases it tries to optimize the resource consumption by reusing virtual machines belonging to different environments. Obviously such an optimisation is not always appropriate: for example a test environment, where performance evaluation is probably requested, cannot be built with machines shared with other experiments.

We can distinguish two main flows: one for creation and management of virtual environments and one for disposal. Each flow in turn is divided into flows related to the specific environment: build or test.

For example let’s consider the flow concerning the creation of a virtual build
environment:

1. ETICS send the request that is received by the RESTful interfaces
2. The request is a creation
3. The requested environment is build
4. The module checks if the same virtual environment exists: in that case the requested modifications will be evaluated, otherwise the request will be considered a simple creation
5. The module evaluates if there is any active VM belonging to other build environments, that can be used also for this build activity. For example a machine containing a web server may also host a DBMS
6. The exact number and features of needed VMs is evaluated, the appropriate commands are generated and sent to the target Infrastructure
7. The response is parsed and a final response is sent to Etics.

The steps requested by the other use cases are very similar: the only remark is that the creation or disposal of a test environment does not include the re-use (or the preservation) of VMs requested in other operations.

Build and Test Environment Manager

ETICS Build and Test Environment Manager will be built by S-Case. It will expose some web services to interact with ETICS and will use some web services to interact with the Virtual Infrastructures.

The exposed web service should take as input the features of the requested
infrastructure. At high level it should provide the following methods:

• Create test environment
• Create build environment
• Modify test environment
• Modify build environment
• Dispose test environment
• Dispose build environment

Every build or test environment should be identified by an ID related with the name of the ETICS project: every management operation on a certain environment can be transparently performed by ETICS using the ID and a set of parameters

The central part of the activity, concerning the BT Environment Manager, describes the logical steps needed to generate the final request for the infrastructure: these functionalities are provided by the Core. In particular it decides if:

• The build or test environment requested exists
• Some VM have to be added or removed
• Some VM can be reused to optimize the resource consumption

These operations may involve matchmaking algorithms and produce a request for a specific web service managing a specific Virtual Environment (for example, Windows Azure).

The web client sends the request, that can be a SOAP or REST request with the operation that the Virtual Environment Manager must perform to create, delete or adapt the Environment for the operation requested.

Requirements

Functionalities

• Must translate the requests from ETICS in commands for the Virtual Infrastructure
◦ This could be accomplished by using a local database
◦ As an alternative by talking with the Infrastructure
• Must know the software installed on each Virtual Machine
• Must be able to create/delete/modify with granularity from environment to single machines Interfaces
• Must expose RESTFul API by which ETICS will be able to request the environments
• It should be possible to obtain information on active environments through RESTFul API
• Must interact with Infrastructure Management API of a set of infrastructures

Third party Services

The Module to be produced will be able to interact with different Virtual Infrastructures. Actually CLOE, Engineering Virtual Infrastructure, will be the most important one, but, since in several use cases Engineering developers need to use build and test environments on other Virtual Infrastructures, it will be very useful to support the most widespread.

A first list is the following:

• Amazon
• OpenStack
• Microsoft Azure.

These Virtual Infrastructures (IaaS) exposes APIs enabling to manage their Virtual Machines. The management of the software on the VMs is more complex and, in general, depends on the VMs.

• Must keep track of the features of the generated environments

A complete software to manage Virtual Infrastructures and to profile Virtual Machines is Foreman (http://projects.theforeman.org/): it exposes a complete set of RESTful APIs to interact with several Virtual Infrastructures (including those in the list) and to profile
the machines.

Foreman’s RESTful API should be added to the list above in order to have a complete and flexible approach to manage several Virtual Infrastructures.