S-CASE | Meet The Project – DELPHIS

We caught up with project partners DELPHIS to talk about their role in the S-CASE project and how they will validate the concept through their use case. Watch this short film to find out more.

When asked about her project involvement Marina Stamatiadou, Software Engineer at DELPHIS R&D department, told us:

“I always loved software engineering so I was inspired when I heard that this has to do with requirements and design and development, testing and all this stuff that has to do with software engineering.”

For Thanos Daskalopoulos, Chief of Product and Technology Development, the pilot will bring them a step closer to realising a crucial commercial goal:

“This is taking us closer to a data analytics as a service model, where different customer needs and discard data sets can be analysed as customised as needed.”

“The pilot involves our vision of creating a PaaS and will actually take our API’s one level down from today, inside WISE. That means that our partner and customer developers will be able to create their own orchestrations instead of only using the ones that WISE exposes.”

For more information on the project and the partners involved then take a look at this recent blog post from DELPHIS.

Familiar Bedfellows | The Automation engine, Model Driven Engineering and Victorian buildings (Part 2)

In this blog post, (you can read part one here), we will move back to S-CASE and the way MDE is utilised so as automated construction of RESTful Services is achieved.

In the S-CASE MDE engine, we encounter all the four discrete software construction stages that the 19th century civil engineer had to go through in order to build a Victorian house. These four phases of MDE have been already presented in part one. Only now, the S-CASE MDE engine does not build Victorian houses, but software of a particular architectural style.

The style that S-CASE follows for the web services it produces, is the REST architectural style (instead of the Victorian Style), enhanced with some design patterns and concepts. But what is REST anyway? It is a software architectural style for the web, which Roy Fielding introduced in 2000. In the same way the Victorian style has some characteristics like steep roofs, sash windows etc. that differentiate it from others, REST has some distinct properties as well (figure 1):

scasepic1

Figure 1: Image courtesy of http://martinfowler.com/

  1. Every RESTful web service (house) comprises only of resources (rooms)
  2. Each such resource has a unique URI (address) at which it is accessible.
  3. Every such URI is accessed with the common HTTP verbs. In contrast to other styles though, the HTTP verbs are used in the intended way. In any case, the POST HTTP verb is used to create a new resource, the GET verb is used to retrieve an existing resource, PUT is used to update an existing resource and DELETE to delete one.
  4. The resources are interconnected with hypermedia links.

The hypermedia links are probably the property that differentiates REST the most from other styles. But what are these links for? They are pretty much the same thing with the navigation arrows in an old-fashioned adventure video game similar to the one shown in figure 2. In this case, no matter wherever you are in the house (web service), you are always given the next possible actions. This means that you do not have to know them a priori, which in turn increases a lot the flexibility of the interaction between web services and clients. In this example, you may open any of the three doors, follow the corridor or go up the stairs. However, you absolutely have to take one of these specified actions! Thus, you may not stand in front of the painting on the wall to take a look! That is, should there exists a bedroom door upstairs, you may not open it from the position shown in figure 2. Instead, you first have to go up the stairs following the provided link and then open the door by using the link that you will be given once you go up the stairs.

Furthermore, the S-CASE’s MDE engine embeds to the REST style some more design patterns. One may think of them as additional decorations in every room other than those already introduced by the Victorian style. For example Model View Controller (MVC) is used in order to further apply the divide and conquer as well as the separation of concerns design principles. Additionally, the repository pattern is used to offer a uniform way to store stuff. This is like every Victorian house also had a storekeeper. That way, whenever anyone living in the house wants to store something in the warehouse, does not have to know where to store it (or from where to retrieve it). He simply gives it to the storekeeper instead.

scasepic2

Figure 2: Image courtesy of www.geffrye-museum.org.uk

So, how would a conversation be between a modern times customer, who wants to buy an e-shop service, and an MDE engine that knows how to build RESTful web services? The customer would probably say to the MDE engine what that service (house), which he wants to buy, should have. In particular, he could ask for some specific resources (rooms), like user accounts, user profiles, product inventory, the capability to pay with credit cards, search the products database or even have email notifications enabled. Additionally, he would probably ask for some properties to be added to those resources. For instance, he may ask that the user profile should contain some text placeholder where the user can write a description of himself. Moreover, the user accounts should have a username, a password and an email. Lastly, the product inventory should have a price and a textual description for each product. Once the customer specifies his needs to the MDE engine, the e-shop web service CIM is formulated.

One should recall at this point that like in the case of Victorian houses, nothing that has to do with software architecture, design patterns or even programming languages and web frameworks is discussed yet. At this initial stage of MDE that S-CASE goes through, only concepts of RESTful web services domain are examined, like resources, their relations and/or some properties of them, period!

From this point on, the MDE engine of S-CASE begins to apply the REST architectural style as well as the extra design patterns to every single concept that its customer included in CIM. For instance, the MDE engine would add to all the resources specified in CIM a controller entity that will handle all the incoming web requests with some type of protocol. Then, it would assign to each one of them a unique URI and input/output representation handlers of some type. Moreover, it would design an RDBMS database schema of some type that is sufficient to store the envisioned e-shop’s data, which is going to be accessed in a uniform way by means of a repository controller. Once the MDE engine transforms every single CIM concept by applying to it the described architectural style, the envisioned e-shop’s PIM is formulated. It must be stressed though, that likewise the Victorian house engineer did not specify the exact make of the materials, the MDE engine does not specify any specific technologies to be used at this point either. Instead, it used an abstract generic form of each of them (some type of protocol, some type of database schema etc.).

Similarly to the Victorian house case, the next step is to pick specific technologies (materials) with which the service will be constructed. In the S-CASE MDE engine for example, one possible combination of technologies could be the JAXB framework to handle the parsing and marshaling of incoming/outgoing representations, the JAX-RS one to handle incoming web requests of the HTTP 1.1 protocol, the Hibernate one to handle RDBMS database schema of (for example) POSTGRESQL and the Java language as the language to code the envisioned service. Once MDE engine applies a specified technology to every PIM component, the e-shop PSM is formulated. Of course, there could be many other combinations of technologies that could be used to form different PSMs. Other customers might choose another one from the available ones! Similarly to the Victorian house case and in any case, the relationship between PIM and PSM is always one PIM to many possible PSMs.

At this point, the appropriate, for the selected PSM, code generators (specialized builders of a Victorian house) write the actual software code. Once this is done, the process of building the e-shop RESTful service ends. Of course, like in the case of Victorian houses, some resources will end up to be fully implemented (fully built and decorated rooms), whilst some others will need some additions so as to be completed or there may be even some resources that are simple wrappers (empty rooms) interconnected with the other resources of the service. Such resources would require human developer intervention to get completed pretty much like the Victorian house garden would need a gardener to plant some flowers etc. to it.

Summing up, S-CASE utilizes MDE to automate the procedure of building RESTful web services. Once S-CASE parses the multi-modal requirements (UML diagrams, semi structured text etc.), it attempts to understand the needs of its “customer” in order to formulate the CIM. Later on, it applies the REST architectural style to transform the CIM to its according PIM. Then, according to user preferences, it transforms the PIM to one of the possible PSMs and finally, it initiates its code generator to produce the actual service code. The crucial point to stress one more time though, is that this process is an automated one. So the customer that “told his needs” to the MDE engine of S-CASE by providing his multi-modal input, would then get back the according RESTful service automatically and instantly.

This pretty much concludes the demonstration of the way the S-CASE’s MDE engine builds a RESTful service. At this point many possibilities arise. Which should be the acceptable concepts in CIM? What should be the enforced design of the PIM etc. Such issues will be discussed in the next entries related to meta-model engineering and automation trade-offs within the S-CASE MDE engine.

Read part one of this series.

S-CASE Blog | S-CASE Development fuel booster

We have just finished hosting the 4th meeting of S-CASE here in Athens and feel excited to have a pilot contributing to our big and exciting development goals.

Not only that, the project and consortium itself has been a tank of inspiration, feeding us with new ideas and food for thought. As the first year comes soon to a wrap and after a lot of creative preparations, our pilot has now a clear positioning and connection to the Watchtower API’s offering, which will be soon available for use from our clients. The API’s environment is designed in a way that the Watchtower platform becomes the backend analytics for the customers, who need automated intelligence in their systems without having to develop their own analytics infrastructure. This way they go fast, low cost and with reliability, since the services consumed have been already tested and in production.

The vision of S-CASE is to provide tools for developers, along with the underpinning technologies that will support the insertion of rough system requirements in a variety of structured, semi-structured or unstructured formats for seamlessly generating draft software prototypes that will form the basis for complete software development.

S-CASE is seen as a rapid prototyping realm aiming at providing automated solutions for (a) the extraction of system specifications and low-level architecture, and (b) the discovery and synthesis of composite workflows of software artefacts from distributed open source and proprietary resources that fulfil the inserted system requirements in the best possible way.

Through the innovations it introduces, S-CASE is expected to have a significant impact on the reduction of the time that is required between the conceptualisation of a software system and its first prototype, thus improving the SE process in terms of development costs.

The S-CASE pilot – Unleashing the power of WISE

 

The power of the Watchtower lies to a big extend in our analytics engine called WISE (Watchtower’s Intelligent System Engine). Inside WISE lives all the automatic orchestration that takes of the shoulders of energy management teams the “discovery” part and lets them focus on their operations. The pilot will actually take our API’s one level down from today, inside the WISE. That means that our partner and customer developers will be able to create their own orchestrations instead of only using the ones that WISE exposes. This is taking us closer to a data analytics as a service model, where different customer needs and discard data sets can be analysed as customised as needed.

The Watchtower SAAS technology stack is an MVC architecture using javascript from end to end (MySQL, Express.js, Angular.js, Node.js) with the combination of WISE core functionality developed in Java. Every rule of the WISE core is being exposed with RESTful web services, creating an ecosystem of intelligence that depending on the functionality needs, separate orchestrations of the web services are being composed. The S-CASE platform like a fuel booster, will rapidly improve our productivity in developing these new web services and the development of new intelligent orchestrations inside WISE.

S-CASE Blog | Natural Language Processing

We recently got another research paper on our work in S-CASE accepted  at a conference on natural language processing. The accepted paper describes our efforts on improving a parsing model that can  automatically map software requirements written in natural language to formal representations based on semantic roles.

State-of-the-art semantic role labelling systems require large  annotated corpora to achieve full performance. Unfortunately, such  corpora are expensive to produce and often do not generalise well  across domains. Even in domain, errors are often made where syntactic  information does not provide sufficient cues. In this paper, we mitigate both of these problems by employing distributional word representations gathered from unlabelled data. The rationale for this  approach lies in the so-called distributional hypothesis by Zellig Harris, which states that words that occur in the same contexts tend  to have similar meanings.

While straight-forward word representations  of predicates and arguments have already been shown to be useful for  semantic analysis tasks, we show that further gains can be achieved by composing representations that model the interaction between predicate  and argument, and capture full argument spans.

S-CASE Blog | S-CASE & ETICS: Infrastructure Pilot Application

By using S-CASE tools, Engineering will develop ETICS Build and Test Environment Manager (BTE Manager).

ETICS BTE is one of the three pilot applications, foreseen by the project, which will help partners validate and evaluate the S-CASE paradigm. ETICS (etics.res.eng.it) is a system which automates and improves the execution of builds and tests and verifies the quality of the software produced. It is especially conceived for distributed, multi-language and multi-platform software.

A build/test lifecycle cannot exclude the management of the environment where builds and tests are performed. In ETICS, at the moment, this is done manually by the developer or tester. The BTE manager, developed by using S-CASE, will be able to create/modify/delete, accordingly to users’ needs, the Virtual Machines of a certain Virtual Infrastructure. It will also optimize the resource consumption by reusing inactive virtual machines for specific build sessions and by quickly disposing useless machines. User’s will no longer need to manually create the needed virtual machines, profile them (adding, for example, Application Servers, Containers etc), tune the environment (e.g. managing the IP addresses, DHCP Servers etc) and, after the development or tests, dispose the whole environment.

Introduction & Overview to ETICS

ETICS (E-infrastructure for Testing, Integration and Configuration of Software) is a system which automates and improves the execution of builds and tests. It is especially conceived for distributed, multi-language and multi-platform software and provides meaningful measurements of the overall software quality. ETICS consists of a build and test execution system, offered as web-services, able to exploit distributed computational resources and a plug-in mechanism for integrating engineering tools, to design, maintain and monitor build and multi-node testing scenarios.

A complete build/test lifecycle cannot exclude the management of the environment where builds and tests are performed. Currently ETICS don’t bother of this aspect, which is manually performed by the developer or tester. In particular he/she has to create the needed virtual machines, profile them (adding, for example, Application Servers, Containers etc), tune the environment (e.g. managing the IP addresses, DHCP Servers etc) and, after the development or tests, dispose the whole environment.

The automation and the optimization of that process would be a very good improvement for Engineering’s development process. For this purpose a good solution may be a web service that receives as input the features of the requested environment and creates/modifies/deletes according the Virtual Machines of a certain Virtual Infrastructure. It may optimize the resource consumption by reusing inactive
virtual machines for specific build sessions and quickly dispose useless machines.

BTE Manager defines the needed VMs on the basis of the request received from ETICS
and of the current status of the requested environment (e.g. is it already present? Do we need new Vms? Do we need to destroy some of the VMs?). In some cases it tries to optimize the resource consumption by reusing virtual machines belonging to different environments. Obviously such an optimisation is not always appropriate: for example a test environment, where performance evaluation is probably requested, cannot be built with machines shared with other experiments.

We can distinguish two main flows: one for creation and management of virtual environments and one for disposal. Each flow in turn is divided into flows related to the specific environment: build or test.

For example let’s consider the flow concerning the creation of a virtual build
environment:

1. ETICS send the request that is received by the RESTful interfaces
2. The request is a creation
3. The requested environment is build
4. The module checks if the same virtual environment exists: in that case the requested modifications will be evaluated, otherwise the request will be considered a simple creation
5. The module evaluates if there is any active VM belonging to other build environments, that can be used also for this build activity. For example a machine containing a web server may also host a DBMS
6. The exact number and features of needed VMs is evaluated, the appropriate commands are generated and sent to the target Infrastructure
7. The response is parsed and a final response is sent to Etics.

The steps requested by the other use cases are very similar: the only remark is that the creation or disposal of a test environment does not include the re-use (or the preservation) of VMs requested in other operations.

Build and Test Environment Manager

ETICS Build and Test Environment Manager will be built by S-Case. It will expose some web services to interact with ETICS and will use some web services to interact with the Virtual Infrastructures.

The exposed web service should take as input the features of the requested
infrastructure. At high level it should provide the following methods:

• Create test environment
• Create build environment
• Modify test environment
• Modify build environment
• Dispose test environment
• Dispose build environment

Every build or test environment should be identified by an ID related with the name of the ETICS project: every management operation on a certain environment can be transparently performed by ETICS using the ID and a set of parameters

The central part of the activity, concerning the BT Environment Manager, describes the logical steps needed to generate the final request for the infrastructure: these functionalities are provided by the Core. In particular it decides if:

• The build or test environment requested exists
• Some VM have to be added or removed
• Some VM can be reused to optimize the resource consumption

These operations may involve matchmaking algorithms and produce a request for a specific web service managing a specific Virtual Environment (for example, Windows Azure).

The web client sends the request, that can be a SOAP or REST request with the operation that the Virtual Environment Manager must perform to create, delete or adapt the Environment for the operation requested.

Requirements

Functionalities

• Must translate the requests from ETICS in commands for the Virtual Infrastructure
◦ This could be accomplished by using a local database
◦ As an alternative by talking with the Infrastructure
• Must know the software installed on each Virtual Machine
• Must be able to create/delete/modify with granularity from environment to single machines Interfaces
• Must expose RESTFul API by which ETICS will be able to request the environments
• It should be possible to obtain information on active environments through RESTFul API
• Must interact with Infrastructure Management API of a set of infrastructures

Third party Services

The Module to be produced will be able to interact with different Virtual Infrastructures. Actually CLOE, Engineering Virtual Infrastructure, will be the most important one, but, since in several use cases Engineering developers need to use build and test environments on other Virtual Infrastructures, it will be very useful to support the most widespread.

A first list is the following:

• Amazon
• OpenStack
• Microsoft Azure.

These Virtual Infrastructures (IaaS) exposes APIs enabling to manage their Virtual Machines. The management of the software on the VMs is more complex and, in general, depends on the VMs.

• Must keep track of the features of the generated environments

A complete software to manage Virtual Infrastructures and to profile Virtual Machines is Foreman (http://projects.theforeman.org/): it exposes a complete set of RESTful APIs to interact with several Virtual Infrastructures (including those in the list) and to profile
the machines.

Foreman’s RESTful API should be added to the list above in order to have a complete and flexible approach to manage several Virtual Infrastructures.