Activity 1.1: The hybrid robotic leg was designed as part of this activity. The main point of the desing was to ensure that the leg is able to pass over small vertical obstacles. The mechanical components for implementing the design were selected and presented. Drivers and control interfaces were also selected. The linear actuator for the z direction of the movement will be MXE-P16 from Tolomatic. This is a balls-screw type of linear actuator. An OpenTorque Actuator will be used for the rotative motion. This is based on a design by MIT. The ROS operationg system will be used for the whole platform.

Activity 1.2: Trajectory design algorithms, based on model inversion which exploits flat representations and B-spline parametrizations, have been extended towards NURBS representations which allow additional degrees of freedom. The analysis has been done from the perspective of keeping a manageable complexity (e.g., through knot refinement methods) while simultaneously respecting position, velocity and heading constraints. Existing algorithms, including these used in generating the previous illustrations are stored in the Gitlab project.

Activity 1.3: The 3D scenes necessary for the creation of the versatile robotic platform for navigation inside crowded and obstacle environments are materialized through the 3D digital model, whose realization must be done with high precision. The objects of 3D modeling are the elements that delimit the respective space (walls), those with a fixed position (fireplace, kitchen furniture, ...) and mobile objects (tables, chairs, sofas, ...). The usual technologies for acquiring the graphic information necessary to make the 3D model are represented by the technology with RGB images, that of the execution of direct measurements (GNSS measurements, total station, laser scanner, LIDAR technology). The stages of creating the 3D scenes are the inventory of the location and the realization of the sketch, the performance of the measurements, the realization of the digital survey of the location and the realization of the 3D model with CAD / MASH functions.

Activity 1.4: The existing database for the Tiago service robot, which previously contained 1380 audio signals, was expanded by another 1920 sounds. All signals were recorded stereo, with an accuracy of 16 bits and a sampling frequency of 48 kHz. The audio data stream has been recorded so that it contains only a certain class of sound event, which belongs to the indoor environment. Currently, the database contains 3300 audio signals (110 classes x 30 signals / class).

Activity 1.5: We have prepared the webpage of the project for wide-scale dissemination.

Activity 1.6: The conceptual models for the development and validation of the business plan were investigated within this activity. We have considered both models which involve the end-users and customers directly and models which are based on statistical data collected via the web by using specific tools. Two canvas models were also presented and compared.

Activity 1.7: A preliminary search in the European Patent Database was performed in order to identify oposable patents. Relevant patents were selected and summarized in this deliverable.

Activity 2.1: Within this activity we have developed the mechanical components and have integrated the mechanical and electrical components of the platform. The mechanical assembly (integrated with the electrical and control/supervision architecture) has been completed. It consists of 4 dynamic sub-assemblies each consisting of a rubberized wheel and an omnidirectional wheel (each with its own motor). This solution made it possible to ensure the horizontality of the platform when overcoming obstacles as well as to switch to omnidirectional travel by lifting the single rubberised wheels.

Activity 2.2: Within this activity we have developed the software needed to connect the new hardware to the ROS. We have perfomed several simulations using dedicated frameworks. In order to decouple the hardware component of the project (omnidirectional robot) from the algorithmic one (planning and obstacle avoidance), the only solution is to build a model of the robot in simulation, which can be used to validate and improve the routes obtained.

Activity 2.3: Improving sound analysis and recognition methods (Partial). MFCC, LPC, LPCC and MPEG-7 features were tested using different values for the extracted coefficients (10:2:38, 64). Several classifiers were also tested: NuSVC, SVM, kNN, GNB, DT, MLP, SGD, LDA and RF. All possible combinations given the number of extracted coefficients, the features used and the classifiers were tested. The conclusion reached is that the best results for audio signal identification are obtained using 34 and 64 MFCC features respectively, in combination with kNN, SVM and LDA respectively.

Activity 2.4: We have developed a library that implements a minimal set of movement. Using parameterizations with spline functions (B-spline or NURBS) primitives for movement in an environment with obstacles were implemented in the MOVE-LIB and MOVE-PATH libraries. Predefined polyhedral shapes, STL-modelled closed environment, points of interest to be reached at pre-specified moments of time were considered.

Activity 2.5: The phase of modelling the test spaces, the robotic platform, consisted in continuing to produce 3D models for Hall 2, lecture hall A0 and exporting these models to formats compatible with robotic simulators such as Gazebo. After modelling, the files, in dwg format, were exported to formats compatible with the test platforms. The obtained files were converted to the .dae format supported by the Gazebo simulator in order to import them into this simulator for testing using an existing robot model.

Activity 2.6: The Min-PC integrated in the robotic platform is the Beelink SEI8 model which has an 8th generation Intel I5-8259U Coffee Lake processor, an Intel UHD (Intel IRIS Plus) embedded graphics processing unit, 16GB DDR4 RAM at 2400MHz and 512GB SSD storage. Network connectivity is provided by a 1000Mbps Ethernet interface and WiFi 6. In order to port the algorithms to the new platform, Python software version 3.9.2 was installed beforehand. To create a virtual working environment and install the necessary libraries, the "venv" and "pip" executables were installed. To implement the algorithms for generating a trajectory using spline functions, the CasaDi library (https://web.casadi.org/) was used as well as Python/C++ (useful for the actual usage stage).

Activity 2.7: To simulate the scene created, it was imported into the Gazebo simulator. Various 3D furniture objects were inserted into the scene. In the first phase, tests were performed using an existing robot model - TurtleBot 3. This was imported into the created scene and simulations of different trajectories for the robot in the scene were made. In order to simulate the behaviour of the robot platform, ROS and auxiliary tools (Gazebo for visualisation, Blender for map processing) were integrated. The associated scripts are made in Python. The 3D model of the museum hall was integrated into the simulation using the Collada format which allowed the faithful rendering of the interior and exterior surfaces. The OMNI-Z platform model has been imported into the created environment.

Activity 2.8: 6 conference papers and 4 journal articles were published in this phase. The articles address issues of audio signal recognition, obstacle avoidance trajectories, visual obstacle identification, power supply, etc. The project page is also maintained in English and Romanian at: www.citst.ro/projects/omni-z.

Activity 2.9: In this phase the business plan started in phase 1 was continued by completing the Canvas model with OMNI-Z product specific notions and by identifying and presenting competitive robotic platforms. The business plan is uploaded as a separate document in the reporting platform.

Activity 2.10: In order to identify the patentable elements, a preliminary search of the U.S. Patent Office database was performed using the keywords used in the previous step for the European database searches. The search in both databases was continued with new terms which yielded the expected results. The translation into Romanian of the abstracts of the selected patents is presented in this activity.

Activity 3.1 - Development of minimal software for manual control of the platform using a joy-stick: During this stage, the functionality of the robot was presented, allowing it to be controlled via a Bluetooth phone app as a remote control or joy-stick. The architecture is based on three microcontrollers in a client configuration that perform the wheel speed regulation based on a reference received from a supervisory microcontroller that analyzes the data received via Bluetooth and manages the robot behavior.

Activity 3.2 - Motion planning in the 3D space: Motion planning in 3D space is based on the use of B-spline functions whose associated control points (weights) are defined in this space. The use of spline functions allows the definition of a reference trajectory that respects both position constraints (obstacles and regions to be avoided) and control constraints as well as cost minimizations (usually minimizing the trajectory energy). We have also analysed methods to reduce the computation time during the run (the "explicit MPC" method, exploiting the geometrical properties of the related control law).

Activity 3.3 - Improving sound analysis and recognition methods (Final): One of the modules that are part of the robotic platform is the audio module. It is used to identify acoustic events that may occur in an indoor environment and can make certain decisions based on them. Six classification models are used in conjunction to identify the acoustic event. The final decision is made based on the predominant result. For the six individual models, in the test phase, the accuracy of identifying the correct event is: 99.091%, 98.939%, 99.697%, 99.545%, 97.576% and 98.333% respectively. The final decision of acoustic event identification is made based on the predominant result increasing the individual accuracy.

Activity 3.4 - Integration of the sound detection and recognition module with a task generator: Depending on the type of event detected, OMNI-Z chooses the appropriate task. If the detected event corresponds to a normal action, such as moving a chair, opening or closing a door, asking for an ID, using the washing machine or the microwave, etc., the generated task is to send a notification email to the person responsible for the permanent remote monitoring of the home where the OMNI-Z is located. On the other hand, if the event detected is a potentially alarming one, such as a strong cough that indicates a potential choking of the person, a request for specific types of medication, etc., the generated task is to send an alert email both to the person who has to intervene in such cases and to the person who permanently monitors the home remotely.

Activity 3.5 - Validation under experimental conditions – DEMO: The platform has demonstrated the following capabilities (illustrated by experimentation during the "DEMO Iasi" related activities held on 15-16 September): it can enter/exit an elevator and cross door thresholds without restrictions; it is able to plan/route a trajectory efficiently and with safety guarantees. Modular structure allows it to integrate various locator modules; it can pass over floor unevenness.

Brief presentation of the OMNI-Z project: The OMNI-Z consortium comprises a SME and three outstanding Romanian universities which have collaborated to develop an economically accessible, omnidirectional, modular and expandable robotic platform suitable for indoor navigation. The platform capabilities for indoor navigation are ensured by its locomotion mechanism which allows it to maneuver in a confined and crowded space by being able to rotated on the spot, navigate between obstacles, or pass over barriers such as doorsteps, ramps, etc.

The OMNI-Z platform is intended to be used both as a stand-alone product and integrated with sensors and assistive devices. The target customers of such a product are institutions for the elderly, hospitals but, not far in the future, individual users. The OMNI-Z products which also including software and databases will be exploited both commercially and for scientific purposes to promote robotic development in Romania.

The images below show the OMNI-Z platform in the validation and testing environment.

Activity 3.6 - Dissemination of the project results.