The coordinating company aims to exploit the niche market represented by the elderly and the emergence of assistive robotic platforms. Noteworthy, the development of smart products, based on information and communication technology (ICT), for this category of the population is strongly supported by the European Active and Assistive Living Program.
Through this project, the coordinating company and associated partners aim to improve the performance of a robotic platform so that it can be suitable for use in AAL environments and integration with eHealth platforms and home automation, available at the industrial partner but also from commercial manufacturers.
The main result of the project is a robotic mobile platform capable of navigating without restriction in a building:
- can enter / leave an elevator;
- can cross door steps without restrictions;
- is able to plan a trajectory efficiently and with safety guarantees;
- the modular structure allows it to integrate various location modules (eg audio module);
- it can pass over uneven floor and over 1-2 steps.
D11 Hardware and Software Architecture Report (M11).
D12 The MOVE-LIB library contains routing loop routines; for generating and tracking a simple reference, for manual control of the movement and vertical positioning of the platform (M20).
D21 The MOVE-PATH library groups the motion planning algorithms and the related internal models (M20).
D22 The MOVE-SCENE library groups the algorithms for generating a synthetic 3D scene, pre-processing and modifying an existing map (M23).
D31 Extension of the sound base (M11).
D32 Sound event detection and recognition mode, compatible with the new hardware platform (M20).
D41 Report on system functionality in simulation (M17).
D42 Report on the functionality of the system under experimental conditions (M24).
D51 Project website detailing purpose, intermediate results, dissemination actions, team structure (M1).
D52 Management and dissemination report (M11).
D53 Management and dissemination report (M17).
D54 Management and dissemination report (M24).
D55 Business plan and patent application (M24).
Phase 1 (01.10.2020 – 31.12.2020)
Activity 1.1: The hybrid robotic leg was designed as part of this activity. The main point of the desing was to ensure that the leg is able to pass over small vertical obstacles. The mechanical components for implementing the design were selected and presented. Drivers and control interfaces were also selected. The linear actuator for the z direction of the movement will be MXE-P16 from Tolomatic. This is a balls-screw type of linear actuator. An OpenTorque Actuator will be used for the rotative motion. This is based on a design by MIT. The ROS operationg system will be used for the whole platform.
Activity 1.2: Trajectory design algorithms, based on model inversion which exploits flat representations and B-spline parametrizations, have been extended towards NURBS representations which allow additional degrees of freedom. The analysis has been done from the perspective of keeping a manageable complexity (e.g., through knot refinement methods) while simultaneously respecting position, velocity and heading constraints. Existing algorithms, including these used in generating the previous illustrations are stored in the Gitlab project.
Activity 1.3: The 3D scenes necessary for the creation of the versatile robotic platform for navigation inside crowded and obstacle environments are materialized through the 3D digital model, whose realization must be done with high precision. The objects of 3D modeling are the elements that delimit the respective space (walls), those with a fixed position (fireplace, kitchen furniture, ...) and mobile objects (tables, chairs, sofas, ...). The usual technologies for acquiring the graphic information necessary to make the 3D model are represented by the technology with RGB images, that of the execution of direct measurements (GNSS measurements, total station, laser scanner, LIDAR technology). The stages of creating the 3D scenes are the inventory of the location and the realization of the sketch, the performance of the measurements, the realization of the digital survey of the location and the realization of the 3D model with CAD / MASH functions.
Activity 1.4: The existing database for the Tiago service robot, which previously contained 1380 audio signals, was expanded by another 1920 sounds. All signals were recorded stereo, with an accuracy of 16 bits and a sampling frequency of 48 kHz. The audio data stream has been recorded so that it contains only a certain class of sound event, which belongs to the indoor environment. Currently, the database contains 3300 audio signals (110 classes x 30 signals / class).
Activity 1.5: We have prepared the webpage of the project for wide-scale dissemination.
Activity 1.6: The conceptual models for the development and validation of the business plan were investigated within this activity. We have considered both models which involve the end-users and customers directly and models which are based on statistical data collected via the web by using specific tools. Two canvas models were also presented and compared.
Activity 1.7: A preliminary search in the European Patent Database was performed in order to identify oposable patents. Relevant patents were selected and summarized in this deliverable.
Phase 2 (01.01.2021 – 31.12.2021)
Activity 2.1: Within this activity we have developed the mechanical components and have integrated the mechanical and electrical components of the platform. The mechanical assembly (integrated with the electrical and control/supervision architecture) has been completed. It consists of 4 dynamic sub-assemblies each consisting of a rubberized wheel and an omnidirectional wheel (each with its own motor). This solution made it possible to ensure the horizontality of the platform when overcoming obstacles as well as to switch to omnidirectional travel by lifting the single rubberised wheels.
Activity 2.2: Within this activity we have developed the software needed to connect the new hardware to the ROS. We have perfomed several simulations using dedicated frameworks. In order to decouple the hardware component of the project (omnidirectional robot) from the algorithmic one (planning and obstacle avoidance), the only solution is to build a model of the robot in simulation, which can be used to validate and improve the routes obtained.
Activity 2.3: Improving sound analysis and recognition methods (Partial). MFCC, LPC, LPCC and MPEG-7 features were tested using different values for the extracted coefficients (10:2:38, 64). Several classifiers were also tested: NuSVC, SVM, kNN, GNB, DT, MLP, SGD, LDA and RF. All possible combinations given the number of extracted coefficients, the features used and the classifiers were tested. The conclusion reached is that the best results for audio signal identification are obtained using 34 and 64 MFCC features respectively, in combination with kNN, SVM and LDA respectively.
Activity 2.4: We have developed a library that implements a minimal set of movement. Using parameterizations with spline functions (B-spline or NURBS) primitives for movement in an environment with obstacles were implemented in the MOVE-LIB and MOVE-PATH libraries. Predefined polyhedral shapes, STL-modelled closed environment, points of interest to be reached at pre-specified moments of time were considered.
Activity 2.5: The phase of modelling the test spaces, the robotic platform, consisted in continuing to produce 3D models for Hall 2, lecture hall A0 and exporting these models to formats compatible with robotic simulators such as Gazebo. After modelling, the files, in dwg format, were exported to formats compatible with the test platforms. The obtained files were converted to the .dae format supported by the Gazebo simulator in order to import them into this simulator for testing using an existing robot model.
Activity 2.6: The Min-PC integrated in the robotic platform is the Beelink SEI8 model which has an 8th generation Intel I5-8259U Coffee Lake processor, an Intel UHD (Intel IRIS Plus) embedded graphics processing unit, 16GB DDR4 RAM at 2400MHz and 512GB SSD storage. Network connectivity is provided by a 1000Mbps Ethernet interface and WiFi 6. In order to port the algorithms to the new platform, Python software version 3.9.2 was installed beforehand. To create a virtual working environment and install the necessary libraries, the "venv" and "pip" executables were installed. To implement the algorithms for generating a trajectory using spline functions, the CasaDi library (https://web.casadi.org/) was used as well as Python/C++ (useful for the actual usage stage).
Activity 2.7: To simulate the scene created, it was imported into the Gazebo simulator. Various 3D furniture objects were inserted into the scene. In the first phase, tests were performed using an existing robot model - TurtleBot 3. This was imported into the created scene and simulations of different trajectories for the robot in the scene were made. In order to simulate the behaviour of the robot platform, ROS and auxiliary tools (Gazebo for visualisation, Blender for map processing) were integrated. The associated scripts are made in Python. The 3D model of the museum hall was integrated into the simulation using the Collada format which allowed the faithful rendering of the interior and exterior surfaces. The OMNI-Z platform model has been imported into the created environment.
Activity 2.8: 6 conference papers and 4 journal articles were published in this phase. The articles address issues of audio signal recognition, obstacle avoidance trajectories, visual obstacle identification, power supply, etc. The project page is also maintained in English and Romanian at: www.citst.ro/projects/omni-z.
Activity 2.9: In this phase the business plan started in phase 1 was continued by completing the Canvas model with OMNI-Z product specific notions and by identifying and presenting competitive robotic platforms. The business plan is uploaded as a separate document in the reporting platform.
Activity 2.10: In order to identify the patentable elements, a preliminary search of the U.S. Patent Office database was performed using the keywords used in the previous step for the European database searches. The search in both databases was continued with new terms which yielded the expected results. The translation into Romanian of the abstracts of the selected patents is presented in this activity.
Phase 3 (01.01.2022 – 30.09.2022)
Activity 3.1 - Development of minimal software for manual control of the platform using a joy-stick: During this stage, the functionality of the robot was presented, allowing it to be controlled via a Bluetooth phone app as a remote control or joy-stick. The architecture is based on three microcontrollers in a client configuration that perform the wheel speed regulation based on a reference received from a supervisory microcontroller that analyzes the data received via Bluetooth and manages the robot behavior.
Activity 3.2 - Motion planning in the 3D space: Motion planning in 3D space is based on the use of B-spline functions whose associated control points (weights) are defined in this space. The use of spline functions allows the definition of a reference trajectory that respects both position constraints (obstacles and regions to be avoided) and control constraints as well as cost minimizations (usually minimizing the trajectory energy). We have also analysed methods to reduce the computation time during the run (the "explicit MPC" method, exploiting the geometrical properties of the related control law).
Activity 3.3 - Improving sound analysis and recognition methods (Final): One of the modules that are part of the robotic platform is the audio module. It is used to identify acoustic events that may occur in an indoor environment and can make certain decisions based on them. Six classification models are used in conjunction to identify the acoustic event. The final decision is made based on the predominant result. For the six individual models, in the test phase, the accuracy of identifying the correct event is: 99.091%, 98.939%, 99.697%, 99.545%, 97.576% and 98.333% respectively. The final decision of acoustic event identification is made based on the predominant result increasing the individual accuracy.
Activity 3.4 - Integration of the sound detection and recognition module with a task generator: Depending on the type of event detected, OMNI-Z chooses the appropriate task. If the detected event corresponds to a normal action, such as moving a chair, opening or closing a door, asking for an ID, using the washing machine or the microwave, etc., the generated task is to send a notification email to the person responsible for the permanent remote monitoring of the home where the OMNI-Z is located. On the other hand, if the event detected is a potentially alarming one, such as a strong cough that indicates a potential choking of the person, a request for specific types of medication, etc., the generated task is to send an alert email both to the person who has to intervene in such cases and to the person who permanently monitors the home remotely.
Activity 3.5 - Validation under experimental conditions – DEMO: The platform has demonstrated the following capabilities (illustrated by experimentation during the "DEMO Iasi" related activities held on 15-16 September): it can enter/exit an elevator and cross door thresholds without restrictions; it is able to plan/route a trajectory efficiently and with safety guarantees. Modular structure allows it to integrate various locator modules; it can pass over floor unevenness.
Brief presentation of the OMNI-Z project: The OMNI-Z consortium comprises a SME and three outstanding Romanian universities which have collaborated to develop an economically accessible, omnidirectional, modular and expandable robotic platform suitable for indoor navigation. The platform capabilities for indoor navigation are ensured by its locomotion mechanism which allows it to maneuver in a confined and crowded space by being able to rotated on the spot, navigate between obstacles, or pass over barriers such as doorsteps, ramps, etc.
The OMNI-Z platform is intended to be used both as a stand-alone product and integrated with sensors and assistive devices. The target customers of such a product are institutions for the elderly, hospitals but, not far in the future, individual users. The OMNI-Z products which also including software and databases will be exploited both commercially and for scientific purposes to promote robotic development in Romania.
The images below show the OMNI-Z platform in the validation and testing environment.
Activity 3.6 - Dissemination of the project results.
- Corneliu Rusu, Lăcrimioara Grama, “Analog Phase Samples Approximation from Gain Samples by Discrete Hilbert Transform,” Circuits, Systems, and Signal Processing, vol. 41, issue 2, 2022, ISSN: 1531-5878, DOI: 10.1007/s00034-022-01965-0, WOS: 000751202000001, https://rdcu.be/cGoia.
- Ștefan Grama, Lăcrimioara Grama, “Acoustic Event Detector for Indoor Enviroments by Use of a Robotic Platform,” Carpathian Journal of Electronic and Computer Engineering, (submitted in august 2022).
- Miruna Daciana Cojocaru, Irina Georgiana MOCANU, Emergency Detection Based on Daily Activities, (sent to Sensors Journal - Q2, IF: 3.847).
- Lăcrimioara Grama, Lorena Muscar, Corneliu Rusu, Toma Telembici, “Audio Database for TIAGo Service Robot,” Carpathian Journal of Electronic and Computer Engineering, vol. 14, no.1, pp. 1-5, 2021, ISSN: 1844-9689, https://doi.org/10.2478/cjece-2021-0001.
- Gabriel Popan, Lorena Muscar, Lăcrimioara Grama, “Security System Aided by Voice Fingerprint,” Carpathian Journal of Electronic and Computer Engineering, vol. 14, no.1, pp. 24-29, 2021, ISSN: 1844-9689, https://doi.org/10.2478/cjece-2021-0005.
- Horațiu Pop, Alin Grama, “Increasing Energy Efficiency Using Photovoltaic Panels,” Carpathian Journal of Electronic and Computer Engineering, vol. 14, no.1, pp. 6-10, 2021, ISSN: 1844-9689, https://doi.org/10.2478/cjece-2021-0002.
- Claudiu Popescu, Lăcrimioara Grama, Corneliu Rusu, “A Highly Scalable Method for Extractive Text Summarization Using Convex Optimization,” Symmetry, vol. 13, no. 10: 1824, https://doi.org/10.3390/sym13101824.
- Lorena Muscar, Lăcrimioara Grama, Corneliu Rusu, “About optimizing the audio operation of the TIAGo Service Robot to assist in medical applications,” International Conference on Electromagnetic Fields, Signals and BioMedical Engineering (ICEMS-BIOMED), May 19-20, 2022, Sibiu, Romania, (presented article).
- Lorena Muscar, Lăcrimioara Grama, Corneliu Rusu, “A Real-Time Warning Based on TIAGo's Audio Capabilities,” International Symposium on Electronics and Telecommunications (ISETC), November 10-11, 2022, Timisoara, Romania, (submitted in july 2022).
- Toma Telembici, Lorena Muscar, Lăcrimioara Grama, Corneliu Rusu, “Emotion Recognition Audio Database for Service Robots,” International Symposium on Electronics and Telecommunications (ISETC), November 10-11, 2022, Timisoara, Romania, (submitted in july 2022).
- Ștefan-Sergiu Mihai, Florin Stoican și Bogdan Ciubotaru. “On the Link between Explicit MPC and the Face Lattice of the Lifted Feasible Domain”, pag. 314–319. Proceedings of the 18th IFAC Workshop on Control Applications of Optimization (CAO’22). 2022 (presented article, under ISI indexing Process).
- Lorena Muscar, Lăcrimioara Grama, “Audio Events Detection to Help TIAGo to Act as a Medical Robot,” 7th International Conference on Advancements of Medicine and Health Care through Technology, Cluj-Napoca, Romania, October 13-15, 2020, online (presented article).
- Toma Telembici, Lăcrimioara Grama, Corneliu Rusu, “Integrating Service Robots into Everyday Life Based on Audio Capabilities,” 14th International Symposium on Electronics and Telecommunications, Timisoara, Romania, November 5-6, 2020, online (presented article).
- Corneliu Rusu, Jaakko Astola, “About the positivity of trigonometric polynomials with positive samples,” 14th International Symposium on Electronics and Telecommunications, Timisoara, Romania, November 5-6, 2020, online (presented article).
- Elena-Mirela Stetco, Ovidiu Aurel Pop, Alin Grama, “Simulation Model of a GMR Based Current Sensor,” 26th International Symposium for Design and Technology in Electronic Packaging, Pitesti, Romania, October 21-24, 2020, online (presented article).
- Lăcrimioara Grama, Lorena Muscar, Corneliu Rusu, “Sound Classification Algorithms for Indoor Human Activities,” 16th International Conference on Engineering of Modern Electric Systems (EMES), pp. 1-4, June 10-11, 2021, Oradea, Romania, ISBN: 978-1-6654-4995-3, https://doi.org/10.1109/EMES52337.2021.9484121, (presented article).
- Lorena Muscar, Lăcrimioara Grama, Corneliu Rusu, “Sound Classification by the TIAGo Service Robot for Healthcare Applications,” International Symposium on Signals, Circuits and Systems (ISSCS), pp. 1-4, July 15-16, 2021, Iași, Romania, ISBN: 978-1-6654-4942-7, https://doi.org/10.1109/ISSCS52333.2021.9497410, (presented article).
- Olimpiu Pop, Corneliu Rusu, Lăcrimioara Grama, “Acoustic monitoring of outdoor areas by a sensor consisting of four microphones,” International Symposium on Signals, Circuits and Systems (ISSCS), pp. 1-4, July 15-16, 2021, Iasi, Romania, ISBN: 978-1-6654-4942-7, https://doi.org/10.1109/ISSCS52333.2021.9497372, (presented article).
- Toma Telembici, Lăcrimioara Grama, Lorena Muscar, Corneliu Rusu, “Results on the MFCC extraction for improving audio capabilities of TIAGo service robot,” 11th Conference on Speech Technology and Human-Computer Dialogue (SpeD), pp, Oct 13-15, 2021, Bucharest, Romania (presented article).
- Stoican Florin, Postolache Alexandru, Prodan Ionela, “NURBS-based trajectory design for motion planning in a multi-obstacle environment”, 2021 European Control Conference (ECC) June 29 - July 2, 2021. Rotterdam, Netherlands, pp. 2014-2019 (presented article).
- M. Dragoi, I.G. Mocanu, O. Cramariuc, “Object Manipulation for Assistive Robots,” IEEE International Conference on e-Health and Bioengineering EHB 2021 - 9-th Edition, 18-19 November 2021, Iasi, Romania (presented article).
Workshop of the project
- The workshop organized in the framework of the project - Versatile and economically viable robotic platform for indoor navigation in crowded and obstructed environments - contract 52PTE/01.10.2020 - funded by the Executive Unit for Financing Higher Education, Research, Development and Innovation took place on 16 September 2022 at the Technical University "Gheorghe Asachi" of Iasi, Faculty of Hydrotechnics, Geodesy and Environmental Engineering. The event was attended by 25 people from the Polytechnic University of Bucharest, Technical University of Iasi, SC CITST SRL Bucharest, SC Minet Râmnicu Vâlcea and other entities.