Tools for Data Driven Research and Development in RoboCup
An empirical scientific discipline requires a set of methods and practices for evaluation and comparison of proposed models and solutions. RoboCup provides a unique common test scenario for robotics, but its potential is, by far, not realized. Conducting games incurs high cost in terms of effort, time, and money. The scientific outcome, however, is quite limited and often not very conclusive. In most cases only the final score of the games provides feedback about the performance of a team. In this project we develop a toolbox (a set of tools) to support collection, organization and analysis of large amounts of RoboCup specific data enabling detailed analysis to promote data driven research and development in RoboCup.
Main components of the robocup.tools ecosystem are:
- Collection and Synchronization – automatic synchronized video recording, communication data, logs recorded by the individual robots, synchronization tool for not synchronized videos;
- Storage, Formats and Query – log files unrolled into a database, events as JSON, logging on robots based on Protobuf (backward compatible), access through web-interfaces;
- Mining and Extraction of Information – scripts to extract events from logs (e.g., robot fell), tool for manual annotation of events, detection and tracking of objects in videos with machine learning;
- Aggregation, Analysis and Visualization – statistics over events (identify systematic problems), visualization of strategies, access quality of the perception of the robots over the course of the game
Collaboration
If you are interested in deploying the recording system at a RoboCup event or want to collaborate, you are welcome to contact us.
Collected Data and Videos
-
List of all collected Videos:
https://github.com/BerlinUnited/SPLGames -
Berlin United Logs + Videos:
https://logs.naoth.de/
Event Videos on YouTube:
Event | Videos | GameController logs |
---|---|---|
German Open 2018 | [GO18 Videos] | [GameController logs] |
RoboCup 2018 | [Day 1] [Day 2] [Day 3] [Day 4] | [GameController logs] |
German Open 2019 | [GO19 Videos] | [GameController logs] |
RoboCup 2019 | [Day 1] [Day 2] [Day 3] [Day 4] | [GameController logs] |
RoboCup Explorer
The RoboCup Data Explorer is a web based server application for organization, inspection and annotation of large amounts of RoboCup data consisting of videos, network communication and log files recorded by individual robots.
- RoboCup Explorer demo:
https://www2.informatik.hu-berlin.de/~naoth/videolabeling/
Resources
- Setup Tutorial for the camera system
Publications
- RCF Project poster 2023 [PDF]
- Toward data driven development in robocup. H. Mellmann, S. A. Schlotter, and P. Strobel Robocup 2019: robot soccer world cup xxiii , 2019.
[PDF] - RCF Project poster 2018 [PDF]
- Project white paper [PDF]
Code
- Source Code on GitHub:
https://github.com/BerlinUnited/RoboCupTools - Event annotation tool source code:
https://github.com/BerlinUnited/RoboCupTools/tree/master/VideoLogLabeling - Berlin United code release:
https://github.com/BerlinUnited/NaoTH