diff options
| author | Arpit Gupta <guptarpit1997@gmail.com> | 2019-12-19 22:48:00 +0530 |
|---|---|---|
| committer | Marcus Müller <mmueller@gnuradio.org> | 2019-12-19 18:18:00 +0100 |
| commit | 116f0401f54e4c8483952118c013b8c668eb3682 (patch) | |
| tree | c4362ba8274ff68d78f8179603d3173481134775 /gr-utils/python | |
| parent | f3dcc45afea4fafa84b0c0e861031105a67bbaf2 (diff) | |
Block header parsing tool: GSoC 2019 (#2750)
* Add base.py file in cli module to import override Click functions
* Create cli and core base module for AST generation of header blocks
* Create basic CLI for blocktool with minimal support
* Add Sequence Completer to CLI and successful generation of AST
* CLI structure complete with parseheader command
* Basic core structure complete
* Add test script gr_blocktool to run the tool
* Add JSON schema and validation for parsed json output file
* Change properties and methods key to list in JSON schema
* Create an independent api from blocktool
* Bug fix for abslute path of the header files
* Create basic parser core api
* Parse the block header documentation
* Expose the core api, minor bug fixes
* Create the code pylint compatible
* Modify cli to accept file_path as an argument, parse default values of make function arguments
* Fix: Namespace parsing of block header file
* Parse the io_signature from the implementation file of the block header
* Create json file generator
* Add key-value io_signature and docstring in json schema, change sample generated json output
* Fix: squash an I/O parsing bug
* Change directory structure for blocktool tests
* Add Blocktool unittest
* Removed empty strings, make the code pylint compatible
* Use str.format() to get output
* Implement YAML generator
* Add a new CLI argument to parse a complete header directory
* Add Logger to log errors without raising exceptions
* Create output schema file in blocktool core
* Change directory structure of blocktool and cli commands
* write unittests for Blocktool Excceptions
* Add sample yaml files
* Simplify blocktool cli structure
* Refactor blocktool exception handling
* Split long blocktool unit-tests
* Parse message ports from the implementation file
* Add tests for parsed message port id, update sample json files
* Add blocktool subdirectory, files in CMakeLists.txt
* Remove test files to run Blocktool
* Fix: locates implementation file by traversing the module
* Integrate blocktool with modtool as an external plugin
* Create proper formatting of io_signature for yaml files
* Extend modtool makeyaml command to extend support for blocktool
* Remove external plugin for modtool support, add blocktool independent script
* Minor formatiing, change function name due to conflict with modtool function
* Add support to read and add blocktool special comments in header file
* Fix: Key Errors, Modify Documentation Reader
* Raise warning in case of conflict in the parsed information and blocktool comments
* Remove all the blocktool boilerplate cli code and provide minimal support
* Remove gr_blocktool script and use blocktool as a python module
* Major refactoring of the modtool cli structure to support the blocktool API
* Check for PyGCCXML dependency during build
* Add README.md for gr-blocktool and remove modtool cli warnings
Diffstat (limited to 'gr-utils/python')
27 files changed, 2178 insertions, 24 deletions
diff --git a/gr-utils/python/blocktool/CMakeLists.txt b/gr-utils/python/blocktool/CMakeLists.txt new file mode 100644 index 0000000000..f086800436 --- /dev/null +++ b/gr-utils/python/blocktool/CMakeLists.txt @@ -0,0 +1,32 @@ +# Copyright 2019 Free Software Foundation, Inc. +# +# This file is part of GNU Radio +# +# GNU Radio is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 3, or (at your option) +# any later version. +# +# GNU Radio is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with GNU Radio; see the file COPYING. If not, write to +# the Free Software Foundation, Inc., 51 Franklin Street, +# Boston, MA 02110-1301, USA. + +include(GrPython) + +GR_PYTHON_INSTALL(FILES + __init__.py + __main__.py + cli.py + DESTINATION ${GR_PYTHON_DIR}/gnuradio/blocktool +) + +######################################################################## +# Add subdirectories +######################################################################## +add_subdirectory(core) diff --git a/gr-utils/python/blocktool/README.blocktool b/gr-utils/python/blocktool/README.blocktool new file mode 100644 index 0000000000..0183b0093c --- /dev/null +++ b/gr-utils/python/blocktool/README.blocktool @@ -0,0 +1,28 @@ +gr_blocktool: Also known as the block header parsing tool, this tool + automatically parses any GNU Radio or OOT block header. + +Block header tool from the Command Line Interface +================================================= + +* Parse any GNU Radio or OOT header file with just the file path as an input. +* Parse a complete header directory with directory-path as the input. +* Get the output in the form of a YAML or a JSON file. +* Add blocktool comments automatically in the header file from the implementation file. + + +Integration of blocktool with modtool +===================================== + +* Blocktool API can also be called from modtool. +* Modtool makeyaml subcommand along with -b flag and a + file path as the input can be used to create YAML files for the GRC. +* YAML output is much better using the blocktool API. + + +Use of blocktool as an independent API +====================================== + +Blocktool can be also be used as an independent API which thus can be used to +parse a block header file during runtime. +* A single mandatory argument block header file path is required. + diff --git a/gr-utils/python/blocktool/__init__.py b/gr-utils/python/blocktool/__init__.py new file mode 100644 index 0000000000..fd4fbe7cd1 --- /dev/null +++ b/gr-utils/python/blocktool/__init__.py @@ -0,0 +1,23 @@ +# +# Copyright 2019 Free Software Foundation, Inc. +# +# This file is part of GNU Radio +# +# GNU Radio is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 3, or (at your option) +# any later version. +# +# GNU Radio is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with GNU Radio; see the file COPYING. If not, write to +# the Free Software Foundation, Inc., 51 Franklin Street, +# Boston, MA 02110-1301, USA. +# + +from .core.parseheader import BlockHeaderParser +from .core.outputschema import RESULT_SCHEMA diff --git a/gr-utils/python/blocktool/__main__.py b/gr-utils/python/blocktool/__main__.py new file mode 100644 index 0000000000..9ef683d4ed --- /dev/null +++ b/gr-utils/python/blocktool/__main__.py @@ -0,0 +1,29 @@ +# +# Copyright 2019 Free Software Foundation, Inc. +# This file is part of GNU Radio +# +# GNU Radio Companion is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public License +# as published by the Free Software Foundation; either version 2 +# of the License, or (at your option) any later version. +# +# GNU Radio Companion is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA +# +""" main function to run the blocktool api from the command line. """ + +from __future__ import print_function +from __future__ import absolute_import +from __future__ import unicode_literals + +import sys +from .cli import cli + + +sys.exit(cli()) diff --git a/gr-utils/python/blocktool/cli.py b/gr-utils/python/blocktool/cli.py new file mode 100644 index 0000000000..dbb7b7cbc5 --- /dev/null +++ b/gr-utils/python/blocktool/cli.py @@ -0,0 +1,140 @@ +# +# Copyright 2019 Free Software Foundation, Inc. +# +# This file is part of GNU Radio +# +# GNU Radio is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 3, or (at your option) +# any later version. +# +# GNU Radio is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with GNU Radio; see the file COPYING. If not, write to +# the Free Software Foundation, Inc., 51 Franklin Street, +# Boston, MA 02110-1301, USA. +# +""" Module to generate parsed header output data """ + +from __future__ import print_function +from __future__ import absolute_import +from __future__ import unicode_literals + +import os +import sys +import json +import logging + +import click +from click import ClickException + +from gnuradio.modtool.core import yaml_generator + +from .core.parseheader import BlockHeaderParser + +LOGGER = logging.getLogger(__name__) + + +class BlockToolException(ClickException): + """ Exception class for enhanced CLI interface """ + + def show(self, file=None): + """ displays the colored message """ + click.secho('BlockToolException: {}'.format( + self.format_message()), fg='red') + + +def run_blocktool(module): + """Call the run function of the core modules.""" + try: + module.run_blocktool() + except BlockToolException as err: + click.echo(err, file=sys.stderr) + exit(1) + + +@click.command('parseheader', + short_help='Generate the parsed output for the header file or directory in a specified format') +@click.argument('file-path', nargs=1) +@click.option('--yaml', is_flag=True, + help='If given, a YAML response will be printed, else default json will be printed') +@click.option('-c', '--blocktool-comments', is_flag=True, + help='blocktool helper comments will be added in the header file') +@click.option('-o', '--output', is_flag=True, + help='If given, a file with desired output format will be generated') +def cli(**kwargs): + """ + Block header parsing tool. + \b + A tool that can be used to automatically parse the headers in GNU Radio project + or the OOT modules + """ + kwargs['modtool'] = False + if os.path.isfile(kwargs['file_path']): + parser = BlockHeaderParser(**kwargs) + run_blocktool(parser) + if kwargs['yaml']: + parser.yaml = True + yaml_generator(parser, **kwargs) + else: + parser.json_confirm = True + json_generator(parser, **kwargs) + elif os.path.isdir(kwargs['file_path']): + parse_directory(**kwargs) + else: + raise BlockToolException('Invalid file or directory path.') + + +def json_generator(parser, **kwargs): + """ + Generate JSON file for the block header + """ + header = parser.filename.split('.')[0] + block = parser.modname.split('-')[-1] + if kwargs['output']: + json_file = os.path.join('.', block+'_'+header + '.json') + with open(json_file, 'w') as _file: + json.dump(parser.parsed_data, _file, indent=4) + else: + print(json.dumps(parser.parsed_data, indent=4)) + + +def parse_directory(**kwargs): + """ + Get parsed json and yaml output for complete header directory + """ + kwargs['output'] = True + dir_path = kwargs['file_path'] + dir_path = os.path.abspath(dir_path) + list_header = [] + dir_name = os.path.basename(dir_path) + for _header in os.listdir(dir_path): + if _header.endswith('.h') and os.path.isfile(os.path.join(dir_path, _header)): + list_header.append(os.path.join(dir_path, _header)) + list_header = sorted(list_header) + if list_header: + for header_path in list_header: + kwargs['file_path'] = header_path + header = os.path.basename(header_path) + try: + parse_dir = BlockHeaderParser(**kwargs) + parse_dir.yaml = True + parse_dir.json = True + run_blocktool(parse_dir) + yaml_generator(parse_dir, **kwargs) + if not kwargs['modtool']: + json_generator(parse_dir, **kwargs) + except: + logging.basicConfig(level=logging.DEBUG, + filename=os.path.join('.', dir_name+'_log.out')) + logging.exception( + 'Log for Exception raised for the header: {}\n'.format(header)) + click.secho('Parsing unsuccessful: {}'.format( + header), fg='yellow') + else: + raise BlockToolException( + 'Invalid directory! No header found to be parsed') diff --git a/gr-utils/python/blocktool/core/CMakeLists.txt b/gr-utils/python/blocktool/core/CMakeLists.txt new file mode 100644 index 0000000000..7b0a704847 --- /dev/null +++ b/gr-utils/python/blocktool/core/CMakeLists.txt @@ -0,0 +1,31 @@ +# Copyright 2019 Free Software Foundation, Inc. +# +# This file is part of GNU Radio +# +# GNU Radio is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 3, or (at your option) +# any later version. +# +# GNU Radio is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with GNU Radio; see the file COPYING. If not, write to +# the Free Software Foundation, Inc., 51 Franklin Street, +# Boston, MA 02110-1301, USA. + +include(GrPython) + +GR_PYTHON_INSTALL(FILES + __init__.py + base.py + comments.py + parseheader.py + iosignature.py + outputschema.py + Constants.py + DESTINATION ${GR_PYTHON_DIR}/gnuradio/blocktool/core +) diff --git a/gr-utils/python/blocktool/core/Constants.py b/gr-utils/python/blocktool/core/Constants.py new file mode 100644 index 0000000000..694f1b146e --- /dev/null +++ b/gr-utils/python/blocktool/core/Constants.py @@ -0,0 +1,67 @@ +# +# Copyright 2019 Free Software Foundation, Inc. +# +# This file is part of GNU Radio +# +# GNU Radio is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 3, or (at your option) +# any later version. +# +# GNU Radio is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with GNU Radio; see the file COPYING. If not, write to +# the Free Software Foundation, Inc., 51 Franklin Street, +# Boston, MA 02110-1301, USA. +# +""" constants file """ + +# Kernel Namespace +KERNEL = 'kernel' + +# I/O Signature (Symbols and constants) +IO_SIGNATURE = 'io_signature::' +SIGNATURE_LIST = ['makev', 'make3', 'make2', 'make'] +MAKE = 'make' +MAKE2 = 'make2' +MAKE3 = 'make3' +MAKEV = 'makev' + + +# message ports id +MESSAGE_INPUT = 'message_port_register_in' +MESSAGE_OUTPUT = 'message_port_register_out' + +# Symbols and constants required for parsing +GR = 'gr-' +UTILS = 'utils' +OPEN_BRACKET = '(' +CLOSE_BRACKET = ')' +STRIP_SYMBOLS = ' ,:)' +EXCLAMATION = '!' + +# Blocktool special comments +BLOCKTOOL = '! BlockTool' +END_BLOCKTOOL = 'EndTool !' +INPUT_SIG = 'input_signature' +OUTPUT_SIG = 'output_signature' +INPUT_MIN = 'input_min_streams' +INPUT_MAX = 'input_max_streams' +OUTPUT_MIN = 'output_min_streams' +OUTPUT_MAX = 'output_max_streams' +INPUT_MAKE_SIZE = 'input_sizeof_stream_item' +INPUT_MAKEV_SIZE = 'input_sizeof_stream_items' +INPUT_MAKE_SIZE1 = 'input_sizeof_stream_item1' +INPUT_MAKE_SIZE2 = 'input_sizeof_stream_item2' +INPUT_MAKE_SIZE3 = 'input_sizeof_stream_item3' +OUTPUT_MAKE_SIZE = 'output_sizeof_stream_item' +OUTPUT_MAKEV_SIZE = 'output_sizeof_stream_items' +OUTPUT_MAKE_SIZE1 = 'output_sizeof_stream_item1' +OUTPUT_MAKE_SIZE2 = 'output_sizeof_stream_item2' +OUTPUT_MAKE_SIZE3 = 'output_sizeof_stream_item3' +INPUT_PORT = 'message_input' +OUTPUT_PORT = 'message_output' diff --git a/gr-utils/python/blocktool/core/__init__.py b/gr-utils/python/blocktool/core/__init__.py new file mode 100644 index 0000000000..e69de29bb2 --- /dev/null +++ b/gr-utils/python/blocktool/core/__init__.py diff --git a/gr-utils/python/blocktool/core/base.py b/gr-utils/python/blocktool/core/base.py new file mode 100644 index 0000000000..5e62835e12 --- /dev/null +++ b/gr-utils/python/blocktool/core/base.py @@ -0,0 +1,53 @@ +# +# Copyright 2019 Free Software Foundation, Inc. +# +# This file is part of GNU Radio +# +# GNU Radio is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 3, or (at your option) +# any later version. +# +# GNU Radio is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with GNU Radio; see the file COPYING. If not, write to +# the Free Software Foundation, Inc., 51 Franklin Street, +# Boston, MA 02110-1301, USA. +# +""" Base class for the modules """ + +from __future__ import print_function +from __future__ import absolute_import +from __future__ import unicode_literals + +from abc import ABC, abstractmethod + + +class BlockToolException(Exception): + """ Standard exception for blocktool classes. """ + pass + + +class BlockTool(ABC): + """ Base class for all blocktool command classes. """ + name = 'base' + description = None + + def __init__(self, modname=None, filename=None, targetdir=None, + target_file=None, module=None, impldir=None, impl_file=None, + yaml=False, json=False, **kwargs): + """ __init__ """ + pass + + def _validate(self): + """ Validates the arguments """ + pass + + @abstractmethod + def run_blocktool(self): + """ Override this. """ + pass diff --git a/gr-utils/python/blocktool/core/comments.py b/gr-utils/python/blocktool/core/comments.py new file mode 100644 index 0000000000..9938eca122 --- /dev/null +++ b/gr-utils/python/blocktool/core/comments.py @@ -0,0 +1,270 @@ +# +# Copyright 2019 Free Software Foundation, Inc. +# +# This file is part of GNU Radio +# +# GNU Radio is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 3, or (at your option) +# any later version. +# +# GNU Radio is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with GNU Radio; see the file COPYING. If not, write to +# the Free Software Foundation, Inc., 51 Franklin Street, +# Boston, MA 02110-1301, USA. +# +""" Module to read and add special blocktool comments in the public header """ + +from __future__ import print_function +from __future__ import absolute_import +from __future__ import unicode_literals + +import warnings + +from ..core import Constants + + +def strip_symbols(line): + """ + helper function to strip symbols + from blocktool comment syntax + """ + return line.split(':')[-1].lstrip().rstrip() + + +def exist_comments(self): + """ + function to check if blocktool special comments + already exist in the public header + """ + _comments = True + _index = None + lines = [] + with open(self.target_file, 'r') as header: + lines = header.readlines() + for line in lines: + if Constants.BLOCKTOOL in line: + _index = lines.index(line) + return bool(_index) + + +def validate_message_port(self, message_ports, suppress_input, suppress_output): + """ + function to solve conflicts if any in the + *message_port* comments and the implementation information + """ + if message_ports['input'] != self.parsed_data['message_port']['input']: + if not suppress_input: + warnings.warn( + 'Conflict in values input message port Id. Add ! at the start of the key-value line to mandatory use the comment value.') + self.parsed_data['message_port']['input'] = message_ports['input'] + if message_ports['output'] != self.parsed_data['message_port']['output']: + if not suppress_output: + warnings.warn( + 'Conflict in values output message port Id. Add ! at the start of the key-value line to mandatory use the comment value.') + self.parsed_data['message_port']['output'] = message_ports['output'] + + +def read_comments(self): + """ + function to read special blocktool comments + in the public header + """ + temp_parsed_data = {} + if self.parsed_data['io_signature'] or self.parsed_data['message_port']: + temp_parsed_data['io_signature'] = self.parsed_data['io_signature'] + temp_parsed_data['message_port'] = self.parsed_data['message_port'] + self.parsed_data['io_signature'] = { + "input": { + "signature": None + }, + "output": { + "signature": None + } + } + self.parsed_data['message_port'] = { + "input": [], + "output": [] + } + _suppress_input = False + _suppress_output = False + parsed_io = self.parsed_data['io_signature'] + message_port = self.parsed_data['message_port'] + special_comments = [] + _index = None + lines = [] + with open(self.target_file, 'r') as header: + lines = header.readlines() + for line in lines: + if Constants.BLOCKTOOL in line: + _index = lines.index(line) + + if _index is not None: + _index = _index+1 + for num in range(_index, len(lines)): + if Constants.END_BLOCKTOOL in lines[num]: + break + special_comments.append(lines[num]) + for comment in special_comments: + if Constants.INPUT_SIG in comment: + parsed_io['input']['signature'] = strip_symbols(comment) + if Constants.INPUT_MAX in comment: + parsed_io['input']['max_streams'] = strip_symbols(comment) + if Constants.INPUT_MIN in comment: + parsed_io['input']['min_streams'] = strip_symbols(comment) + if parsed_io['input']['signature'] is Constants.MAKE and not None: + if Constants.INPUT_MAKE_SIZE in comment: + parsed_io['input']['sizeof_stream_item'] = strip_symbols( + comment) + elif parsed_io['input']['signature'] is Constants.MAKE2 and not None: + if Constants.INPUT_MAKE_SIZE1 in comment: + parsed_io['input']['sizeof_stream_item1'] = strip_symbols( + comment) + if Constants.INPUT_MAKE_SIZE2 in comment: + parsed_io['input']['sizeof_stream_item2'] = strip_symbols( + comment) + elif parsed_io['input']['signature'] is Constants.MAKE3 and not None: + if Constants.INPUT_MAKE_SIZE1 in comment: + parsed_io['input']['sizeof_stream_item1'] = strip_symbols( + comment) + if Constants.INPUT_MAKE_SIZE2 in comment: + parsed_io['input']['sizeof_stream_item2'] = strip_symbols( + comment) + if Constants.INPUT_MAKE_SIZE3 in comment: + parsed_io['input']['sizeof_stream_item3'] = strip_symbols( + comment) + elif parsed_io['input']['signature'] is Constants.MAKEV and not None: + if Constants.INPUT_MAKEV_SIZE in comment: + parsed_io['input']['sizeof_stream_items'] = strip_symbols( + comment) + + if Constants.OUTPUT_SIG in comment: + parsed_io['output']['signature'] = strip_symbols(comment) + if Constants.OUTPUT_MAX in comment: + parsed_io['output']['max_streams'] = strip_symbols(comment) + if Constants.OUTPUT_MIN in comment: + parsed_io['output']['min_streams'] = strip_symbols(comment) + if parsed_io['output']['signature'] is Constants.MAKE and not None: + if Constants.OUTPUT_MAKE_SIZE in comment: + parsed_io['output']['sizeof_stream_item'] = strip_symbols( + comment) + elif parsed_io['output']['signature'] is Constants.MAKE2: + if Constants.OUTPUT_MAKE_SIZE1 in comment: + parsed_io['output']['sizeof_stream_item1'] = strip_symbols( + comment) + if Constants.OUTPUT_MAKE_SIZE2 in comment: + parsed_io['output']['sizeof_stream_item2'] = strip_symbols( + comment) + elif parsed_io['output']['signature'] is Constants.MAKE3 and not None: + if Constants.OUTPUT_MAKE_SIZE1 in comment: + parsed_io['output']['sizeof_stream_item1'] = strip_symbols( + comment) + if Constants.OUTPUT_MAKE_SIZE2 in comment: + parsed_io['output']['sizeof_stream_item2'] = strip_symbols( + comment) + if Constants.OUTPUT_MAKE_SIZE3 in comment: + parsed_io['output']['sizeof_stream_item3'] = strip_symbols( + comment) + elif parsed_io['output']['signature'] is Constants.MAKEV and not None: + if Constants.OUTPUT_MAKEV_SIZE in comment: + parsed_io['output']['sizeof_stream_items'] = strip_symbols( + comment) + + if Constants.INPUT_PORT in comment: + if Constants.EXCLAMATION in comment: + _suppress_input = True + if strip_symbols(comment): + message_port['input'] = strip_symbols(comment).split(', ') + if Constants.OUTPUT_PORT in comment: + if Constants.EXCLAMATION in comment: + _suppress_output = True + if strip_symbols(comment): + message_port['output'] = strip_symbols(comment).split(', ') + validate_message_port( + self, temp_parsed_data['message_port'], _suppress_input, _suppress_output) + self.parsed_data['io_signature'] = temp_parsed_data['io_signature'] + + +def add_comments(self): + """ + function to add special blocktool comments + in the public header + """ + _index = None + lines = [] + parsed_io = self.parsed_data['io_signature'] + message_port = self.parsed_data['message_port'] + with open(self.target_file, 'r') as header: + lines = header.readlines() + for line in lines: + if Constants.BLOCKTOOL in line: + _index = lines.index(line) + if _index is None: + with open(self.target_file, 'a') as header: + header.write('\n') + header.write('/* '+Constants.BLOCKTOOL + '\n') + header.write('input_signature: ' + + parsed_io['input']['signature'] + '\n') + header.write('input_min_streams: ' + + parsed_io['input']['min_streams'] + '\n') + header.write('input_max_streams: ' + + parsed_io['input']['max_streams'] + '\n') + if parsed_io['input']['signature'] is Constants.MAKE: + header.write('input_sizeof_stream_item: ' + + parsed_io['input']['sizeof_stream_item'] + '\n') + elif parsed_io['input']['signature'] is Constants.MAKE2: + header.write('input_sizeof_stream_item1: ' + + parsed_io['input']['sizeof_stream_item1'] + '\n') + header.write('input_sizeof_stream_item2: ' + + parsed_io['input']['sizeof_stream_item2'] + '\n') + elif parsed_io['input']['signature'] is Constants.MAKE3: + header.write('input_sizeof_stream_item1: ' + + parsed_io['input']['sizeof_stream_item1'] + '\n') + header.write('input_sizeof_stream_item2: ' + + parsed_io['input']['sizeof_stream_item2'] + '\n') + header.write('input_sizeof_stream_item3: ' + + parsed_io['input']['sizeof_stream_item3'] + '\n') + elif parsed_io['input']['signature'] is Constants.MAKEV: + header.write('input_sizeof_stream_item: ' + + parsed_io['input']['sizeof_stream_items'] + '\n') + header.write('output_signature: ' + + parsed_io['output']['signature'] + '\n') + header.write('output_min_streams: ' + + parsed_io['output']['min_streams'] + '\n') + header.write('output_max_streams: ' + + parsed_io['output']['max_streams'] + '\n') + if parsed_io['output']['signature'] is Constants.MAKE: + header.write('output_sizeof_stream_item: ' + + parsed_io['output']['sizeof_stream_item'] + '\n') + elif parsed_io['output']['signature'] is Constants.MAKE2: + header.write('output_sizeof_stream_item1: ' + + parsed_io['output']['sizeof_stream_item1'] + '\n') + header.write('output_sizeof_stream_item2: ' + + parsed_io['output']['sizeof_stream_item2'] + '\n') + elif parsed_io['output']['signature'] is Constants.MAKE3: + header.write('output_sizeof_stream_item1: ' + + parsed_io['output']['sizeof_stream_item1'] + '\n') + header.write('output_sizeof_stream_item2: ' + + parsed_io['output']['sizeof_stream_item2'] + '\n') + header.write('output_sizeof_stream_item3: ' + + parsed_io['output']['sizeof_stream_item3'] + '\n') + elif parsed_io['output']['signature'] is Constants.MAKEV: + header.write('output_sizeof_stream_item: ' + + parsed_io['output']['sizeof_stream_items'] + '\n') + + if message_port['input']: + header.write('message_input: ' + + ', '.join(message_port['input']) + '\n') + else: + header.write('message_input: ' + '\n') + if message_port['output']: + header.write('message_output: ' + + ', '.join(message_port['output']) + '\n') + else: + header.write('message_output: ' + '\n') + header.write(Constants.END_BLOCKTOOL + '*/' + '\n') diff --git a/gr-utils/python/blocktool/core/iosignature.py b/gr-utils/python/blocktool/core/iosignature.py new file mode 100644 index 0000000000..cd90eb0a86 --- /dev/null +++ b/gr-utils/python/blocktool/core/iosignature.py @@ -0,0 +1,194 @@ +# +# Copyright 2019 Free Software Foundation, Inc. +# +# This file is part of GNU Radio +# +# GNU Radio is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 3, or (at your option) +# any later version. +# +# GNU Radio is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with GNU Radio; see the file COPYING. If not, write to +# the Free Software Foundation, Inc., 51 Franklin Street, +# Boston, MA 02110-1301, USA. +# +""" Module to get io_signature of the header block """ + +from __future__ import print_function +from __future__ import absolute_import +from __future__ import unicode_literals + +import re +import itertools +import logging +import string + +from ..core import Constants + +LOGGER = logging.getLogger(__name__) + + +def io_signature(impl_file): + """ + function to generate the io_signature of the block + : returns the io parmaters + """ + parsed_io = { + "input": { + "signature": None + }, + "output": { + "signature": None + } + } + with open(impl_file, 'r') as impl: + io_lines = [] + for line in impl: + if Constants.IO_SIGNATURE in line: + io_lines.append(line) + if len(io_lines) > 2: + io_lines = io_lines[0:2] + _io_sig = [] + for line in io_lines: + if Constants.IO_SIGNATURE in line: + line = line.lstrip().rstrip().split(Constants.IO_SIGNATURE) + _io_sig.append(line) + _io_sig = list(itertools.chain.from_iterable(_io_sig)) + for index, _element in enumerate(_io_sig): + _io_sig[index] = _element.lstrip().rstrip() + if all(i in string.punctuation for i in _element): + _io_sig.remove(_element) + _io_sig = list(filter(None, _io_sig)) + io_func = [] + for _io in _io_sig: + if Constants.MAKE in _io: + io_func.append(_io.lstrip().rstrip(Constants.STRIP_SYMBOLS)) + for signature in Constants.SIGNATURE_LIST: + if signature in io_func[0] and parsed_io['input']['signature'] is None: + parsed_io['input']['signature'] = signature + io_func[0] = io_func[0].lstrip(signature+' (') + if signature in io_func[1] and parsed_io['output']['signature'] is None: + parsed_io['output']['signature'] = signature + io_func[1] = io_func[1].lstrip(signature+' (') + io_elements = [] + for _io in io_func: + _io = _io.split(',') + io_elements.append(_io) + io_elements = list(itertools.chain.from_iterable(io_elements)) + for index, _io in enumerate(io_elements): + _io = _io.lstrip(' (').rstrip(' )') + if Constants.OPEN_BRACKET in _io: + _io = _io + Constants.CLOSE_BRACKET + io_elements[index] = _io + + # Because of any possible combination of I/O signature and different number + # of arguments, manual if else loop is required + if parsed_io['input']['signature'] is Constants.MAKE: + parsed_io['input']['min_streams'] = io_elements[0] + parsed_io['input']['max_streams'] = io_elements[1] + parsed_io['input']['sizeof_stream_item'] = io_elements[2] + del io_elements[0:3] + elif parsed_io['input']['signature'] is Constants.MAKE2: + parsed_io['input']['min_streams'] = io_elements[0] + parsed_io['input']['max_streams'] = io_elements[1] + parsed_io['input']['sizeof_stream_item1'] = io_elements[2] + parsed_io['input']['sizeof_stream_item2'] = io_elements[3] + del io_elements[0:4] + elif parsed_io['input']['signature'] is Constants.MAKE3: + parsed_io['input']['min_streams'] = io_elements[0] + parsed_io['input']['max_streams'] = io_elements[1] + parsed_io['input']['sizeof_stream_item1'] = io_elements[2] + parsed_io['input']['sizeof_stream_item2'] = io_elements[3] + parsed_io['input']['sizeof_stream_item3'] = io_elements[4] + del io_elements[0:5] + elif parsed_io['input']['signature'] is Constants.MAKEV: + parsed_io['input']['min_streams'] = io_elements[0] + parsed_io['input']['max_streams'] = io_elements[1] + parsed_io['input']['sizeof_stream_items'] = io_elements[2] + del io_elements[0:3] + + if parsed_io['output']['signature'] is Constants.MAKE: + parsed_io['output']['min_streams'] = io_elements[0] + parsed_io['output']['max_streams'] = io_elements[1] + parsed_io['output']['sizeof_stream_item'] = io_elements[2] + del io_elements[0:3] + elif parsed_io['output']['signature'] is Constants.MAKE2: + parsed_io['output']['min_streams'] = io_elements[0] + parsed_io['output']['max_streams'] = io_elements[1] + parsed_io['output']['sizeof_stream_item1'] = io_elements[2] + parsed_io['output']['sizeof_stream_item2'] = io_elements[3] + del io_elements[0:4] + elif parsed_io['output']['signature'] is Constants.MAKE3: + parsed_io['output']['min_streams'] = io_elements[0] + parsed_io['output']['max_streams'] = io_elements[1] + parsed_io['output']['sizeof_stream_item1'] = io_elements[2] + parsed_io['output']['sizeof_stream_item2'] = io_elements[3] + parsed_io['output']['sizeof_stream_item3'] = io_elements[4] + del io_elements[0:5] + elif parsed_io['output']['signature'] is Constants.MAKEV: + parsed_io['output']['min_streams'] = io_elements[0] + parsed_io['output']['max_streams'] = io_elements[1] + parsed_io['output']['sizeof_stream_items'] = io_elements[2] + del io_elements[0:3] + return parsed_io + + +def message_port(impl_file): + """ + parses message ports from implementation file + """ + parsed_message_port = { + "input": [], + "output": [] + } + with open(impl_file, 'r') as impl: + _input = [] + _output = [] + for line in impl: + if Constants.MESSAGE_INPUT in line: + _input.append(line) + if Constants.MESSAGE_OUTPUT in line: + _output.append(line) + + input_port = [] + output_port = [] + if _input: + for port in _input: + port = port.lstrip().rstrip().strip(Constants.MESSAGE_INPUT) + pattern = port.find('\"') + if pattern != -1: + if re.findall(r'"([^"]*)"', port)[0]: + input_port.append(re.findall(r'"([^"]*)"', port)[0]) + else: + input_port.append(port[port.find('(')+1:port.rfind(')')]) + _temp_port = ''.join(map(str, input_port)) + input_port.clear() + input_port.append(_temp_port) + + if _output: + for port in _output: + port = port.lstrip().rstrip().strip(Constants.MESSAGE_OUTPUT) + pattern = port.find('\"') + if pattern != -1: + if re.findall(r'"([^"]*)"', port)[0]: + output_port.append(re.findall(r'"([^"]*)"', port)[0]) + else: + output_port.append(port[port.find('(')+1:port.rfind(')')]) + _temp_port = ''.join(map(str, output_port)) + output_port.clear() + output_port.append(_temp_port) + + if input_port: + for port in input_port: + parsed_message_port['input'].append(port) + + if output_port: + for port in output_port: + parsed_message_port['output'].append(port) + return parsed_message_port diff --git a/gr-utils/python/blocktool/core/outputschema.py b/gr-utils/python/blocktool/core/outputschema.py new file mode 100644 index 0000000000..baa9b61042 --- /dev/null +++ b/gr-utils/python/blocktool/core/outputschema.py @@ -0,0 +1,169 @@ +# +# Copyright 2019 Free Software Foundation, Inc. +# +# This file is part of GNU Radio +# +# GNU Radio is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 3, or (at your option) +# any later version. +# +# GNU Radio is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with GNU Radio; see the file COPYING. If not, write to +# the Free Software Foundation, Inc., 51 Franklin Street, +# Boston, MA 02110-1301, USA. +# +""" Schema to be strictly followed be parsed header output """ + + +RESULT_SCHEMA = { + "title": "JSON SCHEMA TO BE FOLLOWED BY BLOCK HEADER PARSING TOOL", + "description": "Schema designed for the header file parsed python dict output", + "type": "object", + "properties": { + "namespace": { + "description": "List of nested namspace", + "type": "array", + "minItems": 1, + "uniqueItems": True, + "items": { + "type": "string", + "minLength": 1 + } + }, + "class": { + "description": "Class name", + "type": "string", + "minLength": 1 + }, + "io_signature": { + "description": "I/O signature", + "type": "object", + "properties": { + "input": { + "description": "Input ports", + "type": "object" + }, + "output": { + "description": "Output ports", + "type": "object" + } + }, + "required": ["input", "output"] + }, + "make": { + "description": "Make function", + "type": "object", + "properties": { + "arguments": { + "description": "Arguments of make function", + "type": "array", + "minItems": 1, + "uniqueItems": True, + "items": { + "type": "object", + "properties": { + "name": { + "type": "string", + "minLength": 1 + }, + "dtype": { + "type": "string", + "minLength": 1 + }, + "default": { + "type": "string" + } + }, + "required": ["name"], + "dependencies": { + "name": [ + "dtype", + "default" + ] + } + } + } + } + }, + "methods": { + "description": "Setters", + "type": "array", + "minItems": 0, + "items": { + "type": "object", + "properties": { + "name": { + "type": "string", + "minLength": 1 + }, + "arguments_type": { + "type": "array", + "uniqueItems": True, + "properties": { + "name": { + "type": "string", + "minLength": 1 + }, + "dtype": { + "type": "string", + "minLength": 1 + } + }, + "required": ["name"], + "dependencies": { + "name": ["dtype"] + } + } + }, + "required": ["name"] + } + }, + "properties": { + "description": "Getters", + "type": "array", + "uniqueItems": True, + "items": { + "type": "object", + "properties": { + "name": { + "type": "string", + "minLength": 1 + }, + "dtype": { + "type": "string", + "minLength": 1 + }, + "read_only": { + "type": "boolean" + } + }, + "required": ["name"], + "dependencies": { + "name": [ + "dtype", + "read_only" + ] + } + } + }, + "docstring": { + "description": "documentation of the header file", + "type": "array" + } + }, + "required": [ + "namespace", + "class", + "io_signature", + "make", + "methods", + "properties", + "docstring" + ] +} diff --git a/gr-utils/python/blocktool/core/parseheader.py b/gr-utils/python/blocktool/core/parseheader.py new file mode 100644 index 0000000000..7b1e743554 --- /dev/null +++ b/gr-utils/python/blocktool/core/parseheader.py @@ -0,0 +1,277 @@ +# +# Copyright 2019 Free Software Foundation, Inc. +# +# This file is part of GNU Radio +# +# GNU Radio is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 3, or (at your option) +# any later version. +# +# GNU Radio is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with GNU Radio; see the file COPYING. If not, write to +# the Free Software Foundation, Inc., 51 Franklin Street, +# Boston, MA 02110-1301, USA. +# +""" Module to generate AST for the headers and parse it """ + +from __future__ import print_function +from __future__ import absolute_import +from __future__ import unicode_literals + +import os +import re +import codecs +import logging + +from pygccxml import parser, declarations, utils + +from ..core.base import BlockToolException, BlockTool +from ..core.iosignature import io_signature, message_port +from ..core.comments import read_comments, add_comments, exist_comments +from ..core import Constants + +LOGGER = logging.getLogger(__name__) + + +class BlockHeaderParser(BlockTool): + """ + : Single argument required: file_path + file_path: enter path for the header block in any of GNU Radio module + : returns the parsed header data in python dict + : return dict keys: namespace, class, io_signature, make, + properties, methods + : Can be used as an CLI command or an extenal API + """ + name = 'Block Parse Header' + description = 'Create a parsed output from a block header file' + + def __init__(self, file_path=None, blocktool_comments=False, **kwargs): + """ __init__ """ + BlockTool.__init__(self, **kwargs) + self.parsed_data = {} + self.addcomments = blocktool_comments + if not os.path.isfile(file_path): + raise BlockToolException('file does not exist') + file_path = os.path.abspath(file_path) + self.target_file = file_path + self.initialize() + self.validate() + + def initialize(self): + """ + initialize all the required API variables + """ + self.module = self.target_file + for dirs in self.module: + if not os.path.basename(self.module).startswith(Constants.GR): + self.module = os.path.abspath( + os.path.join(self.module, os.pardir)) + self.modname = os.path.basename(self.module) + self.filename = os.path.basename(self.target_file) + self.targetdir = os.path.dirname(self.target_file) + for dirs in os.scandir(self.module): + if dirs.is_dir(): + if dirs.path.endswith('lib'): + self.impldir = dirs.path + self.impl_file = os.path.join(self.impldir, + self.filename.split('.')[0]+'_impl.cc') + + def validate(self): + """ Override the Blocktool validate function """ + BlockTool._validate(self) + if not self.filename.endswith('.h'): + raise BlockToolException( + 'Cannot parse a non-header file') + + def get_header_info(self): + """ + PyGCCXML header code parser + magic happens here! + : returns the parsed header data in python dict + : return dict keys: namespace, class, io_signature, make, + properties, methods + : Can be used as an CLI command or an extenal API + """ + gr = self.modname.split('-')[0] + module = self.modname.split('-')[-1] + generator_path, generator_name = utils.find_xml_generator() + xml_generator_config = parser.xml_generator_configuration_t( + xml_generator_path=generator_path, + xml_generator=generator_name, + compiler='gcc') + decls = parser.parse( + [self.target_file], xml_generator_config) + global_namespace = declarations.get_global_namespace(decls) + + # namespace + try: + self.parsed_data['namespace'] = [] + ns = global_namespace.namespace(gr) + if ns is None: + raise BlockToolException + main_namespace = ns.namespace(module) + if main_namespace is None: + raise BlockToolException('namespace cannot be none') + self.parsed_data['namespace'] = [gr, module] + if main_namespace.declarations: + for _namespace in main_namespace.declarations: + if isinstance(_namespace, declarations.namespace_t): + if Constants.KERNEL not in str(_namespace): + main_namespace = _namespace + self.parsed_data['namespace'].append( + str(_namespace).split('::')[-1].split(' ')[0]) + except RuntimeError: + raise BlockToolException( + 'Invalid namespace format in the block header file') + + # class + try: + self.parsed_data['class'] = '' + for _class in main_namespace.declarations: + if isinstance(_class, declarations.class_t): + main_class = _class + self.parsed_data['class'] = str(_class).split('::')[ + 2].split(' ')[0] + except RuntimeError: + raise BlockToolException( + 'Block header namespace {} must consist of a valid class instance'.format(module)) + + # io_signature, message_ports + self.parsed_data['io_signature'] = {} + self.parsed_data['message_port'] = {} + if os.path.isfile(self.impl_file) and exist_comments(self): + self.parsed_data['io_signature'] = io_signature( + self.impl_file) + self.parsed_data['message_port'] = message_port( + self.impl_file) + read_comments(self) + elif os.path.isfile(self.impl_file) and not exist_comments(self): + self.parsed_data['io_signature'] = io_signature( + self.impl_file) + self.parsed_data['message_port'] = message_port( + self.impl_file) + if self.addcomments: + add_comments(self) + elif not os.path.isfile(self.impl_file) and exist_comments(self): + read_comments(self) + else: + self.parsed_data['io_signature'] = { + "input": [], + "output": [] + } + self.parsed_data['message_port'] = self.parsed_data['io_signature'] + + # make + try: + self.parsed_data['make'] = {} + self.parsed_data['make']['arguments'] = [] + query_m = declarations.custom_matcher_t( + lambda mem_fun: mem_fun.name.startswith('make')) + query_make = query_m & declarations.access_type_matcher_t('public') + make_func = main_class.member_functions(function=query_make, + allow_empty=True, + header_file=self.target_file) + criteria = declarations.calldef_matcher(name='make') + _make_fun = declarations.matcher.get_single(criteria, main_class) + _make_fun = str(_make_fun).split( + 'make')[-1].split(')')[0].split('(')[1].lstrip().rstrip().split(',') + if make_func: + for arg in make_func[0].arguments: + for _arg in _make_fun: + if str(arg.name) in _arg: + make_arguments = { + "name": str(arg.name), + "dtype": str(arg.decl_type), + "default": "" + } + if re.findall(r'[-+]?\d*\.\d+|\d+', _arg): + make_arguments['default'] = re.findall( + r'[-+]?\d*\.\d+|\d+', _arg)[0] + elif re.findall(r'\"(.+?)\"', _arg): + make_arguments['default'] = re.findall( + r'\"(.+?)\"', _arg)[0] + elif "true" in _arg: + make_arguments['default'] = "True" + elif "false" in _arg: + make_arguments['default'] = "False" + self.parsed_data['make']['arguments'].append( + make_arguments.copy()) + except RuntimeError: + self.parsed_data['make'] = {} + self.parsed_data['make']['arguments'] = [] + + # setters + try: + self.parsed_data['methods'] = [] + query_methods = declarations.access_type_matcher_t('public') + setters = main_class.member_functions(function=query_methods, + allow_empty=True, + header_file=self.target_file) + getter_arguments = [] + if setters: + for setter in setters: + if str(setter.name).startswith('set_') and setter.arguments: + setter_args = { + "name": str(setter.name), + "arguments_type": [] + } + for argument in setter.arguments: + args = { + "name": str(argument.name), + "dtype": str(argument.decl_type) + } + getter_arguments.append(args['name']) + setter_args['arguments_type'].append(args.copy()) + self.parsed_data['methods'].append(setter_args.copy()) + except RuntimeError: + self.parsed_data['methods'] = [] + + # getters + try: + self.parsed_data['properties'] = [] + query_properties = declarations.access_type_matcher_t('public') + getters = main_class.member_functions(function=query_properties, + allow_empty=True, + header_file=self.target_file) + if getters: + for getter in getters: + if not getter.arguments or getter.has_const: + getter_args = { + "name": str(getter.name), + "dtype": str(getter.return_type), + "read_only": True + } + if getter_args['name'] in getter_arguments: + getter_args["read_only"] = False + self.parsed_data['properties'].append( + getter_args.copy()) + except RuntimeError: + self.parsed_data['properties'] = [] + + # documentation + try: + _index = None + header_file = codecs.open(self.target_file, 'r', 'cp932') + self.parsed_data['docstring'] = re.compile( + r'//.*?$|/\*.*?\*/', re.DOTALL | re.MULTILINE).findall( + header_file.read())[2:] + header_file.close() + for doc in self.parsed_data['docstring']: + if Constants.BLOCKTOOL in doc: + _index = self.parsed_data['docstring'].index(doc) + if _index is not None: + self.parsed_data['docstring'] = self.parsed_data['docstring'][: _index] + except: + self.parsed_data['docstring'] = [] + + return self.parsed_data + + def run_blocktool(self): + """ Run, run, run. """ + self.get_header_info() diff --git a/gr-utils/python/blocktool/tests/README.blocktool_test b/gr-utils/python/blocktool/tests/README.blocktool_test new file mode 100644 index 0000000000..1ca66e9c6e --- /dev/null +++ b/gr-utils/python/blocktool/tests/README.blocktool_test @@ -0,0 +1,12 @@ +gr_blocktool: Block Header parsing tool. +Parses GNU Radio header files to generate YAML or JSON output. + +This directory consists of test for the parsed header output and Blocktool Exceptions. +============================================================================================ +- Schema defined in the test will be strictly followed for every parsed JSON output file. + +Two sample response for header files are available in this directory: +===================================================================== + +- sample_agc2_cc.json for public header file agc2_cc.h from gr-analog directory. +- sample_additive_scrambler_bb.json for public header file additive_scrambler_bb.h from gr-digital directory.
\ No newline at end of file diff --git a/gr-utils/python/blocktool/tests/sample_json/analog_agc2_cc.json b/gr-utils/python/blocktool/tests/sample_json/analog_agc2_cc.json new file mode 100644 index 0000000000..fa9eae3551 --- /dev/null +++ b/gr-utils/python/blocktool/tests/sample_json/analog_agc2_cc.json @@ -0,0 +1,131 @@ +{ + "namespace": [ + "gr", + "analog" + ], + "class": "agc2_cc", + "io_signature": { + "input": { + "signature": "make", + "min_streams": "1", + "max_streams": "1", + "sizeof_stream_item": "sizeof(gr_complex)" + }, + "output": { + "signature": "make", + "min_streams": "1", + "max_streams": "1", + "sizeof_stream_item": "sizeof(gr_complex)" + } + }, + "message_port": { + "input": [], + "output": [] + }, + "make": { + "arguments": [ + { + "name": "attack_rate", + "dtype": "float", + "default": "0.10000000000000001" + }, + { + "name": "decay_rate", + "dtype": "float", + "default": "0.01" + }, + { + "name": "reference", + "dtype": "float", + "default": "1" + }, + { + "name": "gain", + "dtype": "float", + "default": "1" + } + ] + }, + "methods": [ + { + "name": "set_attack_rate", + "arguments_type": [ + { + "name": "rate", + "dtype": "float" + } + ] + }, + { + "name": "set_decay_rate", + "arguments_type": [ + { + "name": "rate", + "dtype": "float" + } + ] + }, + { + "name": "set_reference", + "arguments_type": [ + { + "name": "reference", + "dtype": "float" + } + ] + }, + { + "name": "set_gain", + "arguments_type": [ + { + "name": "gain", + "dtype": "float" + } + ] + }, + { + "name": "set_max_gain", + "arguments_type": [ + { + "name": "max_gain", + "dtype": "float" + } + ] + } + ], + "properties": [ + { + "name": "attack_rate", + "dtype": "float", + "read_only": true + }, + { + "name": "decay_rate", + "dtype": "float", + "read_only": true + }, + { + "name": "reference", + "dtype": "float", + "read_only": false + }, + { + "name": "gain", + "dtype": "float", + "read_only": false + }, + { + "name": "max_gain", + "dtype": "float", + "read_only": false + } + ], + "docstring": [ + "/*!\n * \\brief high performance Automatic Gain Control class with\n * attack and decay rates.\n * \\ingroup level_controllers_blk\n *\n * \\details\n * For Power the absolute value of the complex number is used.\n */", + "// gr::analog::agc2_cc::sptr", + "/*!\n * Build a complex value AGC loop block with attack and decay rates.\n *\n * \\param attack_rate the update rate of the loop when in attack mode.\n * \\param decay_rate the update rate of the loop when in decay mode.\n * \\param reference reference value to adjust signal power to.\n * \\param gain initial gain value.\n */", + "/* namespace analog */", + "/* namespace gr */", + "/* INCLUDED_ANALOG_AGC2_CC_H */" + ] +}
\ No newline at end of file diff --git a/gr-utils/python/blocktool/tests/sample_json/digital_additive_scrambler_bb.json b/gr-utils/python/blocktool/tests/sample_json/digital_additive_scrambler_bb.json new file mode 100644 index 0000000000..265b5a6ddc --- /dev/null +++ b/gr-utils/python/blocktool/tests/sample_json/digital_additive_scrambler_bb.json @@ -0,0 +1,95 @@ +{ + "namespace": [ + "gr", + "digital" + ], + "class": "additive_scrambler_bb", + "io_signature": { + "input": { + "signature": "make", + "min_streams": "1", + "max_streams": "1", + "sizeof_stream_item": "sizeof(unsigned char)" + }, + "output": { + "signature": "make", + "min_streams": "1", + "max_streams": "1", + "sizeof_stream_item": "sizeof(unsigned char)" + } + }, + "message_port": { + "input": [], + "output": [] + }, + "make": { + "arguments": [ + { + "name": "mask", + "dtype": "int", + "default": "" + }, + { + "name": "seed", + "dtype": "int", + "default": "" + }, + { + "name": "len", + "dtype": "int", + "default": "" + }, + { + "name": "count", + "dtype": "int", + "default": "0" + }, + { + "name": "bits_per_byte", + "dtype": "int", + "default": "1" + }, + { + "name": "reset_tag_key", + "dtype": "std::string const &", + "default": "" + } + ] + }, + "methods": [], + "properties": [ + { + "name": "mask", + "dtype": "int", + "read_only": true + }, + { + "name": "seed", + "dtype": "int", + "read_only": true + }, + { + "name": "len", + "dtype": "int", + "read_only": true + }, + { + "name": "count", + "dtype": "int", + "read_only": true + }, + { + "name": "bits_per_byte", + "dtype": "int", + "read_only": true + } + ], + "docstring": [ + "/*!\n * \\ingroup coding_blk\n *\n * \\brief\n * Scramble an input stream using an LFSR.\n *\n * \\details\n * This block scrambles up to 8 bits per byte of the input\n * data stream, starting at the LSB.\n *\n * The scrambler works by XORing the incoming bit stream by the\n * output of the LFSR. Optionally, after \\p count bits have been\n * processed, the shift register is reset to the \\p seed value.\n * This allows processing fixed length vectors of samples.\n *\n * Alternatively, the LFSR can be reset using a reset tag to\n * scramble variable length vectors. However, it cannot be reset\n * between bytes.\n *\n * For details on configuring the LFSR, see gr::digital::lfsr.\n */", + "// gr::digital::additive_scrambler_bb::sptr", + "/*!\n * \\brief Create additive scrambler.\n *\n * \\param mask Polynomial mask for LFSR\n * \\param seed Initial shift register contents\n * \\param len Shift register length\n * \\param count Number of bytes after which shift register is reset, 0=never\n * \\param bits_per_byte Number of bits per byte\n * \\param reset_tag_key When a tag with this key is detected, the shift register is reset (when this is set, count is ignored!)\n */", + "/* namespace digital */", + "/* namespace gr */", + "/* INCLUDED_DIGITAL_ADDITIVE_SCRAMBLER_BB_H */" + ] +}
\ No newline at end of file diff --git a/gr-utils/python/blocktool/tests/sample_yaml/analog_agc2_cc.yml b/gr-utils/python/blocktool/tests/sample_yaml/analog_agc2_cc.yml new file mode 100644 index 0000000000..fef19eff95 --- /dev/null +++ b/gr-utils/python/blocktool/tests/sample_yaml/analog_agc2_cc.yml @@ -0,0 +1,67 @@ +id: analog_agc2_cc +label: AGC2 +category: '[Analog]' +flags: '[python, cpp]' +templates: + imports: from gnuradio import analog + make: analog.agc2_cc(${attack_rate}, ${decay_rate}, ${reference}, ${gain}, ${max_gain}) + callbacks: !!python/tuple + - set_attack_rate(${rate}) + - set_decay_rate(${rate}) + - set_reference(${reference}) + - set_gain(${gain}) + - set_max_gain(${max_gain}) +parameters: +- id: attack_rate + label: Attack_rate + dtype: float + read_only: true +- id: decay_rate + label: Decay_rate + dtype: float + read_only: true +- id: reference + label: Reference + dtype: float + read_only: false +- id: gain + label: Gain + dtype: float + read_only: false +- id: max_gain + label: Max_gain + dtype: float + read_only: false +inputs: +- domain: stream + dtype: sizeof(gr_complex) +outputs: +- domain: stream + dtype: sizeof(gr_complex) +cpp_templates: + includes: '#include <gnuradio/analog/agc2_cc.h>' + declartions: analog::agc2_cc::sptr ${id} + make: this->${id} = analog::agc2_cc::make(${attack_rate}, ${decay_rate}, ${reference}, + ${gain}, ${max_gain}) + callbacks: !!python/tuple + - set_attack_rate(${rate}) + - set_decay_rate(${rate}) + - set_reference(${reference}) + - set_gain(${gain}) + - set_max_gain(${max_gain}) + link: gnuradio-analog +documentation: +- "/*!\n * \\brief high performance Automatic Gain Control class with\n * + attack and decay rates.\n * \\ingroup level_controllers_blk\n *\n * + \\details\n * For Power the absolute value of the complex number is used.\n + \ */" +- // gr::analog::agc2_cc::sptr +- "/*!\n * Build a complex value AGC loop block with attack and decay rates.\n + \ *\n * \\param attack_rate the update rate of the loop when in attack + mode.\n * \\param decay_rate the update rate of the loop when in decay mode.\n + \ * \\param reference reference value to adjust signal power to.\n * + \\param gain initial gain value.\n */" +- /* namespace analog */ +- /* namespace gr */ +- /* INCLUDED_ANALOG_AGC2_CC_H */ +file_format: 1 diff --git a/gr-utils/python/blocktool/tests/sample_yaml/digital_additive_scrambler_bb.yml b/gr-utils/python/blocktool/tests/sample_yaml/digital_additive_scrambler_bb.yml new file mode 100644 index 0000000000..0001653273 --- /dev/null +++ b/gr-utils/python/blocktool/tests/sample_yaml/digital_additive_scrambler_bb.yml @@ -0,0 +1,63 @@ +id: digital_additive_scrambler_bb +label: ADDITIVE SCRAMBLER +category: '[Digital]' +flags: '[python, cpp]' +templates: + imports: from gnuradio import digital + make: digital.additive_scrambler_bb(${mask}, ${seed}, ${len}, ${count}, ${bits_per_byte}) +parameters: +- id: mask + label: Mask + dtype: int + read_only: true +- id: seed + label: Seed + dtype: int + read_only: true +- id: len + label: Len + dtype: int + read_only: true +- id: count + label: Count + dtype: int + read_only: true +- id: bits_per_byte + label: Bits_per_byte + dtype: int + read_only: true +inputs: +- domain: stream + dtype: sizeof(unsigned char) +outputs: +- domain: stream + dtype: sizeof(unsigned char) +cpp_templates: + includes: '#include <gnuradio/digital/additive_scrambler_bb.h>' + declartions: digital::additive_scrambler_bb::sptr ${id} + make: this->${id} = digital::additive_scrambler_bb::make(${mask}, ${seed}, ${len}, + ${count}, ${bits_per_byte}) + link: gnuradio-digital +documentation: +- "/*!\n * \\ingroup coding_blk\n *\n * \\brief\n * Scramble an input + stream using an LFSR.\n *\n * \\details\n * This block scrambles up + to 8 bits per byte of the input\n * data stream, starting at the LSB.\n *\n + \ * The scrambler works by XORing the incoming bit stream by the\n * output + of the LFSR. Optionally, after \\p count bits have been\n * processed, the shift + register is reset to the \\p seed value.\n * This allows processing fixed length + vectors of samples.\n *\n * Alternatively, the LFSR can be reset using a + reset tag to\n * scramble variable length vectors. However, it cannot be reset\n + \ * between bytes.\n *\n * For details on configuring the LFSR, see gr::digital::lfsr.\n + \ */" +- // gr::digital::additive_scrambler_bb::sptr +- "/*!\n * \\brief Create additive scrambler.\n *\n * \\param mask + \ Polynomial mask for LFSR\n * \\param seed Initial shift register contents\n + \ * \\param len Shift register length\n * \\param count Number of + bytes after which shift register is reset, 0=never\n * \\param bits_per_byte + Number of bits per byte\n * \\param reset_tag_key When a tag with this key + is detected, the shift register is reset (when this is set, count is ignored!)\n + \ */" +- /* namespace digital */ +- /* namespace gr */ +- /* INCLUDED_DIGITAL_ADDITIVE_SCRAMBLER_BB_H */ +file_format: 1 diff --git a/gr-utils/python/blocktool/tests/test_blocktool.py b/gr-utils/python/blocktool/tests/test_blocktool.py new file mode 100644 index 0000000000..f8dd9798fd --- /dev/null +++ b/gr-utils/python/blocktool/tests/test_blocktool.py @@ -0,0 +1,199 @@ +# +# Copyright 2019 Free Software Foundation, Inc. +# +# This file is part of GNU Radio +# +# GNU Radio is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 3, or (at your option) +# any later version. +# +# GNU Radio is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with GNU Radio; see the file COPYING. If not, write to +# the Free Software Foundation, Inc., 51 Franklin Street, +# Boston, MA 02110-1301, USA. +# +""" unittest for gr-blocktool api """ + +from __future__ import print_function +from __future__ import absolute_import +from __future__ import unicode_literals + +import os +import unittest +import warnings +try: + import pygccxml + SKIP_BLOCK_TEST = False +except: + SKIP_BLOCK_TEST = True + +try: + import apt + CACHE = apt.cache.Cache() + CACHE.open() + + PKG = CACHE['castxml'] + if PKG.is_installed: + SKIP_BLOCK_TEST = False + else: + SKIP_BLOCK_TEST = True +except: + SKIP_BLOCK_TEST = True + +from jsonschema import validate + +from blocktool import BlockHeaderParser +from blocktool.core.base import BlockToolException +from blocktool.core import Constants +from blocktool import RESULT_SCHEMA + + +class TestBlocktoolCore(unittest.TestCase): + """ The Tests for blocktool core """ + + def __init__(self, *args, **kwargs): + super(TestBlocktoolCore, self).__init__(*args, **kwargs) + self.module = os.path.abspath(os.path.join(os.path.dirname(__file__), + '../../../../gr-analog')) + self.test_dir = os.path.abspath(os.path.join(self.module, + 'include/gnuradio/analog')) + + def is_int(self, number): + """ + Check for int conversion + """ + try: + int(number) + return True + except ValueError: + return False + + @classmethod + def setUpClass(cls): + """ create a temporary Blocktool object """ + try: + warnings.simplefilter("ignore", ResourceWarning) + except NameError: + pass + test_path = {} + target_file = os.path.abspath(os.path.join(os.path.dirname( + __file__), '../../../../gr-analog/include/gnuradio/analog', 'agc2_cc.h')) + test_path['file_path'] = target_file + cls.test_obj = BlockHeaderParser(**test_path).get_header_info() + + @unittest.skipIf(SKIP_BLOCK_TEST, 'pygccxml not found, skipping this unittest') + def test_blocktool_exceptions(self): + """ + tests for blocktool exceptions + """ + # test for non-existent header or invalid headers + test_dict = {} + test_dict['file_path'] = os.path.abspath( + os.path.join(self.test_dir, 'sample.h')) + with self.assertRaises(BlockToolException): + BlockHeaderParser(**test_dict).run_blocktool() + # test for invalid header file + test_dict['file_path'] = os.path.abspath( + os.path.join(self.test_dir, 'CMakeLists.txt')) + if not os.path.basename(test_dict['file_path']).endswith('.h'): + with self.assertRaises(BlockToolException): + BlockHeaderParser(**test_dict).run_blocktool() + + @unittest.skipIf(SKIP_BLOCK_TEST, 'pygccxml not found, skipping this unittest') + def test_namespace(self): + """ test for header namespace """ + module_name = os.path.basename(self.module) + self.assertTrue(self.test_obj['namespace'][0] == 'gr') + self.assertTrue(self.test_obj['namespace'] + [1] == module_name.split('-')[-1]) + + @unittest.skipIf(SKIP_BLOCK_TEST, 'pygccxml not found, skipping this unittest') + def test_io_signature(self): + """ test for io_signature """ + input_signature = self.test_obj['io_signature']['input']['signature'] + output_signature = self.test_obj['io_signature']['output']['signature'] + valid_signature = False + if input_signature and output_signature in Constants.SIGNATURE_LIST: + valid_signature = True + self.assertTrue(valid_signature) + valid_io_stream = False + input_max = self.test_obj['io_signature']['input']['max_streams'] + input_min = self.test_obj['io_signature']['input']['min_streams'] + output_max = self.test_obj['io_signature']['output']['max_streams'] + output_min = self.test_obj['io_signature']['output']['min_streams'] + if self.is_int(input_max) and self.is_int(input_min) and self.is_int(output_max) and self.is_int(output_min): + valid_io_stream = True + self.assertTrue(valid_io_stream) + + @unittest.skipIf(SKIP_BLOCK_TEST, 'pygccxml not found, skipping this unittest') + def test_message_port(self): + """ test for message ports """ + input_port = self.test_obj['message_port']['input'] + output_port = self.test_obj['message_port']['output'] + valid_input_message_port = True + valid_output_message_port = True + if input_port: + for port in input_port: + if not port['id']: + valid_input_message_port = False + if output_port: + for port in output_port: + if not port['id']: + valid_output_message_port = False + self.assertTrue(valid_input_message_port) + self.assertTrue(valid_output_message_port) + + @unittest.skipIf(SKIP_BLOCK_TEST, 'pygccxml not found, skipping this unittest') + def test_factory_signature(self): + """ test for factory signature in the header """ + valid_factory_arg = True + if self.test_obj['make']['arguments']: + for arguments in self.test_obj['make']['arguments']: + if not arguments['name'] or not arguments['dtype']: + valid_factory_arg = False + self.assertTrue(valid_factory_arg) + + @unittest.skipIf(SKIP_BLOCK_TEST, 'pygccxml not found, skipping this unittest') + def test_methods(self): + """ test for methods """ + valid_method = True + if self.test_obj['methods']: + for arguments in self.test_obj['methods']: + if not arguments['name']: + valid_method = False + if arguments['arguments_type']: + for args in arguments['arguments_type']: + if not args['name'] or not args['dtype']: + valid_method = False + self.assertTrue(valid_method) + + @unittest.skipIf(SKIP_BLOCK_TEST, 'pygccxml not found, skipping this unittest') + def test_properties(self): + """ test for properties """ + valid_properties = True + if self.test_obj['properties']: + for arguments in self.test_obj['properties']: + if not arguments['name'] or not arguments['dtype']: + valid_properties = False + self.assertTrue(valid_properties) + + @unittest.skipIf(SKIP_BLOCK_TEST, 'pygccxml not found, skipping this unittest') + def test_result_format(self): + """ test for parsed blocktool output format """ + valid_schema = False + try: + validate(instance=self.test_obj, schema=RESULT_SCHEMA) + valid_schema = True + except BlockToolException: + raise BlockToolException + self.assertTrue(valid_schema) + + +if __name__ == '__main__': + unittest.main() diff --git a/gr-utils/python/blocktool/tests/test_json_file.py b/gr-utils/python/blocktool/tests/test_json_file.py new file mode 100644 index 0000000000..1d79aa78ca --- /dev/null +++ b/gr-utils/python/blocktool/tests/test_json_file.py @@ -0,0 +1,53 @@ +# +# Copyright 2019 Free Software Foundation, Inc. +# +# This file is part of GNU Radio +# +# GNU Radio is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 3, or (at your option) +# any later version. +# +# GNU Radio is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with GNU Radio; see the file COPYING. If not, write to +# the Free Software Foundation, Inc., 51 Franklin Street, +# Boston, MA 02110-1301, USA. +# +""" testing the JSON files generated by gr-blocktool """ + +from __future__ import print_function +from __future__ import absolute_import +from __future__ import unicode_literals + +import sys +import json +import jsonschema + +from blocktool import RESULT_SCHEMA + + +def is_valid(): + """ Validate json file """ + + with open(sys.argv[1], 'r') as json_file: + data = json.load(json_file) + try: + print("Validating...") + jsonschema.validate(data, RESULT_SCHEMA) + except jsonschema.ValidationError as exception: + print("Record JSON file # {}: NOT OK".format(sys.argv[1])) + raise Exception(exception) + else: + print("Record JSON file # {}: OK".format(sys.argv[1])) + + +if __name__ == '__main__': + if len(sys.argv) == 2: + is_valid() + else: + raise Exception('Please input only one json file') diff --git a/gr-utils/python/modtool/cli/makeyaml.py b/gr-utils/python/modtool/cli/makeyaml.py index 621d444958..8e38f9a3a0 100644 --- a/gr-utils/python/modtool/cli/makeyaml.py +++ b/gr-utils/python/modtool/cli/makeyaml.py @@ -24,14 +24,27 @@ from __future__ import print_function from __future__ import absolute_import from __future__ import unicode_literals +import os import click -from ..core import get_block_candidates, ModToolMakeYAML +try: + from gnuradio.blocktool import BlockHeaderParser + from gnuradio.blocktool.core.base import BlockToolException +except ImportError: + have_blocktool = False +else: + have_blocktool = True + +from ..core import get_block_candidates, ModToolMakeYAML, yaml_generator from ..tools import SequenceCompleter from .base import common_params, block_name, run, cli_input @click.command('makeyaml', short_help=ModToolMakeYAML.description) +@click.option('-b', '--blocktool', is_flag=True, + help='Use blocktool support to print yaml output. FILE PATH mandatory if used.') +@click.option('-o', '--output', is_flag=True, + help='If given, a file with desired output format will be generated') @common_params @block_name def cli(**kwargs): @@ -42,10 +55,24 @@ def cli(**kwargs): Note: This does not work on python blocks """ kwargs['cli'] = True - self = ModToolMakeYAML(**kwargs) - click.secho("GNU Radio module name identified: " + self.info['modname'], fg='green') - get_pattern(self) - run(self) + if kwargs['blocktool']: + kwargs['modtool'] = True + if kwargs['blockname'] is None: + raise BlockToolException('Missing argument FILE PATH with blocktool flag') + kwargs['file_path'] = os.path.abspath(kwargs['blockname']) + if os.path.isfile(kwargs['file_path']): + parse_yml = BlockHeaderParser(**kwargs) + parse_yml.run_blocktool() + parse_yml.cli = True + parse_yml.yaml = True + yaml_generator(parse_yml, **kwargs) + else: + raise BlockToolException('Invalid file path.') + else: + self = ModToolMakeYAML(**kwargs) + click.secho("GNU Radio module name identified: " + self.info['modname'], fg='green') + get_pattern(self) + run(self) def get_pattern(self): """ Get the regex pattern for block(s) to be parsed """ diff --git a/gr-utils/python/modtool/core/__init__.py b/gr-utils/python/modtool/core/__init__.py index eb9386cdf6..d895e88d12 100644 --- a/gr-utils/python/modtool/core/__init__.py +++ b/gr-utils/python/modtool/core/__init__.py @@ -27,7 +27,7 @@ from .base import ModTool, ModToolException, get_block_candidates from .add import ModToolAdd from .disable import ModToolDisable from .info import ModToolInfo -from .makeyaml import ModToolMakeYAML +from .makeyaml import ModToolMakeYAML, yaml_generator from .newmod import ModToolNewModule from .rm import ModToolRemove from .rename import ModToolRename diff --git a/gr-utils/python/modtool/core/add.py b/gr-utils/python/modtool/core/add.py index e74509f930..2ec57f4d2a 100644 --- a/gr-utils/python/modtool/core/add.py +++ b/gr-utils/python/modtool/core/add.py @@ -161,7 +161,7 @@ class ModToolAdd(ModTool): return try: append_re_line_sequence(self._file['cmlib'], - 'list\(APPEND test_{}_sources.*\n'.format(self.info['modname']), + r'list\(APPEND test_{}_sources.*\n'.format(self.info['modname']), 'qa_{}.cc'.format(self.info['blockname'])) append_re_line_sequence(self._file['qalib'], '#include.*\n', @@ -183,7 +183,7 @@ class ModToolAdd(ModTool): return try: append_re_line_sequence(self._file['cmlib'], - 'list\(APPEND test_{}_sources.*\n'.format(self.info['modname']), + r'list\(APPEND test_{}_sources.*\n'.format(self.info['modname']), 'qa_{}.cc'.format(self.info['blockname'])) self.scm.mark_files_updated((self._file['cmlib'],)) except IOError: @@ -255,7 +255,7 @@ class ModToolAdd(ModTool): if re.search('#include', oldfile): append_re_line_sequence(self._file['swig'], '^#include.*\n', include_str) else: # I.e., if the swig file is empty - regexp = re.compile('^%\{\n', re.MULTILINE) + regexp = re.compile(r'^%\{\n', re.MULTILINE) oldfile = regexp.sub('%%{\n%s\n' % include_str, oldfile, count=1) with open(self._file['swig'], 'w') as f: f.write(oldfile) diff --git a/gr-utils/python/modtool/core/base.py b/gr-utils/python/modtool/core/base.py index b9009f4891..55b6f48076 100644 --- a/gr-utils/python/modtool/core/base.py +++ b/gr-utils/python/modtool/core/base.py @@ -198,7 +198,7 @@ class ModTool(object): for f in files: if os.path.isfile(f) and f == 'CMakeLists.txt': with open(f) as filetext: - if re.search('find_package\(Gnuradio', filetext.read()) is not None: + if re.search(r'find_package\(Gnuradio', filetext.read()) is not None: self.info['version'] = '36' # Might be 37, check that later has_makefile = True elif re.search('GR_REGISTER_COMPONENT', filetext.read()) is not None: diff --git a/gr-utils/python/modtool/core/disable.py b/gr-utils/python/modtool/core/disable.py index 1fb8d4a830..034e028724 100644 --- a/gr-utils/python/modtool/core/disable.py +++ b/gr-utils/python/modtool/core/disable.py @@ -95,7 +95,7 @@ class ModToolDisable(ModTool): as well as the block magic """ with open(self._file['swig']) as f: swigfile = f.read() - (swigfile, nsubs) = re.subn('(.include\s+"({}/)?{}")'.format( + (swigfile, nsubs) = re.subn(r'(.include\s+"({}/)?{}")'.format( self.info['modname'], fname), r'//\1', swigfile) if nsubs > 0: diff --git a/gr-utils/python/modtool/core/makeyaml.py b/gr-utils/python/modtool/core/makeyaml.py index 05903238b6..dc506e64ce 100644 --- a/gr-utils/python/modtool/core/makeyaml.py +++ b/gr-utils/python/modtool/core/makeyaml.py @@ -28,12 +28,45 @@ import os import re import glob import logging +import yaml + +from collections import OrderedDict + +try: + from yaml import CLoader as Loader, CDumper as Dumper +except: + from yaml import Loader, Dumper + +try: + from gnuradio.blocktool.core import Constants +except ImportError: + have_blocktool = False +else: + have_blocktool = True from ..tools import ParserCCBlock, CMakeFileEditor, ask_yes_no, GRCYAMLGenerator from .base import ModTool, ModToolException + logger = logging.getLogger(__name__) +## setup dumper for dumping OrderedDict ## +_MAPPING_TAG = yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG + + +def dict_representer(dumper, data): + """ Representer to represent special OrderedDict """ + return dumper.represent_dict(data.items()) + + +def dict_constructor(loader, node): + """ Construct an OrderedDict for dumping """ + return OrderedDict(loader.construct_pairs(node)) + + +Dumper.add_representer(OrderedDict, dict_representer) +Loader.add_constructor(_MAPPING_TAG, dict_constructor) + class ModToolMakeYAML(ModTool): """ Make YAML file for GRC block bindings """ @@ -55,7 +88,8 @@ class ModToolMakeYAML(ModTool): # This portion will be covered by the CLI if not self.cli: self.validate() - logger.warning("Warning: This is an experimental feature. Don't expect any magic.") + logger.warning( + "Warning: This is an experimental feature. Don't expect any magic.") # 1) Go through lib/ if not self.skip_subdirs['lib']: if self.info['version'] in ('37', '38'): @@ -106,10 +140,10 @@ class ModToolMakeYAML(ModTool): file_exists = True logger.warning("Warning: Overwriting existing GRC file.") grc_generator = GRCYAMLGenerator( - modname=self.info['modname'], - blockname=blockname, - params=params, - iosig=iosig + modname=self.info['modname'], + blockname=blockname, + params=params, + iosig=iosig ) grc_generator.save(path_to_yml) if file_exists: @@ -120,7 +154,8 @@ class ModToolMakeYAML(ModTool): ed = CMakeFileEditor(self._file['cmgrc']) if re.search(fname_yml, ed.cfile) is None and not ed.check_for_glob('*.yml'): logger.info("Adding GRC bindings to grc/CMakeLists.txt...") - ed.append_value('install', fname_yml, to_ignore_end='DESTINATION[^()]+') + ed.append_value('install', fname_yml, + to_ignore_end='DESTINATION[^()]+') ed.write() self.scm.mark_files_updated(self._file['cmgrc']) @@ -138,18 +173,21 @@ class ModToolMakeYAML(ModTool): 'std::vector<int>': 'int_vector', 'std::vector<float>': 'real_vector', 'std::vector<gr_complex>': 'complex_vector', - } + } if p_type in ('int',) and default_v is not None and len(default_v) > 1 and default_v[:2].lower() == '0x': return 'hex' try: return translate_dict[p_type] except KeyError: return 'raw' + def _get_blockdata(fname_cc): """ Return the block name and the header file name from the .cc file name """ - blockname = os.path.splitext(os.path.basename(fname_cc.replace('_impl.', '.')))[0] + blockname = os.path.splitext(os.path.basename( + fname_cc.replace('_impl.', '.')))[0] fname_h = (blockname + '.h').replace('_impl.', '.') - contains_modulename = blockname.startswith(self.info['modname']+'_') + contains_modulename = blockname.startswith( + self.info['modname']+'_') blockname = blockname.replace(self.info['modname']+'_', '', 1) return (blockname, fname_h, contains_modulename) # Go, go, go @@ -157,15 +195,171 @@ class ModToolMakeYAML(ModTool): (blockname, fname_h, contains_modulename) = _get_blockdata(fname_cc) try: parser = ParserCCBlock(fname_cc, - os.path.join(self.info['includedir'], fname_h), + os.path.join( + self.info['includedir'], fname_h), blockname, self.info['version'], _type_translate - ) + ) except IOError: - raise ModToolException("Can't open some of the files necessary to parse {}.".format(fname_cc)) + raise ModToolException( + "Can't open some of the files necessary to parse {}.".format(fname_cc)) if contains_modulename: return (parser.read_params(), parser.read_io_signature(), self.info['modname']+'_'+blockname) else: return (parser.read_params(), parser.read_io_signature(), blockname) + + +def yaml_generator(self, **kwargs): + """ + Generate YAML file from the block header file using blocktool API + """ + header = self.filename.split('.')[0] + block = self.modname.split('-')[-1] + label = header.split('_') + del label[-1] + yml_file = os.path.join('.', block+'_'+header+'.block.yml') + _header = (('id', '{}_{}'.format(block, header)), + ('label', ' '.join(label).upper()), + ('category', '[{}]'.format(block.capitalize())), + ('flags', '[python, cpp]') + ) + params_list = [ + '${'+s['name']+'}' for s in self.parsed_data['properties'] if self.parsed_data['properties']] + _templates = [('imports', 'from gnuradio import {}'.format(block)), + ('make', '{}.{}({})'.format(block, header, ', '.join(params_list))) + ] + + if self.parsed_data['methods']: + list_callbacks = [] + for param in self.parsed_data['methods']: + arguments = [] + for args in param['arguments_type']: + arguments.append(args['name']) + arg_list = ['${'+s+'}' for s in arguments if arguments] + list_callbacks.append( + param['name']+'({})'.format(', '.join(arg_list))) + callback_key = ('callbacks') + callbacks = (callback_key, tuple(list_callbacks)) + _templates.append(callbacks) + _templates = tuple(_templates) + + data = OrderedDict() + for tag, value in _header: + data[tag] = value + + templates = OrderedDict() + for tag, value in _templates: + templates[tag] = value + data['templates'] = templates + + parameters = [] + for param in self.parsed_data['properties']: + parameter = OrderedDict() + parameter['id'] = param['name'] + parameter['label'] = param['name'].capitalize() + parameter['dtype'] = param['dtype'] + parameter['read_only'] = param['read_only'] + parameters.append(parameter) + if parameters: + data['parameters'] = parameters + + input_signature = [] + max_input_port = self.parsed_data['io_signature']['input']['max_streams'] + i_sig = self.parsed_data['io_signature']['input']['signature'] + for port in range(0, int(max_input_port)): + input_sig = OrderedDict() + if i_sig is Constants.MAKE: + input_sig['domain'] = 'stream' + input_sig['dtype'] = self.parsed_data['io_signature']['input']['sizeof_stream_item'] + elif i_sig is Constants.MAKE2: + input_sig['domain'] = 'stream' + input_sig['dtype'] = self.parsed_data['io_signature']['input']['sizeof_stream_item' + + str(port+1)] + elif i_sig is Constants.MAKE3: + input_sig['domain'] = 'stream' + input_sig['dtype'] = self.parsed_data['io_signature']['input']['sizeof_stream_item' + + str(port+1)] + elif i_sig is Constants.MAKEV: + input_sig['domain'] = 'stream' + input_sig['dtype'] = self.parsed_data['io_signature']['input']['sizeof_stream_items'] + input_signature.append(input_sig) + + if self.parsed_data['message_port']['input']: + for _input in self.parsed_data['message_port']['input']: + m_input_sig = OrderedDict() + m_input_sig['domain'] = 'message' + m_input_sig['id'] = _input + input_signature.append(m_input_sig) + if input_signature: + data['inputs'] = input_signature + + output_signature = [] + max_output_port = self.parsed_data['io_signature']['output']['max_streams'] + o_sig = self.parsed_data['io_signature']['output']['signature'] + for port in range(0, int(max_output_port)): + output_sig = OrderedDict() + if o_sig is Constants.MAKE: + output_sig['domain'] = 'stream' + output_sig['dtype'] = self.parsed_data['io_signature']['output']['sizeof_stream_item'] + elif o_sig is Constants.MAKE2: + output_sig['domain'] = 'stream' + output_sig['dtype'] = self.parsed_data['io_signature']['output']['sizeof_stream_item' + + str(port+1)] + elif o_sig is Constants.MAKE3: + output_sig['domain'] = 'stream' + output_sig['dtype'] = self.parsed_data['io_signature']['output']['sizeof_stream_item' + + str(port+1)] + elif o_sig is Constants.MAKEV: + output_sig['domain'] = 'stream' + output_sig['dtype'] = self.parsed_data['io_signature']['output']['sizeof_stream_items'] + output_signature.append(output_sig) + + if self.parsed_data['message_port']['output']: + for _output in self.parsed_data['message_port']['output']: + m_output_sig = OrderedDict() + m_output_sig['domain'] = 'message' + m_output_sig['id'] = _output + output_signature.append(m_output_sig) + if output_signature: + data['outputs'] = output_signature + + _cpp_templates = [('includes', '#include <gnuradio/{}/{}>'.format(block, self.filename)), + ('declarations', '{}::{}::sptr ${{id}}'.format(block, header)), + ('make', 'this->${{id}} = {}::{}::make({})'.format( + block, header, ', '.join(params_list))) + ] + + if self.parsed_data['methods']: + list_callbacks = [] + for param in self.parsed_data['methods']: + arguments = [] + for args in param['arguments_type']: + arguments.append(args['name']) + arg_list = ['${'+s+'}' for s in arguments if arguments] + list_callbacks.append( + param['name']+'({})'.format(', '.join(arg_list))) + callback_key = ('callbacks') + callbacks = (callback_key, tuple(list_callbacks)) + _cpp_templates.append(callbacks) + + link = ('link', 'gnuradio-{}'.format(block)) + _cpp_templates.append(link) + _cpp_templates = tuple(_cpp_templates) + + cpp_templates = OrderedDict() + for tag, value in _cpp_templates: + cpp_templates[tag] = value + data['cpp_templates'] = cpp_templates + + if self.parsed_data['docstring'] is not None: + data['documentation'] = self.parsed_data['docstring'] + data['file_format'] = 1 + + if kwargs['output']: + with open(yml_file, 'w') as yml: + yaml.dump(data, yml, Dumper=Dumper, default_flow_style=False) + else: + print(yaml.dump(data, Dumper=Dumper, allow_unicode=True, + default_flow_style=False, indent=4)) diff --git a/gr-utils/python/modtool/templates/templates.py b/gr-utils/python/modtool/templates/templates.py index f811dc610a..f272791ae4 100644 --- a/gr-utils/python/modtool/templates/templates.py +++ b/gr-utils/python/modtool/templates/templates.py @@ -557,7 +557,7 @@ templates: # Make one 'parameters' list entry for every parameter you want settable from the GUI. # Keys include: -# * id (makes the value accessible as \$keyname, e.g. in the make entry) +# * id (makes the value accessible as keyname, e.g. in the make entry) # * label (label shown in the GUI) # * dtype (e.g. int, float, complex, byte, short, xxx_vector, ...) parameters: |