Lexing arbitrarily large input programs

Frank-Rene Schäfer — Wed, 22 Dec 2021 12:02:06 -0000

Did you have a look a 'demo/Cpp/11-ManualBufferFilling', there it is explained how a feeder or gavager can help out. The example 'lexer-point.cpp' shows how simply to point to a piece of allocated memory. For plain C look at 'demo/C/11-ManualBufferFilling'.

Lexing arbitrarily large input programs

Axe Roland — Wed, 22 Dec 2021 07:00:31 -0000

Hi! I really want to use Quex in a project, but I'm having a problem with lexing large input programs. The buffer always fills up to the buffer limit specified in the configuration, but I want to be able to support arbitrary file sizes. This happens even if I create a ByteLoader from a file. Quex just loads it up to the buffer limit.

I'm writing this in C, and using a UTF-8 converter. For context, I am given the entire program as a char* string in memory (not a file!). The lexing works perfectly for smaller programs, the buffer size is the only problem. I have tried changing up the source code to not use the default setting, and instead malloc based on an argument I pass, but end up with weird memory double-freeing issues when I call the destructor.

This is essentially what my code looks like at the moment:

blah quex_lexer;
quex_Converter* converter = ...;
blah_from_ByteLoader(&quex_lexer, NULL, converter);
quex_lexer.buffer.fill(&quex_lexer.buffer, program_string, program_string + program_length);
receive_tokens(...);
converter->destruct(&converter);
blah_destruct(&quex_lexer);

Is there something that I'm missing here? Or some stable way around the issue? All I need is for Quex to take into account the entire string I give it, rather than having a fixed maximum size decided at compilation :)

Thanks!

No such file or directory during demos compilation

Frank-Rene Schäfer — Sat, 29 Dec 2018 20:20:07 -0000

This error should no longer occur with the versions > 0.69.1

No such file or directory during demos compilation

Silvio Manuel Silva Santos — Fri, 08 Sep 2017 12:32:02 -0000

Hi, I'm trying to compile the demos from 0.67.5 but all I get are endless "no such file or directory"
I installed:
Python python-2.7.13.amd64.msi
Quex: quex-0.67.5-windows-installer.exe
MinGW: g++ (TDM-2 mingw32) 4.4.1

Here the output when i try to compile demo 000:

C:\quex\quex-0.67.5\demo\Cpp\000>mingw32-make
quex -i                 simple.qx  \
             -o                 EasyLexer  \
                 --token-id-offset  0x1000
g++ -ggdb -pedantic -Wconversion -Wall -Werror -W -c -I./ -IC:/quex/quex-0.67.5/   ../example.cpp -o example.o
../example.cpp:4:21: error: EasyLexer: No such file or directory
In file included from C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/iconv/Converter_IConv:8,
                 from ../example.cpp:5:
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/iconv/special_headers.h:25:22: error: iconv.h: No such file or directory
In file included from C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/iconv/Converter_IConv.i:18,
                 from ../example.cpp:6:
C:/quex/quex-0.67.5/quex/code_base/analyzer/configuration/validation:14:5: error: #error "Either little endian or big endian must be defined."
In file included from C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/iconv/Converter_IConv:7,
                 from ../example.cpp:5:
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/Converter:13: error: expected constructor, destructor, or type conversion before 'typedef'
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/Converter:99: error: expected constructor, destructor, or type conversion before '(' token
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/Converter:101: error: 'QUEX_NAME' declared as an 'inline' variable
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/Converter:101: error: 'Converter_construct' was not declared in this scope
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/Converter:101: error: expected ',' or ';' before '(' token
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/Converter:118: error: variable or field 'QUEX_NAME' declared void
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/Converter:118: error: 'Converter_reset' was not declared in this scope
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/Converter:119: error: variable or field 'QUEX_NAME' declared void
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/Converter:119: error: 'Converter_print_this' was not declared in this scope
In file included from C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/iconv/Converter_IConv:8,
                 from ../example.cpp:5:
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/iconv/special_headers.h:24: error: expected constructor, destructor, or type conversion before 'extern'
In file included from ../example.cpp:5:
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/iconv/Converter_IConv:13: error: expected constructor, destructor, or type conversion before 'typedef'
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/iconv/Converter_IConv:18: error: expected constructor, destructor, or type conversion before '(' token
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/iconv/Converter_IConv:20: error: ISO C++ forbids declaration of 'QUEX_NAME' with no type
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/iconv/Converter_IConv:20: error: 'QUEX_NAME' declared as an 'inline' variable
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/iconv/Converter_IConv:20: error: conflicting declaration 'int QUEX_NAME'
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/Converter:101: error: 'QUEX_NAME' has a previous declaration as 'bool QUEX_NAME'
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/iconv/Converter_IConv:20: error: 'Converter' was not declared in this scope
In file included from ../example.cpp:6:
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/iconv/Converter_IConv.i:7: error: expected constructor, destructor, or type conversion before 'extern'
In file included from ../example.cpp:6:
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/iconv/Converter_IConv.i:22: error: expected constructor, destructor, or type conversion before 'inline'
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/iconv/Converter_IConv.i:27: error: 'QUEX_NAME' declared as an 'inline' variable
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/iconv/Converter_IConv.i:27: error: redefinition of 'bool QUEX_NAME'
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/Converter:101: error: 'bool QUEX_NAME' previously defined here
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/iconv/Converter_IConv.i:27: error: 'Converter_IConv_initialize_by_bom_id' was not declared in this scope
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/iconv/Converter_IConv.i:31: error: 'QUEX_NAME' declared as an 'inline' variable
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/iconv/Converter_IConv.i:31: error: conflicting declaration 'E_LoadResult QUEX_NAME'
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/Converter:101: error: 'QUEX_NAME' has a previous declaration as 'bool QUEX_NAME'
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/iconv/Converter_IConv.i:31: error: 'Converter_IConv_convert' was not declared in this scope
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/iconv/Converter_IConv.i:37: error: variable or field 'QUEX_NAME' declared void
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/iconv/Converter_IConv.i:37: error: 'Converter_IConv_delete_self' was not declared in this scope
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/iconv/Converter_IConv.i:40: error: 'QUEX_NAME' declared as an 'inline' variable
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/iconv/Converter_IConv.i:40: error: conflicting declaration 'ptrdiff_t QUEX_NAME'
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/Converter:101: error: 'QUEX_NAME' has a previous declaration as 'bool QUEX_NAME'
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/iconv/Converter_IConv.i:40: error: 'Converter_IConv_stomach_byte_n' was not declared in this scope
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/iconv/Converter_IConv.i:43: error: variable or field 'QUEX_NAME' declared void
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/iconv/Converter_IConv.i:43: error: 'Converter_IConv_stomach_clear' was not declared in this scope
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/iconv/Converter_IConv.i:46: error: variable or field 'QUEX_NAME' declared void
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/iconv/Converter_IConv.i:46: error: 'Converter_IConv_print_this' was not declared in this scope
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/iconv/Converter_IConv.i:48: error: ISO C++ forbids declaration of 'QUEX_NAME' with no type
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/iconv/Converter_IConv.i:48: error: 'QUEX_NAME' declared as an 'inline' variable
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/iconv/Converter_IConv.i:48: error: conflicting declaration 'int QUEX_NAME'
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/Converter:101: error: 'QUEX_NAME' has a previous declaration as 'bool QUEX_NAME'
C:/quex/quex-0.67.5/quex/code_base/buffer/lexatoms/converter/iconv/Converter_IConv.i:48: error: 'Converter' was not declared in this scope
mingw32-make: *** [example.o] Error 1

Utf-8 in Quex 0.67.4

Frank-Rene Schäfer — Sat, 03 Jun 2017 13:22:13 -0000

Patrik, BTW,

you seem to be pretty familiar with 'cmake', at least more
than I am. There has been an issue with the demos
(in the '$QUEX_PATH/demo' subdirectories) which only
used 'gmake'. Would you be willing to contribute by
adding cmake-files for the demos? That would be a
tremendous improvement, because everybody could generate
the the complete build setup for his IDE.

Utf-8 in Quex 0.67.4

Frank-Rene Schäfer — Tue, 23 May 2017 10:20:49 -0000

I am not able to find it.

Utf-8 in Quex 0.67.4

Patrik — Tue, 23 May 2017 07:38:09 -0000

I've have updated the thread on stackoverflow with an example.

Utf-8 in Quex 0.67.4

Patrik — Mon, 15 May 2017 14:24:03 -0000

I've updated the stackoverflow thread with some more information.

Utf-8 in Quex 0.67.4

Patrik — Wed, 10 May 2017 09:31:21 -0000

Hi,

I've posted a new Quex related question on stack overflow:
https://stackoverflow.com/questions/43888553/utf-8-in-quex-0-67-4

Best regards,

Patrik J

LoopRestartP - undeclared identifier

Patrik — Tue, 09 May 2017 11:35:30 -0000

While attempting to create a minimal example for this bug I
accidentally found another bug as well:
https://sourceforge.net/p/quex/bugs/300/

Recent posts to Discussion

Lexing arbitrarily large input programs

Lexing arbitrarily large input programs

No such file or directory during demos compilation

No such file or directory during demos compilation

Utf-8 in Quex 0.67.4

Utf-8 in Quex 0.67.4

Utf-8 in Quex 0.67.4

Utf-8 in Quex 0.67.4

Utf-8 in Quex 0.67.4

LoopRestartP - undeclared identifier