First off Happy New Year to everyone on the board!

I've spent the last week getting to know the Mathematical Parser and I thought I'd bring up a few things missing that could make it more useful. I've tried to stick to what I believe is the original philosophy of what it was create to be. I'll start with the easier stuff and work out from there.

1. Create a means to put comments inside an expression set. When you have 100 lines of code in an expression, comments make it more usable for everyone else.

2. This one's a bit vague but some means of displaying a variable to aid with debugging. At the moment, I've been using a copy of python to run a converted copy of the code. There must be a better way.

3. There seems to be a need (at least for me!) for some sort of tuple-like variable. I have to admit that a two element one fits most of my needs. I understand that regular single value variables can fill in for almost everything that a tuple would do but having worked with the mathematical Parser without them, the code could be a lot more verbose with them. Maybe have a non-alphabetic first character would make it easier to identify such variables.

4. User defined functions. I've been playing with Mathmap code seeing what can be converted. One of the things Mathmap has is support for complex numbers that can get used in graphical transformation. Now it's true that most people won't want to do anything like this so I wouldn't expect any internal support for this but it would be good if I could create functions for my own usage. A model for this could be Pixel Bender such as this example:


In this example for Pixel Bender, this is a function definition for doing complex number multiplication. The float2 is a two member tuple as in #3 above. The ".x" part references the first entry in the tuple and the ".y" part the second entry. It takes in two variables (each a complex number - 2 entry tuple) and passes back a two entry tuple.

Again, all this can be done without user defied functions and tuples but look at what different code bases look like with this example that I had to deal with.

First this is the original Mathmap code:


The variables "z" and "xymiddle" are complex and "I" is the single unit imaginary number.

Next is the code that does the same thing with the Mathematical Parser:


Here I used the variables "z" and "zi" to hold the real and imaginary parts of "z" in the mathmap case. Similarly, "xymiddle" and "xymidlei" do the same thing. But it isn't really clear what is going on and I wouldn't want to give anyone else the task of maintaining it.

The third example is how Pixel Bender would do it with use of tuples and user defined functions:


Assuming that "complexMult", "complexDivision" and "complexExp" are defined, it is much clearer what is going on. Plus once the function is working OK, I can reuse it elsewhere with confidence.

Anyway, this posting has gone on way too long already so I'll stop now. What are your thoughts on this.

Souphead.

Hello Souphead,



This is already possible, just insert comments starting with '#', like in this example:



This is also already possible with the function 'print(x)', which prints the value of the expression 'x' on the stderr (by default), or to a file (if "Output messages" -> "logfile" has been selected). 'print(x)' also returns the value of 'x'.


This is an interesting remark. I don't think we really need explicit tuple-like structures, because the parser have already a lot of constructs allowing to deal with tuple variables:
- If you have plenty of variables values to manage, then you can create a 'blank' image before starting your expression evaluation, and use these empty slots as variable values in your code (by reading/setting pixel values in this blank image). If you have tuple with a lot of dimensions, then use a for() loops in your expression to fill your tuple values.
- Or you can create your own 'macros' able to define, set or read several values at the same time. I'll post an example later in the day if I have some time.
But this may be not very convenient at the end, I'll try to think about it.



Macros are possible in a math expression, simply by defining it as a regular G'MIC command that returns a status string. But again, this may be not very convenient to use.
I think I need to think a bit more about all this before having an opinion Thanks for your message.

Thanks for the response.

On #2, of course now I see the print(x) function in the text - not sure how I missed it. But if using G'MIC as a GIMP plugin, where does the output go?

Souphead.

By default, it goes on the terminal (stderr). But you can select "Output Messages -> Logfile" as well, and then it should be written in your logfile (in G'MIC resources folder / gimp_log).

Hello Souphead,

Some news on the (purely theoretical) advances I've made on this subject. Your message made me think a lot these two last days, I must admit it !
I'm trying to figure out how I could add the management of "tuples" and "custom functions" (or "macros") in the math parser. I've already tested some ideas but this is definitely not simple to implement, and I'm afraid in most cases it will slow down the evaluation by a non negligible factor.
The less cumbersome approach would be to allow a variable to be a 'pointer' to an array, and make each available function and operation working differently with a pointer or a regular value (for instance, the operator + between two pointers would sum each values in the two arrays).
This would be possible only for arrays with a fixed size (known during the compilation step, so basically a constant).
At the end, we could write things like that :



I don't think I'll add a way to make the vector 'dynamic' because it would require a dynamic memory allocation/deallocation procedure which is something very complex to achieve correctly.
But if these 'static' vectors work as expected, then of course, I would add some 'vector' or 'complex'-related functions, as 'dot(varA,varB)' or 'mul_complex(varA,varB)', etc..
And if, afterwards, I allow users to define their own custom functions (which could be really cool of course), then a function should be able to
get arguments passed either by value or by pointer. Not sure I'll be able to deal with recursive functions anyway (or maybe only if they have
a predictive end, known at compilation time).

There are still a lot of questions remaining, and I think I need to be sure of all the answers before starting to code something for real.
What do you think about it ?
The advantage of using 'pointers' for dealing with tuples is that it won't slow down the code when using 'regular' scalar values, but it will take a little more space to store a vector (we'll need the N values of the vector + 1 value to store the pointer).

First thanks for the consideration of my ideas. Sorry for making you think so hard over the last few days

I think your work on the mathematical parser rests on several pillars - fast to compile, fast to run and general purpose. While my suggestions have application with what I am doing at the moment, I only mentioned things that i could see being usable by others too.

I like what you have for vectors. A general purpose size one has lots of uses beyond a simple two element one (a four element xyzc one for example) and being able to easily reference individual elements with the [] notation looks good. I'm not sure I would use the pointer notation myself but then maybe I would! I don't see a major loss in not being able to dynamically define vector sizes either.

As for user defined functions/macros, if there are not USER defined ones, then you will be left with having to create and maintain them. I can see the benefit of certain high usage one being defined in C for maximum speed but but how often would someone want tan(z) - z being complex, for example? If functions could be user defined, perhaps a web page of user created functions could be created and anyone that needs something specific can cut and paste a copy if needed. If it doesn't already exist, they can either post for help to get it written or better still, research, code and post the new function to further expand the library.

Best for all it looks like should still be intact! Thanks for thinking about this.

Souphead.

Some news about the math evaluator in G'MIC.
I'm almost done with the support of static-size vectors in the expression evaluator. This was more complex to implement that I thought at a first glance, since I had to introduce hard typing mechanisms to make things coherent. I've also not strictly followed my initial plans about having 'pointers'.
So now, the math parser accepts the following constructs :



I admit this was a hard work, but this is very exciting at the same time, because it doesn't break the previous used syntax, and
introduce new vector variables to play with. It also means I have to add some useful functions to deal with vectors and complex numbers.
I've also added dot() for computing the dot product. I think I'll add functions to compute the complex modulus and argument.
Probably the cross product between two 3d vectors too.
If you have some ideas and suggestions for new useful functions working with vector-valued arguments, please let me know.

All in all, this also made me discover some optimizations to do in the math evaluator, so everything should be better now.
Still a lot of work to do on it (and perform some tests!). But this is in the good way.

I'll think about allowing the definition of user's own functions a bit later (probably at the beginning of February).

Gracious. Been awhile since I visited. Hope you are all well here in GimpChat land.

I'd like to touch a bit on one point which David (Ronounours) made in his post:

It so happens that there is a pretty mature (and heavily commented) example of using an image as a data structure in one of my more recent tutorials on color mapping in G'MIC.
Median Cut Algorithm
which the -colormap command utilizes to estimate the dominant colors of an image (if any). It is an algorithm based on a tree structure; what the tutorial calls a 'variance digest' tracks where the implementation is in a 'divide-and-conquer' scheme. The variance-digest lets the implementation identify which node in the tree exhibits the widest color spectrum variance, which then becomes the next candidate for splitting into two nodes where one or another color is likely to dominate.

There are a couple of advantages to using images as data structures. Trivially, you allocate them dynamically. Images come with a large host of properties and methods that quickly tell you about the most/least brightest pixel in an image, its location in the raster, various statistical image properties, and more. The i(...) and j(...) mathematical parser functions serve as both getters and setters into the raster, depending on which side of the assignment operator(=) they appear. Using images as data structures involve an interplay between the command processor, where some variant of -input fetches you storage, and image properties and methods, letting you access and change the raster, or observe some of its global properties. For linear algebra fans, images do double duty as matrices. You may, for example, find least squares solutions in images representing data points in some space. The G'MIC -svd command does most of the heavy lifting for you. Examples are forthcoming in scheduled tutorials. Stay tuned at G'MIC Tutorials. As David is aware, (having read through early drafts - Thank you, David!) there will be a series of math parser specific tutorials arriving in that space over the next couple of months. Stay tuned.

Before departing, I would also like to note that, in conjunction with the overall command parsing context, the mathematical parser does have access to a vector-like construct, though probably not as convenient as David is considering. But it is worth knowing about.

One may assign a list to a command line level variable, determine its size and get an element. The 'vector' is a variable that is created by the command parser, but such can be accessed within the math parser in separate math parsing contexts -- the command variable persists from context to context. An illustration:

Note the syntactical shell escaping of characters: \" and such. foo, a comma-separated list of numbers, came into existence for the command parser. It is visible in the first math parser context where we ask about its size. In the second context we ask for the third element. Note that variables created in the command context can persist from one to another math context, which can be useful at times. Images persist as well from one to the next math context. The caution about variables created within a math construct is that they cease to exist at the end of the context. Thus, these more persistent constructs, images and argument lists, furnished to the math environment by the command parser, are handy to have around.

Garry

As Garry says, the 'persistence' of arrays of values is something useful, and we have already a plenty of operators to act on vector/matrices seen as images.
At the same time, the math parser is often used as a "pixel processor" (I see it like a kind of 'shader' in OpenGL), so it can be considered as something more "low-level".
As Souphead said, it may be useful to have some way to deal with complex numbers in the math parser. I extended this idea to 'general arrays', but in practice I doubt
the vector-valued variables that are available in the math parser will be used to store full image data (this shouldn't be used for that).
I assume the new vectors will be useful to work with complex numbers, RGB triplets (or RGBA), etc..
Maybe I'll add a kind of import/export mechanism, to be able to import image values into a vector variable, or export a vector variable as a new image.

At this point, I can already say using the vectors as complex numbers allows to simplify codes a little bit. The following example of the Mandelbrot fractal rendering illustrates that.

Here is the code that render a simple Julia fractal, using the new math parser features (operations on complex numbers) :

that can be compared with the "old" code using the limited math features we had previously (so, no complex numbers):


Anyway, this has a cost in terms of performance : the command '-julia_new' takes 2.5x more time to compute in this case (343ms instead of 172ms).
Indeed, I can't afford having some 'optimization steps' in the compiler since it has to compile the expression in a fast way.
In the case we use the complex numbers, the total computation time is reasonnable anyway (and if needed, the "old" optimized code is still possible
to execute).

I just "came clean" with what I've been working on in another topic and I managed to get it all done by coding around what was missing. Having said that, everything you looks useful to me and even if the code runs a bit slower but is ten times clearer as to what is happening in it, it's just what I'm looking for.

Thanks,
Souphead.

Some news about the improvement of the math evaluator you asked for
I'm almost done with all my changes (took my a bit more than expected, something like 3000 lines of code).

I've done a quick summary of the advances here : https://discuss.pixls.us/t/gmic-math-ev ... ctions/656

Basically, we have now : vectors (and a bit of matrices) available in the math parser, as well as user-defined custom functions as well.

So you can write things like this for instance (here, mixing vector values and custom function) :



which does basically :
- Compute for each pixel, the norminf of the vector I = [ R,G,B], i.e. 'max(R,G,B)' in our case.
- Use this max to linearly interpolate between two colors (green and purple) and put the resulting color in the image at current location (x,y).

The number of things we can do now is really cool. So thanks for the suggestion (even if that made me work a lot ).

I have seen your postings on pixels.us though I try to post on only one site. Let me ask a few questions.

- Functions can have multiple statements? The example above has only one.

- Are variables local or global in scope inside the function?

-Function definitions happen before usage (ie above it in the code) but after the init(), right?

- Any view yet on the overhead?

I haven't got too far with adding them in to Droste (real like has got in the way) but I'll get onto it soon. If nothing else this should significantly reduce the number of lines of code.

Do you need (as apposed to want) some feedback before fully releasing 1.6.9? Also, any view of when 1.6.9 is being released? If you don't want to go public on it, I understand.

I have a few other thoughts on features - these are much smaller and maybe they are already there, somewhere.

Inside a "Do" loop, is there a way to CONTINUE (go directly to the test condition) or immediately BREAK from the loop?

Is there any direct handling of multiple choice logic either by an "elseif" or a switch-like control block? Nested 'if' blocks always work but they soon becomes unclear.

Sorry again about making you work a lot. Working a lot is good, right? Thanks again for all your efforts.

Souphead

Yes, you can have multiple statements, just as any expression. By the way, the example above has already two statements (there is a ';' delimitor if you look at it closely). Just put your multiple statements between parentheses, and you're done with it.


In fact, there is no 'scope' for the function. It's more a macro than as function to be honest. When the expression parser find a function call like 'foo(1,2,3)', it just replaces 'foo()' by its definition with the argument substituted into it. Nothing more.
I think this is good enough and do not slow down the expression parser too much. You have to take care of possible side effects of course, like :

This example returns 4, as 'foo(++z)' is replaced by '(++z) + (++z)' equivalent to '++z;++z;z+z' i.e. 4.

As these are macros rather than regular functions, it also avoids recursive calls. I know this is a limitation, but this is how it works for now.
Maybe in the future, I'll find an easy way to do the calls as regular functions, but if I do that, I'm sure this will be slower to execute
(I'll need to copy the functions arguments in a stack). I don't really see the advantages of having regular functions rather than macros right now
(except for recursive calls).



They can appear anywhere before they are used. Defining a function takes some time during the expression compilation, but nothing during the evaluation. I guess that writing them in the 'init()' section could be a good practice
Also, a function that has been already defined can be re-defined in another place (just like a macro), with a different signature.



Yes, that's the point. As I know this math parser is a JIT compiler, I've made a lot of efforts to reduce the overhead.
If you don't use custom functions at all, it costs nothing. Otherwise, it takes just a slight amount of time during the
expression compilation, nothing more during the evaluation itself (as they are macros, they don't have an existence
by themselves in the compiled bytecode).



Yes, I'd be interested by any feedback on the new math parser features. I'm trying to do some tests here, but I'm sure I'll miss a lot of weird configurations which may crash the math parser. Feel free to play with it.
I'll try to release new 'pre-release' packages this afternoon (GMT+1). If everything goes well, I'll try to release 1.6.9 maybe in one or two weeks
(I have to write some more docs also before releasing).



For that, I'd use something like this :



and same for the break:






Same here, the ternary operator allows to write things, that, when correcty indented, really look like a '-if...-elif...-else...-endif' bloc:

I've released some pre-versions of 1.6.9, for testing.
They are available here : http://gmic.eu/files/prerelease/

Feel free to test the new math evaluator features, and tell me if you find something strange or unexpected.

As long as I know the rules, the macro approach is more than good. I can see it being useful even if there is only one occurrence of a block of code for the sake of clarity.

Souphead.