It is currently Mon Jul 22, 2024 2:35 am


All times are UTC - 5 hours [ DST ]



Post new topic Reply to topic  [ 6 posts ] 
Author Message
 Post subject: Multiple G'MIC processes sharing memory?
PostPosted: Thu Feb 20, 2014 7:43 pm  (#1) 
Offline
GimpChat Member

Joined: Dec 03, 2013
Posts: 10
I have run into a strange issue. I am using G'MIC for a massively parallel image processor. On one 4-core machine, I have a process shelling out to G'MIC to perform some inpainting. There are four copies of this process running. 99% of the time, this works fine. However, over the course of processing many images over many hours, I will sometimes get an incorrect output based on the input. The cause seems to be from reading from another image, but when I look at the command G'MIC is running, it is pointing at the correct image and the image itself is correct!

This is difficult to explain, so I have created an example script that causes this to happen very quickly. I have tested on G'MIC 1.5.8.3 and 1.5.8.2. What I am doing here is creating 99 images, and then having 50 G'MIC processes spin up and inpaint the images, saving output in individual directories based on an integer I feed ARGV. I don't know if this issue arises on other functions, but I would be glad to test. Of course, I'm not going to be running 50 G'MIC processes on a 4-core machine, but this just forces the bug to happen in a single quick run (hopefully) rather than over the course of many hours.

I'm sorry I used ImageMagick to create the images! I'd like to know how to do so with G'MIC but I had trouble with the syntax :)

Simply save these files on a machine with Ruby, G'MIC, ImageMagick, and Ghostscript. Then run:

ruby debug.rb


I'd be happy to answer any questions.

Thanks!

Oh, I should note an easy way to find troublesome images is to go to the output directory and run:

ls -la * | awk '{print$5, $9}' | sort | uniq -c


This will group images by name and file size so you can identify the bad seeds from there. I got 12 'bad' images in my run.

# debug.rb (run me!)

# create an inpaint mask
`convert -background transparent -fill black -pointsize 72 -size 165x70 -gravity center label:+++++++++ i.gif`
`convert i.gif -alpha extract -alpha on i.gif`

# create a bunch of images where the filename is a number and the image is the number
(1..99).each do |n|
  `convert -background lightblue -fill blue -pointsize 72 -size 165x70 -gravity center label:#{n} i#{n}.gif`
end

# make a directory for output
Dir.mkdir("output")

# run 50 inpaint processes and feed it the process number
`seq 1 50 | xargs -P 50 -I {} ruby inpaint.rb {}`


# inpaint.rb

# create an array 1-99
images = *(1..99)

randomize the array
images.shuffle!

# make a directory for each argument (the process "id")
dir = Dir.mkdir("#{Dir.pwd}/output/#{ARGV[0]}")

# iterate through the array, inpainting the image with that number... ouput to output/process/inpaint-[#].png
images.each do |n|
  `gmic i#{n}.gif i.gif --inpaint[-2] [-1] -remove[0] -remove[0] -output output/#{ARGV[0]}/inpaint-#{n}.png`
end


Share on Facebook Share on Twitter Share on Orkut Share on Digg Share on MySpace Share on Delicious Share on Technorati
Top
 Post subject: Re: Multiple G'MIC processes sharing memory?
PostPosted: Mon Feb 24, 2014 2:49 pm  (#2) 
Offline
GimpChat Member

Joined: Dec 03, 2013
Posts: 10
Just an update:

I've tested with 1.5.8.4 (beta) with no change.


Top
 Post subject: Re: Multiple G'MIC processes sharing memory?
PostPosted: Mon Feb 24, 2014 3:54 pm  (#3) 
Offline
GimpChat Member
User avatar

Joined: Jan 03, 2011
Posts: 1656
Actually, I don't have any idea on what it behaves like this. Sure, it requires some investigations, but unfortunately, I *really* run out of time for the moment.


Top
 Post subject: Re: Multiple G'MIC processes sharing memory?
PostPosted: Mon Feb 24, 2014 4:13 pm  (#4) 
Offline
GimpChat Member

Joined: Dec 03, 2013
Posts: 10
No worries! I just wanted to bring it to your attention and provide whatever information I can. I just read the notes for the beta release and thought it might be worth a shot to run my tests against it. Thanks for all of you work on G'MIC!


Top
 Post subject: Re: Multiple G'MIC processes sharing memory?
PostPosted: Tue Feb 25, 2014 11:34 am  (#5) 
Offline
GimpChat Member

Joined: Sep 24, 2012
Posts: 275
Location: Scotland
Hmm this one sounds nasty to troubleshoot. I could try this out I guess but before I do I'm wondering some things...

If you run the individual gmic command which produces the faulty output (i.e. outside the script as a single command, with the relevant input image) does it produce correct output?

Also if you try altering the gmic part as such:

`gmic i#{n}.gif i.gif -inpaint[-2] [-1] -output[-2] output/#{ARGV[0]}/inpaint-#{n}.png`


It might be interesting to know if it does the same; it should use less memory per process, the idea being if it's some kind of sharing problem the results will be different. Perhaps still incorrect but it would prove a point.


Top
 Post subject: Re: Multiple G'MIC processes sharing memory?
PostPosted: Tue Feb 25, 2014 1:48 pm  (#6) 
Offline
GimpChat Member

Joined: Dec 03, 2013
Posts: 10
Running the command individually works fine.

Removing the -remove commands didn't make things any better. In fact, it had a lot more 'errors' than my original run. However, I have not done multiple runs so take this with a grain of salt.


Top
Post new topic Reply to topic  [ 6 posts ] 

All times are UTC - 5 hours [ DST ]


   Similar Topics   Replies 
No new posts Attachment(s) You Got Something - In Memory Of JJ Cale

6

No new posts MULTIPLE GIDES

3

No new posts Attachment(s) add multiple guides

5

No new posts Multiple Continuous Droste GIF

1

No new posts Attachment(s) SETUP MULTIPLE GUIDES

2



* Login  



Powered by phpBB3 © phpBB Group