Prints hanging at same GCode line with Teacup + Gen7

Prints are hanging at the same line of GCode every time, very similar to this thread. I can print a 20mm square of one layer thickness fine, but other bigger shapes I have tried will hang.

My setup: a Huxley with home made Gen7 electronics and the latest commit 131d7390c00e9fda19be of Teacup. I use Skeinforge 40 with retraction (M101 and M103 commands commented out so no firmware retraction), Pronterface to send the prints, and a Dell laptop running Windows XP.

Thoughts so far:

• Communications or interference issues can likely be eliminated due to the precise repeatability of the issue.
• A different laptop running Kubuntu still experiences the same problem.
• Pronterface controls my Mendel with Gen3 electronics and FiveD firmware no problem, printing the same shapes (but with lower feed rates).

I can't get Teacup to compile in Arduino 022 with debugging enabled, but the problem would appear to be the firmware not liking the GCode. I'm not sure if enabling debugging would help diagnosis? On a vanilla setup, this is the error I get trying to compile with debug enabled:

gcode_process.c: In function 'process_gcode_command':
gcode_process.c:758: error: 'DDA' has no member named 'c'
gcode_process.c:762: error: 'DDA' has no member named 'c'

On the advice of the post linked above, I have tried reducing the STEPS_PER_MM by putting all axes to 1/2 stepping but the freezes still occur at 30mm/s. Config.h settings:

STEPS_PER_MM_X = 10.186
STEPS_PER_MM_Y = 10.186
STEPS_PER_MM_Z = 200
STEPS_PER_MM_E = 65.423

MAXIMUM_FEEDRATE_X = 6000
MAXIMUM_FEEDRATE_Y = 6000
MAXIMUM_FEEDRATE_Z = 540
MAXIMUM_FEEDRATE_E = 240

The different shapes don't freeze in the same or similar places in the GCode either:

butterfly.gcode

G1 X43.76 Y74.52 Z0.3 F1800.0 E0.018
G1 X29.45 Y60.22 Z0.3 F1800.0 E0.46
G1 X29.6 Y.59.72 Z0.3 F1800.0 E0.012
G1 X29.54 Y59.67 Z0.3 F1800.0 E0.002     <--Last line completed (1017)
G1 F240.0
G1 E-1.5
G1 F1800.0
;M103
G1 X29.37 Y60.77 Z0.3 F2700.0

i.e. hanging on an E retract after line 1017 (despite having performed many successfully before this point)

star.gcode

G1 X78.06 Y36.73 Z0.3 F1800.0 E0.537
G1 X77.93 Y60.36 Z0.3 F1800.0 E0.537
G1 X96.5 Y75.0 Z0.3 F1800.0 E0.537
G1 X77.93 Y89.64 Z0.3 F1800.0 E0.537
G1 X78.06 Y113.27 Z0.3 F1800.0 E0.537
G1 X55.05 Y107.89 Z0.3 F1800.0 E0.537
G1 X36.65 Y122.73 Z0.3 F1800.0 E0.537    <-- Last line completed (160)
G1 X26.51 Y101.37 Z0.3 F1800.0 E0.537
G1 X3.44 Y96.24 Z0.3 F1800.0 E0.537
G1 X13.81 Y75.0 Z0.3 F1800.0 E0.537
...etc

i.e. in the middle of 14 consecutive moves of equal distance and not near an E retract or restart.

Desperate for suggestions, anyone have any ideas? I had a quick look at using Sprinter instead but have no idea how to set it up for Gen 7. Is there an alternative firmware to use that would fix this problem, is it set up for Gen 7 and if not how would I do that?

Thanks!

try enabling ramping acceleration. I committed a fix to restore c to the DDA regardless of acceleration settings. I'm not sure how you're printing at all with that bug in place!

As for the pauses, teacup has an 8 move buffer by default, so it'll actually be choking on the move 8 lines up from the last one sent. Also, M111 S2 will enable a ton of debug output which may help figure out what's going on but may give your host talker a conniption- try it and see how you go.

Another interesting possibility is to enable XON/XOFF in teacup, then start your favourite serial terminal, enable XON/XOFF in it too then simply paste your gcode file in. Teacup will automatically limit the flow of data using the XON/XOFF protocol.

---

-----------------------------------------------
Wooden Mendel
Teacup Firmware

FWIW, I get these occasional lockups on a Gen2 as well. No extruder, no heater, no thermistors, no G-code related to that. G-code is sent by a shell tool similar to sender.sh.

They're really rare and so far I cheat by simply hitting the enter key, which continues the build without further issues. It's likely some missing "ok" response.

---

Generation 7 Electronics	Teacup Firmware	RepRap DIY

I've been thinking about this.. perhaps we need to add the line number to ok responses, so if the hostware doesn't get a response for too long it can retry?

---

-----------------------------------------------
Wooden Mendel
Teacup Firmware

There's a fairly simple host protocol used in Bluetooth called BCSP that would be very easy to port across. It handles lost packets, stalls and re-transmits.

Regards,

James

---

--
Check out my blog: AdventuresIn3-DPrinting

Problem solved, turns out I had left XON/XOFF enabled from my original testing with a serial terminal. Oops!

I had ramping acceleration on the whole time, and only saw the DDA bug when trying to enable debugging. The change you made has fixed that, I can compile with debugging (although fingers crossed no need now)

Thanks for the speedy responses.

In my investigations I did come across several other threads e.g. here and here where lots of people had communications problems, although I think they are running different setups. If the cost of dealing with these comms problems isn't too high, it seems it would save a lot of people some headaches.

the thing that bugs me is that XON/XOFF should never trigger unless there's a /single/ line of gcode with 47 (64 * 3/4 - 1) or more characters. I find skeinforge almost never produces them, so I wonder what's going on...

---

-----------------------------------------------
Wooden Mendel
Teacup Firmware

Hello!

I have the same issue with my Gen7 (self built) and latest Teacup (master and Gen7 branches). So I decided to try and debug the thing. Here is what I got (as the topic says my print always stops at the same place):

...
> G1 X69.029 Y50.572 Z4.67 F1800.0 E566.8152
< {DDA_CREATE: [212,742,0,18] [ts:30,ds:792] }
< Q7/6F
< ok
> G1 X68.92 Y49.933 Z4.67 F1800.0 E566.8302
< {DDA_CREATE: [-521,9,0,12] [ts:21,ds:516] }
< Q0/7F
< ok
> G1 X68.035 Y50.109 Z4.67 F1800.0 E566.851
< {DDA_CREATE: [-109,-639,0,15] [ts:26,ds:650] }
< Q1/0F
< ok
> G1 X68.499 Y50.572 Z4.67 F1800.0 E566.8661
< {DDA_CREATE: [-885,176,0,21] [ts:36,ds:912] }
< Q2/1F
< ok
> G1 X67.969 Y50.572 Z4.67 F1800.0 E566.8783
< {DDA_CREATE: [464,463,0,15] [ts:19,ds:638] }
< Q3/2F
< ok
> G1 X67.391 Y49.995 Z4.67 F1800.0 E566.8971
< {DDA_CREATE: [-530,0,0,12] [ts:21,ds:521] }
< Q4/3F
< Pos: 69.956,50.433,4.670,-559.358,1800
< Dst: 69.656,49.608,4.670,566.776,1800
< Q4/3F
< Pos: 69.956,50.433,4.670,-557.131,1800
< Dst: 69.656,49.608,4.670,566.776,1800
< Q4/3F
< Pos: 69.956,50.433,4.670,-555.960,1800
< Dst: 69.656,49.608,4.670,566.776,1800
< Q4/3F
< Pos: 69.956,50.433,4.670,-555.036,1800
< Dst: 69.656,49.608,4.670,566.776,1800
< Q4/3F
...

So this (Pos:, Dst:, Q4/3F) sequence goes on for many more lines it the log (I guess it has looped somewhere indefinitely) with Dst being the same, while Pos: is changing all the time. As I am quite new to Teacup code and structure, so it might take a while before I figure out what to do next. In the mean time if anyone with more knowledge of the firmware has any ideas on where to insert reasonable debug, please feel free to share. I really would like to nail this one down to save the frustration for many

Quote

So I decided to try and debug the thing.

Wow! That's the right attitude!

Everything involved with serial communications is in serial.c, so you can concentrate on that. Looking at your log it looks more like the serial code isn't even touched, though.

< Pos: 69.956,50.433,4.670,-555.960,1800
< Dst: 69.656,49.608,4.670,566.776,1800

More than 1000 mm to go for the E axis. Doesn't look quite right.

---

Generation 7 Electronics	Teacup Firmware	RepRap DIY

A little update on the issue. I've let the print ran for some time and to my surprise the log became normal after some 5000 lines of the (Pos: Dst sequence.

Quote
Traumflug

< Pos: 69.956,50.433,4.670,-555.960,1800
< Dst: 69.656,49.608,4.670,566.776,1800

More than 1000 mm to go for the E axis. Doesn't look quite right.

Yeap. And the other interesting thing I've noticed is that E-axis value is slowly increasing up to zero and then up to 559.484 and right in this moment things start to be normal again.

I have to confess that after three ruined prints in a row I am now testing this issue only with my motors detached from the Gen7, so no movement actually happen, but that doesn't stop the issue from reappearing right at the same place all the time (well, no big wonder actually).

Traumflug, could you please elaborate a little bit more on your idea to start looking in serial.c What do you think is the cause of the issue? I mean what should I be looking for? I have some experience with embedded development which might help.

If E is calculated to move more than 1000 mm it's unlikely this is an issue of the serial code. Looks much more like a variable overflow in dda_create() in dda.c. Watch out how many steps the E motor should do (calculate it by hand) and how many steps are in the queue variable at the end of dda_create() and dda_start(). The debug statement in line 260 should tell you this, check also wether the values are kept until dda_start().

dda_create() looks a bit confusing at the first glance, because it implements three different acceleration strategies, ACCELERATION_REPRAP, ACCELERATION_RAMPING and none. There are #ifdefs.

The lifetime of a movement is simple. The G-code interpreter queues up the movement in micrometers and then calls dda_create() to fill in all the acceleration and step count stuff. Then the movement sits there untouched, waiting to be activated in dda_start(). Once it was activated dda_step() is called repeatedly, once for each step of the axis with the most steps, until the movement is done.

---

Generation 7 Electronics	Teacup Firmware	RepRap DIY

Traumflug, thank you for your guidance on this one! I am glad to say that together we've been able to hunt this bad boy down It is indeed an int32_t overflow.

So the story goes like this. After you've said I should look into dda.c for E steps, I've checked the code and figured out what 'ts' stands for in the log. Then I looked at my log a bit upwards (from the place I've posted) and spotted this interesting snippet ('ts' goes crazy!!!):

...
70886 > G1 X70.347 Y50.299 Z4.67 F1800.0 E566.7464
70887 < {DDA_CREATE: [-266,264,0,8] [ts:11,ds:365] }
70888 < Q2/1F
70889 < ok
70890 > G1 X69.972 Y50.454 Z4.67 F1800.0 E566.7558
70891 < {DDA_CREATE: [-629,432,0,18] [ts:25,ds:780] }
70892 < Q3/2F
70893 < ok
70894 > G1 X69.656 Y49.608 Z4.67 F1800.0 E566.7766
70895 < {DDA_CREATE: [-375,155,0,9] [ts:15,ds:421] }
70896 < Q4/3F
70897 < ok
70898 > G1 X69.338 Y49.821 Z4.67 F1800.0 E566.7854
70899 < {DDA_CREATE: [-316,-846,0,21] [ts:429487,ds:935] }     <--------- CRAZY ts
70900 < Q5/4F
70901 < ok
70902 > G1 X69.55 Y50.563 Z4.67 F1800.0 E566.8032
70903 < {DDA_CREATE: [-318,213,0,9] [ts:12,ds:392] }
...

I've added some log to checked that it is really E which is causing total_steps go that high. When it has confirmed, I've traced all the dda->e related values. dda->e_delta came back really big. And the reason for that was in the 'steps' variable, which is calculated like this:

um_to_steps_e(steps, target->E);

Turns out the value of 566776 is critical for target->E, because target->E is signed int (only couple billions one way) and overflows when is multiplied by (STEPS_PER_M_E/100L), my STEPS_PER_M_E = 378900

So the problem is in implementation of um_to_steps_e macro. It also would explain the suggestions like decreasing the microstepping (as it would effectively decrease the STEPS_PER_M_E).

As a quick fix for everyone with the issue - switching to E-relative should fix it, as the overflow will never happen. Unless you'll get the same thing with X,Y,Z if you have a huge bed and your extruder travels from one corner to another.

Unfortunately, I don't feel myself being able to provide a rock solid patch for the issue at the moment, as I am not quite familiar with the overall firmware. So I better leave it to the guys who wrote it.

Quote

I am not quite familiar with the overall firmware.

You don't have to. Just fixing this macro is sufficient.

That said, thanks for tracking down the issue. Looks like this needs review again. Likely replacing the macro with a function this time, as the community acceptance goes away from editing a configuration before uploading to editing it after that.

---

Generation 7 Electronics	Teacup Firmware	RepRap DIY

Ok. If you say so, I'll try to come up with a fix. Do I need to send a patch or should I create my own branch in the git tree and someone will merge it in master later?

After googling for about the entire day, I've finally found a suitable algorithm on stackoverflow: [stackoverflow.com]. See also prettyfied in the attached file; it runs on my PC so far, but I see no reason why it shouldn't work on an ATmega, too. It's a combined multiply-divide operation which works without playing foul ( = using floats or 64 bit maths). Too late today to move it into Teacup, though.

The file contains lots of test code, too. Compile and run it with

gcc overflow.c -o overflow && ./overflow

Quote

Do I need to send a patch or should I create my own branch in the git tree and someone will merge it in master later?

While I have the habit to never merge and cherry-pick commits instead, do what you think is most convenient. Sending patches here or sending commits to a fork on Github or whatever.


P.S.: Sorry if we have some duplicate work now.

Edited 1 time(s). Last edit at 04/22/2012 05:23PM by Traumflug.

---

Generation 7 Electronics	Teacup Firmware	RepRap DIY

Here is the patch. Haven't tested it on a real print yet. Could you please take a look at it and merge it into the git (it is for Gen7 branch)?

The funny thing is, it also has fixed my problem with very-very slow homing and interrupts in the homing procedure (homing would stop half way to the end stop always). Turns out it was due to the same int overflow!

I'll let you know when I'll make the test.

Thanks, Andrey. Let me add my current state of the work, too, see two attachements. One patch to get muldiv() into place, another one to make use of it. Should fit on top of the Gen7 branch, also totally untested.

Looking at your patch I see two things I don't understand:

- What is this __attribute__((hot)) ? Looks like some caching hint, but ATmegas have no cache.

- What is this business with the remainder?

---

Generation 7 Electronics	Teacup Firmware	RepRap DIY

Traumflug Wrote:
-------------------------------------------------------

>
> - What is this __attribute__((hot)) ? Looks like
> some caching hint, but ATmegas have no cache.

Optimisation hint for the compiler - gcc specific I believe.

See [gcc.gnu.org]

---

Most of my technical comments should be correct, but is THIS one ?
Anyway, as a rule of thumb, always double check what people write.

Quote
Traumflug
- What is this __attribute__((hot)) ? Looks like some caching hint, but ATmegas have no cache.

It is a hint for the gcc:

Quote
http://gcc.gnu.org/onlinedocs/gcc/Function-Attributes.html
hot
The hot attribute is used to inform the compiler that a function is a hot spot of the compiled program. The function is optimized more aggressively and on many target it is placed into special subsection of the text section so all hot functions appears close together improving locality.

I think the function is "hot" in this sense indeed. ATmega architecture is quite new to me but even if this won't do anything good here it might help on other architectures (like ARM). Anyway, if you think it is useless, please fell free to remove it - I trust your judgment.

Quote
Traumflug
- What is this business with the remainder?

I am sorry, but I don't quite understand what you mean, could you please be more specific?

Some comments to your patch:

1. I like your idea of starting new dda_math. Also, to be more consistent, I think we should move following functions there as well: approx_distance, approx_distance_3, int_sqrt, msbloc. But unfortunately this doesn't play well with my optimization, which is explained below. So I prefer keeping muldiv local in dda.c.

2. muldiv MUST have first parameter and return type as signed int otherwise there will be problems. The reason is following. Although steps variable in dda_create is uint32_t, startpoint_steps.X (YZE) are int32 and muldiv is used to calculate them as well in dda_new_startpoint. The value of these variables is subtracted later in dda_create, so the sign really does matter.

3. The second and third parameters of muldiv are always the same (within one axis), so their ratios (/ and %) are the same too. And it seems to me to be an unjustified waste of processor time to calculate these ratios every time muldiv is called (division is expensive). That is why I've introduced new variables that would store the values of the ratios for all four axes and be passed into muldiv as parameters. Of course, this optimization is achieved at the cost of memory usage and code readability, but I believe the footprint isn't that big compared to performance gain.

Edited 1 time(s). Last edit at 04/26/2012 08:02AM by andrey2345.

Quote

2. muldiv MUST have first parameter and return type as signed int

That's true. Well observed!

Quote

there as well: approx_distance, approx_distance_3,

Also true. Can be done in future commits.

I think you can drop this part:

         //INT32_MIN will remain to be INT32_MIN event after negation!!!
         if (multiplicand == INT32_MIN)
             multiplicand = INT32_MAX;
         else

The comment is true, but it widens the valid range by just one micrometer.

I think this:

     return negative_flag == 0 ? quotient : -quotient;

should be replaced by

     return negative_flag == 0 ? (int32_t)quotient : -((int32_t)quotient);

Quote

3. The second and third parameters of muldiv are always the same (within one axis), so their ratios (/ and %) are the same too. And it seems to me to be an unjustified waste of processor time to calculate these ratios every time muldiv is called (division is expensive).

Also true. We need both, though, as muldiv() fits nicely in a number of timer/feedrate/acceleration calculations as well. How about:

int32_t muldiv(int32_t multiplicand, uint32_t multiplier, uint32_t divisor) {
  muldivQR(multiplicand, multiplier / divisor, multiplier % divisor, divisor);
}

int32_t muldivQR(int32_t multiplicand, uint32_t qn, uint32_t rn, uint32_t divisor) {
...

Regarding optimisation: I wouldn't make too much headache about this. dda_start() is called once per move only, not once per step. Reliability of the calculations is much more important, so code simplification has it's advantages, too. If you want to optimise, search for "plain wrong" in dda.c, you'll find it around line 420. Big time waster there

---

Generation 7 Electronics	Teacup Firmware	RepRap DIY

Quote
Traumflug
I think you can drop this part:

      //INT32_MIN will remain to be INT32_MIN event after negation!!!
         if (multiplicand == INT32_MIN)
             multiplicand = INT32_MAX;
         else

The comment is true, but it widens the valid range by just one micrometer.

If you don't really mind I'd like to keep it there. One comparison won't take too much, but on the other hand we'll have a rock solid function that anyone can use without double thinking if it is going to behave properly at the limits. Like you plan to use it somewhere else.

Quote
Traumflug
Also true. Can be done in future commits.

Only if you promise not to forget to do so

Quote
Traumflug
should be replaced by

return negative_flag == 0 ? (int32_t)quotient : -((int32_t)quotient);

True! And I was thinking may be declaring quotient as int32_t would be even better?

Well if you have plans for muldiv somewhere else, of course it is a good idea to implement is this way!

As for the optimization I'd like to keep it (well I am biased on this one ). For two reasons:
1. First of all we are working with small devices here such things play bigger role than on desktops. Of course there still should be a balance between optimization and maintainability. That is why I've tried to comment those things. And we also should respect optimization and not to sacrifice it every time it threatens maintainability.
2. It would be nice to have the optimization-culture in this firmware to avoid "plain wrong" things

Anyway, I guess it is your decision to make.

Hallo, Traumflug!

I have tested the fix and the print went without a problem.

So are you going to merge the changes or do I have to do something else? Please let me know. I would like to end this story.

I'm going to merge it. I'm also busy with the Gen7 v1.4 release at the moment, so this woll take another week or two. Just follow the repo on Github and you'll see it.

---

Generation 7 Electronics	Teacup Firmware	RepRap DIY

Ok! Good to hear that! Good luck with Gen7 1.4

Hi all!

for me too andrey2345's patch fixed the problem, which occured all the time…! Many many thanks!…

---

François

Finally(!) it's in the official repository. Incremental movements were tested to be accurate uo to 210'000mm, so you can print an entire spool now without overflowing these variables.

For now on the Gen7 branch, until a few additional success reports come back: [github.com] . It's the most recent 7 commits: [github.com]

The implementation is pretty much what we discussed here, without this "optimisation", though. For now, STEPS_PER_M_{XYZE} are compile time variables, so there's no need to store them in RAM, too. I left a note there pointing to this discussion.

Thanks a ton, Andrey.

---

Generation 7 Electronics	Teacup Firmware	RepRap DIY

Markus, thanks to have merged Andrey's patch!

But but but… same problem again: a new print hanging at the same Gcode line… with these two firmware versions:

-patched commit 85e29e8c91
-recent commit f2383d542d

I rounded some E values near the hanging place into the Gcode (I don't know the exact place because of Teacup's buffer) and the problem was solved…

With those two firmware versions, the problem occurs rarely. With the unpatched commit 85e29e8c91 it occured very very frequently…

---

François

Ah, still not perfect? Tse tse

What do you mean with "rounding some E values"? Removing a few trailing digits?

Also, a G-code file exposing the hang and the "fixed" version of it would be welcome. Ideally together with your config.h, so I can duplicate your setup. Only demonstratable issues can be fixed

---

Generation 7 Electronics	Teacup Firmware	RepRap DIY

Traumflug Wrote:
-------------------------------------------------------
> Ah, still not perfect? Tse tse

perfection is annoying some times…

> What do you mean with "rounding some E values"?
> Removing a few trailing digits?

Yes, from 3 to 2 digits with rounding: E32.327 => E32.33 for example

> Also, a G-code file exposing the hang and the
> "fixed" version of it would be welcome. Ideally
> together with your config.h, so I can duplicate
> your setup. Only demonstratable issues can be
> fixed

I zipped all that into the attachment. Finally I could find the hanging Gcode line, you'll find comments into the header of the file. I reproduced the hanging and it's fix several times…

For the little story Markus the file I am hanging/printing is your oozebane test… and hangings come with Early Shutdown Distance values different of the 1.20 default value (SK50), I am exploring that, there no link of course with the int overflow question, it's just funny…

---

François