email2trac borked

[sra@…]

Jen Linkova reported problems creating tickets via email. Both Lucy and I had some trouble reproducing, but that appears to be because of the twisty maze, not because it's not a real error. email2trac bombs:

<tickets@permatrac.noc.ietf.org>: Command died with status 1:
    "/usr/local/bin/run_email2trac --project=noc"

Cranking up logging in /usr/local/etc/email2trac.conf eventually yielded a backtrace:

Traceback (most recent call last):
  File "/usr/local/bin/email2trac", line 2955, in <module>
    tktparser.parse(sys.stdin)
  File "/usr/local/bin/email2trac", line 1864, in parse
    self.parse_subject_field(m, subject, spam_msg)
  File "/usr/local/bin/email2trac", line 1960, in parse_subject_field
    elif not self.ticket_update(m, result.group('reply')[:-1], spam_msg):
  File "/usr/local/bin/email2trac", line 1129, in ticket_update
    tkt.save_changes(self.author, body_text, when, None, str(cnum))
  File "/usr/local/lib/python2.7/site-packages/trac/ticket/model.py", line 390, in save_changes
    """, (self.id, when_ts, author, cnum, comment))
  File "/usr/local/lib/python2.7/site-packages/trac/db/api.py", line 155, in __exit__
    self.db.commit()
OperationalError: disk I/O error

Not sure what that's about yet, might be a strange way of saying "permission error". Most likely induced by an upgrade, as usual, but that's just conjecture.

[sra@…]

Status update before end-of-IETF-week consumes remaining brain cells:

• The error is down in the guts of SQLite3, specifically the Python SQLite3 library.
• All permissions, ownerships, setuid/setgid wrapper etc appear to be correct, so if this is a permission error, multiple sets of eyes have failed to see it.
• df reports plenty of disk space and plenty of inodes.
• We can't get a direct ktrace because of the setuid wrapper which comes from our binary-only package of this plugin.
• If we simulate the behavior of the wrapper with a five line Python script which launches the Python script under ktrace with the right UID and GID, the failure does not occur, so we get a trace but it's not useful.

permatrac is running one version back of the email2trac package from what my own servers run (2.10 vs 2.11). This may be the binary packaging again, email2trac may be unpopular enough that nobody has bothered regenerating the binary package. Or permatrac may just be behind, we chose not to try to update it on Thursday of an IETF week.

We *think* this was working earlier this week, but evidence is sketchy, the message Lucy sent me as an attempted proof of this appears to have gone directly from the replying user to Lucy's mailbox without actually passing through the ticket system. So inconclusive.

Current plan is to try reimplementing this without the wretched setuid wrapper, using the postfix pipe transport, which is what I do on my own servers. Won't necessarily solve the problem (although it might -- the data above could be the result of the wrapper doing something that's not obvious from reading its source code), but fewer UID changes required via that path, better control over exactly how the Python script starts up, and more opportunities to insert debugging without having to revert to source packages.

Current theory is to leave this alone for the remainder of IETF100 and get the pipe transport thing running in time for IETF101.

[Hans Kuhn]

bump

[llynch@…]

need aliases updates to test

[sra@…]

Testing email2trac on tickets.meeting.  Assuming consistency:

* Lucy will get a direct copy, because that's how the alias reads;
* email2trac will drop this update, reporting exit status 1; and
* /tmp/email2trac.log will show another sqlite3 disk I/O error.

[Yet another try running email2trac manually]

[sra@…]

Testing email2trac with new config for tickets.meeting.ietf.org.  In
theory this now goes via a postfix "pipe" channel which replaces the
nasty setuid wrapper program.

[sra@…]

One more test with slightly cleaner postfix voodoo.

[llynch@…]

Thanks to Rob and Hans for the mail pipe hack