Shake, baby, shake, you know I can fit you in my arms

[venta]

In a recent responsibility-reshuffle at work, I have inherited from zandev the care and feeding of an assortment of Linux servers. The obvious downside is that I know remarkably little about how to do this, but hey, I can learn stuff, I have Google, and Zandev is there to field questions.

The other obvious downside is the visibility of any resulting cock-ups. Do something wrong with one server, and my company's external website will disappear. Do something wrong with another, and an important internal website that almost all employees need to get on with their work on a day-to-day basis will disappear.

It is this latter server, the internal one, which concerns us today. A little while ago, it started making plaintive pleas to be upgraded to Ubuntu 16.04.1[*] That's fine, but upgrading requires restarts, and will make the internal website unavailable for a while. So I should do it outside office hours. And probably outside NYC office hours too.

Today, I had a physio appointment at 10am. So getting up, upgrading the server and then getting dressed and heading off to the physio seemed like a good plan. So I did that.

The server rebooted post-upgrade (always a relief) but the website displayed that most helpful of messages, "Internal Server Error 500". Everyone's favourite.

Never mind, there are some notes from Zandev about how to check for Django[**] errors. So I do that, and find an error, and Google to find out what the problem is, and install a few new packages and reinstall a Python module and fix it. Hurrah! Except our friend the 500 error hasn't actually gone away.

Well, to be honest, it doesn't look like a Django error anyway, I'd start by blaming Apache[***]. I don't really know what I'm looking for, but Apache has an error log and I have Google.

Sadly, Apache's error logs are empty. I figure hey, never mind, I'll restart Apache - because turning something off and back on again always helps. Except in this case, it doesn't. I tried again, but still nothing.

I revert to Google, but Internal Server Error is pretty vague, and a lot of the top hits are either (a)very specific to one web-provider or (b) aimed at people who are wanting to visit a website and want to know why it doesn't work, not the person whose job it is to fix it.

Zandev popped online about half an hour later, and fixed the whole issue in an embarrassingly short amount of time. He pointed out I was looking at the wrong log file (dammit, shouldn't have left my screen session lying around...) and told me where the right one was.

He'd found the error in the logs ("populate() isn't reentrant"), Googled for it, and discovered the solution was to restart Apache. So he restarted it, and suddenly there was the website, all beautifully working.

I pointed out that this was rather unfair. Especially since I had already done that. Twice. Indeed, I checked back through the logs and found the error, plus the evidence of my own restart, all appearing in exactly the same sequence as the ones at the tail of the file that showed him fixing it.

He referred me to the koan of Tom Knight and the Lisp Machine.

Apache does not respect me. Or my dressing gown.

[*] Remember recently when everyone with Windows 8 suddenly found themselves with an option to upgrade to Windows 10? Like that. Only less aggressive marketing. And on Linux.
[**] Django is the web framework which the internal website is built on.
[***] Apache is a ubiquitous piece of software which is the magic glue that makes the files on a server that make up a website be available to someone visiting in a browser.

Comments

[lathany.livejournal.com]

I pointed out that this was rather unfair. Especially since I had already done that. Twice. Indeed, I checked back through the logs and found the error, plus the evidence of my own restart, all appearing in exactly the same sequence as the ones at the tail of the file that showed him fixing it.

I know this happens. But it's still really disturbing.

[drdoug.livejournal.com]

Argh. Sympathy. Happily I'm more often the one who shows up and does the restart/reboot that works after someone else less confident has done the same to no effect. Probably the effect of being among people who are mostly less geeky than I am. But it has happened the other way round. I hate getting superstitious about computers but my experience does make this seem a Thing.

[venta.livejournal.com]

I try to avoid superstition, but when your local guru concludes "I think it just doesn't like you"...

[drdoug.livejournal.com]

You know they can smell your fear?

[drdoug.livejournal.com]

Oh, and Lush '500'. With which I am now earworms, ta for that.

[venta.livejournal.com]

Will a kudo help?

If it's any consolation, I am also earworms!

[drdoug.livejournal.com]

Ta for the kudo. I shall put it in my trophy cupboard with the other N. (I think N may be around 4 or 5, though shamefully I have not been keeping careful count.)

'I am earworms' is one of those rare autocorrect wins, I think.

[exspelunca.livejournal.com]

Our mantra, in the long-ago days of Windows 95, Quark Xpress and Mentor Editorial Tracking System (don't EVER go there, and "virtual METS" launched on Windows 95 because the owners were too mean to upgrade to Windows 2000 was a nightmare in a class of its own) "they are all programmed to fight back". Nothing seems to have changed, does it?.

[waistcoatmark.livejournal.com]

Blimey that's brave. I'd have insisted on setting up a new machine with 16.04, getting stuff working there, pointing over the DNS/whatever to point at the new machine and then retire the old machine.

Or at least some VM/container equivalent.

[exspelunca.livejournal.com]

Sadly, sub editors had no control over budgets or that mysterious glass box in the old line-o-type room wherein dwelt "systems", who emerged when we yelled that it had fallen over ... again..