The occasional scrivener

why I know perl

I learned Perl in my first real consulting gig at Agero. A large business directory company in Sweden wanted to synchronise their print catalogue with the Web. Additionally, they wanted an interface for customers to create their own ads on the Web. This was the sexy part of the project. I wasn’t involved there.

The synchronisation didn’t work yet, so every Monday my colleague had to take a 650 MB XML-file and feed it to a Java program that inserted the contents into a big old Oracle database running on a Sun Starfire. She was much more billable than I was, so as I incautiously admitted to Un*x knowledge I was asked if I could take over this job.

The XML was full of errors, unescaped ampersands, invalid characters… The Java program choked if it couldn’t parse the file, so you had to manually search for the error and fix it, then try again. A successful run took about 9 hours.

I started by chopping up the file into the component entries and checking for bad stuff. This is trivial, just set $/ to whatever end element suits your fancy, but it took some reading of the Perl Cookbook before I had it nailed.

Then I started looking at how to automate this stuff. I eventually wrote a sophisticated run-control program that could be started with at, and that sent email when something went wrong.

Just when I had cut down the effective load time from three days to about 11 hours, the whole project got axed. I later learned that this was the third attempt to integrate the print version with an online database.

The contractor more or less blamed the whole debacle on us, even though it could be fairly laid at bad project management and unrealistic promises from the client to its customers. Oh well.

In the middle of my next project, I was cding up from a directory over a slow ssh link and accidentally rmd all my perl code. When I called the admins of the machine they helpfully informed me that as the machine wasn’t in production it didn’t have backups.

So now I know more Perl than I really want to. But I’m still learning more every day.

Posted at 14:55, in the comp category. Comments [0]
Submit this story to: » del.icio.us » digg » reddit. Search for it on technorati.