Impressions from my first big project

I've been coding, depending on how you measure it, for about 10 years now. Making money from it for 6. Doing it every day for 17 months. But my current project is the first one that I can really say is "big".

Sure, other projects have taken months - a few took more than a year from start to end (including all of the bugfix and client changes and so on) - but few have really felt large in the scope of both what remains to be done and it's potential impact.

I can't say precisely what we're doing yet (I could, but my boss would throw a shit-fit) but a very vague summary is "a distributed MIMP processor, producing prepared views on massive discrete datasets". Oh, and it's real-time. And just overflowing with the phrase "embarrassingly parallel".

So now that we are about 3 months in, and about 6 weeks from releasing something approximately "beta" shaped, I feel it's a good time to summarise my feelings thus far:

1. Plan first, code later: every decision you get wrong takes twice as long to fix, because you have to undo all the assumptions you made based on the prior code.

2. But don't be afraid to refactor: no matter how much you plan, you will let a few bad ideas slip by. Just because something is difficult, don't let it stop you from rewriting large chunks of the program. The sooner you change something large, the cheaper it is. In a previous post I wrote about moving from Beanstalkd to Redis for our messaging queue: whilst beanstalkd worked, it didn't provide some important features, and switching over has led to other optimisations for data processing.

3. Think at least a month in advance: whilst we had the core product figured out before we started, the need for certain new features has become apparent; as long as these don't get put into the timeline for the current milestone, then it's not a problem generally. However, just because something isn't important now doesn't mean you should not think about it at all. Consider the technical and business implications of the planned features when you need some downtime; the sooner you table a technical solution, the longer you have to think why you got it wrong.

4. Sometimes you should just go home: there is often a temptation to spend all night coding. Whilst "the groove" is definitely a powerful tool, sometimes it is easy to pretend like you're in it when infact you spent the last 20 minutes alternating between the same file and Reddit. Going home isn't admitting defeat.

5. A bad environment is your worst enemy: I work in an office with a bunch of other developers, a few salespeople, and a couple of other staff. It's pretty natural that with 20 people in a ~1500sqft space that it gets very loud from all the chatter and ringing phones... but however "normal" it is, it absolutely destroys any ability to get into the zone. Staying after work is often the best way to get shit done - no distractions other than some speakers playing easy music, and 8 hours fly by doing as much work as 20 when everyone else is around.

6. Everyone cares less than you: I often forget, having spent the last few months thinking about almost nothing but this project, that other people are involved in this project as much as me. Other people may be more relaxed, or less vocal about what they are thinking about, or just working on something you aren't familiar with (and thus can't appreciate the complexity), but it's definitely a trap I keep having to extricate myself from: their are other people committed to this project, and they are putting in as much effort as you.

I'm sure in the next few months I'll have more random thoughts to share. feel free to comment below with any sage words of wisdom for what I need to watch out for!