Let’s set the scene. It’s July 1996, and I’m holed up in a small Michigan Avenue office psyching myself up for my first ever internet startup all-nighter. I’m a college student working for a small company that does online restaurant reservations in Chicago; un-creatively yet descriptively called “Reservations On-Line”. It’s 8 or 9pm, I’ve filled up on Chicago hot dogs, and I’ve got a deadline to launch a prototype version of the site the next day. Our small five person office is a little lonely. It’s empty except for me (the other two members of the tech staff apparently had the good sense to finish their work on time), but I’ve got a T1 line, some Jolt cola, and a stack of CDs with beats-per-minute so high they’ll keep me coding long after the Jolt runs out.
Fast forward to the next morning when the first of my co-workers comes in around 12 hours later. I’m asleep on the floor of the server closet and I’ve got a bunch of janky, half-working Perl scripts and sloppy hand-coded HTML to show for my efforts. Success? Well, it kind of felt like it at the time.
Spoiler alert, an internet startup based upon online restaurant reservations before most people and restaurants were online did not stand a high chance of success. A good idea, just too early. In 1996 only about 22% of Americans were online, and the web itself was only really starting to gain traction as the primary way people interacted with the internet. In 1995 only about 3% of Americans used the web, and while that spiked up to 16% in 1996, there was still a long way to go. Forget iPhones and WiFi, this was the era of 28.8k baud dial-up. That’s .0288 Mbps, or nearly 7,000 times slower than the current internet speed average in the US of 195 Mbps. (Dear fellow BBS SysOp dinosaurs, I know it’s not really .0288 Mpbs, but I’m going to do my best to not have this be a post entirely about antediluvian technology.)
So Reservations On-Line was not an early dot-com success story, we did not succeed and beat OpenTable to the punch by two years (hey, that’s a long time in internet years). It’s worth noting that OpenTable was not anything like an instant success itself; in fact the Chicago Tribune says it “barely survived the tech bubble”. And the New York Times talks about reservations really starting to move online in 2007, with OpenTable at that point having 7,000 restaurants available worldwide. In 2021 there’s almost double that 2007 number on OpenTable in Chicago alone (12,933), so it has taken a long time to get to the ubiquity we now take for granted. 2007 was also the launch of the iPhone, un-coincidentally. In retrospect — we were at least 2 years too early and really more like 10, but it’s hard to see that at the time. It’s virtually impossible to see that when you’re an inexperienced 21 year-old and the web has just started.
As Marc Andreessen wrote, “it is the market that matters” and the market wasn’t ready for this product. Also, for my own part I was very green and made a lot of mistakes — like thinking that an all-nighter before launch gives any better results than an all-nighter before a test. I learned a lot too, even if much of those lessons were learnt the hard way.
My ancient archives also indicate that I used the Afterstep window manager for Linux at the time, so rather than a clean Macintosh interface it would have looked a little more like this:
Radical! (screenshot from xwinman.org/afterstep.php)
Nostalgia aside, a detailed dissection of what did and didn’t work in a project from 25 years ago is even more useless than an explanation of modem baud rates. So let’s call the post-mortem part of this article dead.
Would I do it again? Absolutely. The pay may have only been $12.50 an hour (NB to potential clients, my rates have gone up since), but the experience was worth a whole lot more than that.
What I want to explore is what we learned about tech and startups way back then that still resonates, or that we are still trying to learn fully.
I say “we” because in this post we’ll also be talking to my two erstwhile co-workers in the Reservations On-Line tech department. Maybe it’s less navel-gazey if there are more navels? Let’s meet the team:
|Reservations On-Line Position, 1996
|Job Title Now
|“Systems Programmer” — Perl CGI scripts, HTML, Design.
== “Front End Developer” in 2021 terms.
|“Analytics Engineer and Consultant”, Quantable.com
|Hsin Tsao (pronounced “Shin”)
|“Systems Programmer” — Perl CGIs, Database management and integration, overall site architecture.
== “Full Stack Developer” in 2021 terms.
|“Staff Engineering Tech Lead/Manager” – Engineering Lead at Google
|“Unix Sysadmin” — setup and config servers, internet connection, production configuration and deployments.
== “DevOps” in 2021 terms.
|“Senior Staff Site Reliability Engineer” – Google
All three of us were college students in the same program at the same time, and I’m happy to say we’ve kept in touch throughout the years.
We’ve all had pretty successful IT careers. Narayan and Hsin both work for Google, and I write excessively long blog posts.
What I find most interesting is that we also all have stayed heavily involved with the technical side of things. I think we all have been managers at some point, but we’ve tended to drift back more toward the engineering side. Twenty years ago when we were coming up there was a well-trod path from front-line coder to pure manager, but that’s changed. With many more engineering manager, tech lead, and architect positions available, there’s a myriad of paths to choose. Still I might argue the main choice is what mix you want of: people managing, architecture, and hands-on engineering… these things just come in a lot more variations than before.
What was the most important thing you learned that summer in 1996 that has really stuck with you?
Hsin: That the details and final 10% are the hardest. We built a prototype in just a few weeks (if I remember correctly) — and it seemed like it wasn’t too hard, but to actually productionalize the idea raised so many new unforeseen problems outside of the primary task of making online reservations.
Jason: Similarly to Hsin, I learned that there’s no substitute for actually doing the development and then getting ready to iterate. Estimates and project planning are useful, but not when they keep you from just getting something launched. But be prepared that what you get out the door will suck and need to be improved. The old adage of “Release early. Release often. And listen to your customers.”. I feel like this is accepted wisdom in this age of agile programming, but many projects still don’t follow this wisdom.
Narayan: The largest lesson for me was that the tech stuff isn’t the hardest part. Validating the value proposition, finding a market — someone actually willing to spend money — those are much tougher than building a working system.
What did you think you had figured out back then that you’ve since decided you were completely wrong about?
Hsin: Thinking that technology always makes things simpler when sometimes it just adds complexity (especially for those not in the tech field).
When introducing new technology, it isn’t always just automating the same process that people normally do. It sometimes requires a paradigm shift, but people might not want to adopt the shift either.
Jason: Back then I thought “the right” technology really mattered. That a project would be doomed if it used the wrong development platform (which at the time was Microsoft). I now believe in most projects there are many technological paths to success.
Narayan: Similar to Jason’s point, I thought it was important to build “enterprise grade” systems. We spent a lot of money and time building infrastructure (a Sun workstation, running Ingres!) that could have been put to better use.
What did you think you learned back in 1996 that you’re really still trying to internalize?
Hsin: How to get the public to embrace technological change.
Jason: I thought I had learned to not trust what the vendor says until I’ve actually used and integrated the product, but I’m still working on that one. It’s hard to not buy into a glossy marketing message and good sales pitch. I remember that there was a lot of discussion with our database vendor Ingres about their “web enabled” database product. I didn’t really understand what “web enabled” meant, but figured they must be right that we needed it, since we were making a web application. Turns out it didn’t really mean anything. Now I try to ask very specific questions about a product I’m considering purchasing until I get an answer that explains it in a way I understand, even if it sounds like I’m asking a stupid question.
Narayan: For me, it is that last point – scoping the MVP. The M and V are often really subjective, and end up being a tricky question. We’re even struggling with this now on a project — trying to answer questions like “how scalable should the first implementation be?” and “what functionality should be in the first version?”
We spent a lot of time having to “invent the wheel”. We had long discussions about different ways to architect what were by today’s standards very simple and established design patterns. Does doing engineering today at this more abstracted level following frameworks, IDE suggestions, and general established best practices hurt younger developers? Is there truly ongoing value in hard-earned work of figuring out these things yourself (even when you turned out to be wrong)? Or are we just cranky old men begrudging that things are easier now because we didn’t have Google and Stack Overflow?
The definition of software development has changed and has expanded beyond engineering and computer science. For engineers and computer science students, it does hurt to some degree when they ignore understanding the fundamentals and there is value in learning those to advance their careers.
Jason: Wow, I forgot about DejaNews. Usenet was so important, it was the first part of the internet I used back in the early 90s. It’s a good point that collaboration to solve tech problems was around long before the current set of tools, it’s just easier now and the corpus of existing solutions is so much bigger.
Narayan: It does feel like we were doing all of this back in the stone ages, piecing together information from Usenet and people who knew more than us — I did so much cargo culting, early on. There were so many fewer people working in the field at that point, and the creative flywheel just wasn’t running as fast as it is today.
It is worth pointing out that “higher abstraction levels considered harmful” type arguments have been ongoing since well before 1996 – at that point, some people still considered C to be too high-level! A few years later, there was a big debate about whether CS students could learn enough when computing in new high-level languages like Java. While there are clearly kids on my lawn, I think it might be OK if they showed up knowing how to use a lawnmower — it is better than needing to deal with scythe injuries all of the time.
Jason: “Cargo culting” is a great term, it’s been a long time since I’ve heard it. Everyone copy & pastes, the problem is when you do that without even trying to understand what you’ve pasted in or including it because you think you need it. Stack Overflow has a great post recently on this showing that 25% of visitors copy code within 5 minutes of looking at an answer, and it doesn’t matter if the answer was accepted or not. Not to mention dependencies, etc. Learning and building as a community is great and what we want to do, we just don’t want to do it blindly.
How do you continue to find motivation to learn new skills instead of relying on what you already know? How much of what you learned 25 years ago is still relevant?
Hsin: I still find new technical skills to be exciting and stimulating, but I’m more selective in picking what new skills to learn. Maybe I’ve developed a little more cynicism, but I will challenge some of the new things. Just because it’s new doesn’t mean it’s good. I’ve developed a healthy respect that not all old skills are the best, but some old skills are good because they work.
For example, computer books covering fundamentals and principles stand the test of time much better than a book on a tactical skill like “Learn C++ 10”.
Jason: I imagine that 25+ years of experience has turned us all into cynics of some sort? I’m still excited by learning, but I’ve found myself becoming resistant to learning completely new paradigms. I’m always looking for the easiest solution to a problem that passes my internal “correctness” test, and that’s typically going to be using something I already know. I find learning things that are close to something I already know (basically iterative learning) to be the easiest. Fundamentals like networking and unix have provided more ongoing benefit than anything else for me. I’ll admit I haven’t cracked any of the old CS books open in a long long time, though I’ve still got a copy of The Structure and Interpretation of Computer Programs (the Wizard book) on my shelf, yikes.
Narayan: I wouldn’t consider myself to be particularly cynical, but I’ll readily admit that most new stuff won’t be here in a few years. I’m selective about what I learn mainly due to a lack of time; I’m in my 40’s, have kids and so forth. I don’t run an ATM network for fun at home anymore.
And while I find technical things interesting, my motivation comes from having problems to solve. A GI doctor former collaborator of mine used to tell his research fellows “You need to be fearless to solve the medical problems we’re tackling – if that means learning new biology, statistical methods, or computing techniques, that’s the job”. So most of my new learning comes from having a problem I’m invested in solving, and identifying tools that would be useful. I’ve been working to put our understanding of distributed systems reliability onto a solid conceptual footing for the last 18 months, and that has required me to learn a bunch of statistics and data science approaches. And that has taken me back to the basics like Hsin described.
The technologies we worked with in 1996 have changed, but still mostly exist (Perl, Ingres, Linux, Solaris). As developers older than us support things like COBOL, Fortran, etc., is it inevitable that we (or our contemporaries) work disproportionately on legacy systems? Is that ok?
Hsin: My personal experience hasn’t indicated to me that our generation will get locked in legacy systems and technology. I feel like our generation is the first that was built on transformative technology that changes very rapidly. We didn’t specialize in just COBOL or Fortran. We had to adapt throughout our career (maybe Perl in January and then it’s Python by June) that change became the norm.
Jason: I do think we will work disproportionately on legacy systems, but I agree with Hsin that we won’t be “locked in”. Previous generations weren’t locked in either, but there’s just a lot more choices across the board now. ROL was written in Perl, and I still use Perl on a regular basis, but that’s a personal choice… and I use a lot of more modern technology as well. This idea of constant change and the huge amount of change in development frameworks that there was in the 90s and 00s has contributed to this idea that being a polyglot programmer is a worthy goal. It’s debatable to what extent that is worthwhile, but there’s no doubt that if you have half-a-dozen languages under your belt that the next one will be a lot easier.
Hsin: Previous generations might have been able to specialize in a particular technology for their entire career (e.g. COBOL), but it feels much harder now to have that kind of stability.
Narayan: I think it is exceedingly hard to completely kill technology. Most of the technologies mentioned above (aside from Linux) are substantially diminished from their peak, but they are still around. I think that change has to be the norm in computing, along with flexibility. However, there is an interesting conflict with human nature here — most people don’t want to be living in a state of flux.
I think that maintaining legacy systems is increasingly a specialization. Folks that have worked on government systems (Medicare billing, for example) talk about how the precise behavior of the implementation, as opposed to the specification (i.e., the law) is the key invariant to maintain, because our medical system has settled around the behavior of these software systems. That’s a fundamentally different core problem than standard software development has.
Jason: Real-world compliance against the reality of a legacy computer system instead of the actual rules is quite a thought, wow. I like that framing though, “legacy systems” as a specialization more generally — irrespective of it being Perl, COBOL, or AS/400s.
What percentage of your working time now is doing coding, ops, or things you might consider “front line” engineering, vs. architecting, vs. managing. How do you think this will change in another 10, 20 years?
Hsin: For me, it depends on the team/project I’m working on and whether to combine professional and personal work.
Coding is therapeutic for me so I will try to do it at home if my work doesn’t allow me to do much.
A change I see in 10, 20 years is something that has already started in that there is a lot of “basic” development that can be done by the masses with less technical training.
Jason: The percentages are hard to measure because they are so colored by perception. If I don’t like managing (and I don’t), then the time spent doing that seems to be twice as much as coding, which seems to go by quickly. I would add that I’ve never found coding to be particularly “therapeutic”, just very engrossing. I’d estimate: 20% engineering, 40% architecture, 40% managing & admin. I think those percentages are not likely to change that much, but I hope to do more architecture and more teaching/mentoring (which I suppose I bucket as managing).
Narayan: I’m the senior engineer for a set of teams responsible for the reliability of a bunch of GCP products. My day to day varies, but includes a decent chunk of time setting high level direction for the org, mentoring folks, and engaging on particular technical problems, usually in a less hands-on way. I also have a reliability research project that I spend about 25% of my time on. That is concrete, hands-on technical work.
I view this split much more as a function of my role than the state of the industry. I have a lot of trouble figuring out where I’m likely to go in 10-20 years — that’s a really long time. I’m probably not still going to be working in another 25 years, for example, and we’ve all come a long way since Reservations On-Line, so it is hard to predict. I can say that my interests have changed a lot over the years.
Do you think the industry will become less ageist as its technical leaders become older themselves?
Hsin: I believe this will be shaped more by society than by technology leaders.
Jason: I doubt it. There’s too much emphasis put on too small a set of leaders for those guys (and yes, way too many of them are men) to not have pretty distorted world views. I don’t consider myself a leader but we all need to try and be aware of the world beyond just the tech problems themselves. I’d like to see a democratization of leadership, not hero worship of Elon Musk or whoever.
Narayan: Changing culture is the hardest thing. We need to work at it, but it will be a long slog.
We’ve all worked for companies big, small, startup, established, etc. If your kid came home from college with three summer job offers:
1) A well-paid position at a FAA(N|M)G company with a clear end date and limited responsibilities.
2) A modestly paid position at a Consulting company with a chance to work on a number of different projects and maybe a shot at a post-graduation position.
3) A more poorly paid position at a Startup with potentially a lot of responsibility but very low stability.
Which one would you encourage them to take and why?
Hsin: For a summer job, I would pick FAANG and then #2 (consulting). The benefit of a startup might be lost on a college student — even though it could be fun and exciting. For FAANG, I’d also want to be very selective to find a role that really has good learning opportunities.
Career-wise (beyond summer internships), I believe exposing yourself to all those options early on in your career is beneficial. This helps to soak in as many different experiences as possible to find out which environment works best for you.
Jason: I would say FAANG also, probably… with consulting as a close second. Maybe that’s because that’s the opposite of what I did early on (which was startups and consulting), but it seems to me like getting one of those big companies on your resume early is a way to have a stamp of approval on your talent level that is hard to get otherwise. Plus I think it gives you the best connections for the future.
Narayan: I’m not sure, to be honest. I’d probably make recommendations based on their goals and personality. I’m not a big believer in one size fits all career advice.
Jason: Funny that none of us are gung-ho for the next generation to repeat our startup experiences. I agree with Narayan that it depends on the person, but I’d also argue that the startup experience is not anything like what it was 25 years ago as far as opportunities to learn for inexperienced people just starting out in industry.
Thanks for reading, we hope this has been an interesting and helpful discussion. Many thanks to Hsin and Narayan for taking this trip down memory lane with me! Maybe we’d do it again in another 20 years. Let us know in the comments what you think (never reading the comments is not one of the things I’ve learned).
Also shoutout and best wishes to Doug and Frank, the two guys who hired us and gave us this opportunity all those years ago!
Want to get hyped for some hard-core coding, 1996 style? Throttle your connection speed down to 1.54Mbps, open up your copy of Netscape Navigator to Altavisa… and <blink>click here</blink> for my embarrassing but historically accurate Spotify Playlist.
Extra credit! Hsin’s old-school programming fundamentals reading list:
- Programming Pearls by Jon Bentley – A great collection of essays on programming including ways to approach programming problems.
- The Art of Computer Programming, Volumes 1-4A Boxed Set by Donald Knuth – This weighty 4 volume set is a thick academic tomb of knowledge for computer science.
- Software Tools or Software Tools in Pascal by Kernighan and Plauger – Writing software tools and utilities is a special discipline.
- The Pragmatic Programmer: From Journeyman to Master by Thomas and Hunt
- The Mythical Man-Month: Essays on Software Engineering, Anniversary Edition (2nd Edition) by Frederick Brooks Jr. – Describes Brooke’s Law that adding man power to a late project makes it later and other wisdom of software development project management.
- Peopleware: Productive Projects and Teams (3rd Edition) by DeMarco and Lister – Software development is ultimately done by people (not cogs and robots) and it’s important to remember that.
- The Practice of Programming (Addison-Wesley Professional Computing Series) by Kernighan and Pike – Talks about the practical aspect of programming including “topics like testing, debugging, portability, performance, design alternatives, and style.”
- Facts and Fallacies of Software Engineering by Becker and Glass – A lot of common sense stuff about software engineering that are often forgotten in the thick of our work.