AlliedModders recently passed its ten year anniversary. That’s literally crazy. At some point I’d like to do a longer post on its history and the people who made it happen – but today I wanted to talk about the evolution of our build process. It’s quite complicated for your average gritty open-source project – complicated enough that we wrote multiple custom build systems. Right now we’re pretty close to state-of-the-art: AMBuild 2 is a general-purpose, optimal-by-design DAG solver. But we started off in the dregs, and it took 10 years of iteration to get where we are now.
Without further ado…
The Beginning
The first project that would become part of AlliedModders was AMX Mod X. Felix Geyer (SniperBeamer) recruited me to the team, and I provided hosting. It was January of 2004, and there were two compilers you could use: GCC 2.95, and Visual Studio 6.0. The project had a single binary built from about 20 .cpp files.
On Linux it used a single Makefile. On Windows you had to run the Visual Studio IDE. For about six months our release process was entirely hand-driven. Even as we added more binary components, I’d build each one by hand, sort them into packages by hand, drop them into SourceForge FTP, then one by one add each package to a release.
As components became more complicated, I eventually got frustrated by Make’s lack of simple programmatic control. If you’re unfamiliar with GNU Make, it has something resembling the worst scripting language ever made. I couldn’t bring myself to touch it, and in August 2004 created two Perl scripts:
- Makefile.pl – A straight translation of the Makefile to Perl, but with more programmatic control over how and what gets built.
- package.pl (which I no longer have) – after building binaries by hand, this script would build packages and upload them to SourceForge.
Over the next year, AMX Mod X grew rapidly. The development team and community were very active, and the Half-Life 1 modding scene was at its peak. By July 2005, AMX Mod X had over 20 binary components and supported three platforms: Windows, Linux i386, and Linux AMD64. Releasing by hand took hours, and maintaining 20+ random copies of a Perl script was really difficult. Something had to change.
Fixed Pipeline Era
The first step was ditching Makefile.pl. I replaced it with a much cleaner Makefile. Most of the complexity was stripped away and reduced to a few variables and filters.
But I was still pretty new to software engineering, and made a major mistake – I didn’t separate the guts of the Makefile into something reusable. As a result the whole thing got copied and pasted to every component, and to components in other projects. To this day about 50 variations of that Makefile are scattered around, though it’s unclear whether anyone still uses them. Up until 2009 we were still syncing changes across them by hand.
The next step was automating all the component builds and packaging. AMX Mod X had weird packages. There was a “base” package, and then optional addon packages to supplement it. The steps for each package were similar, but they all built different sets of components. The result: I ended up writing something we affectionately called “the C# tool”: AMXXRelease. It had three pieces:
- A general pipeline for constructing an AMX Mod X package.
- An abstraction layer that decided whether to build via GNU Make or MSBuild.
- Classes for each package, usually just supplying filenames but sometimes adding custom steps to the pipeline.
The C# tool survived many years. Though it was a hard-coded tool written in a verbose, rigid framework, it got the job done. AMX Mod X continued to use it until 2014, when we finally ditched it as part of a transition to GitHub.
Interlude: SourceMod
In 2006, Borja Ferrer (faluco) and I secretly begun work on the next major AlliedModders project: SourceMod. SourceMod was a rewrite of AMX Mod X for the Half-Life 2 Source engine. By 2007 it was nearing ready to go public as beta, so I ported the C# tool. Everything was peachy. Until Half-Life Episode 1 came out.
At the time, it wasn’t conceivable to us that Valve would maintain multiple forks of the Source engine. Half-Life 1 was all we knew. Valve had never changed Half-Life 1 in a way that couldn’t be fudged with some detection and binary hacks. But Source was different – it was vast and complex compared to its predecessor. Valve began changing it frequently – first with Episode 1 – and then again with the Orange Box and Team Fortress. Now there were three versions of the Source engine, all totally binary incompatible. The same code didn’t even build across their SDKs, and linking was too complex for something like a “universal Source binary”.
Suddenly our Makefile and Visual Studio project files were getting ridiculously hairy. Not only did SourceMod have a dozen binary components, but it had to build each one multiple times with different parameters, across two different platforms. By mid-2009 we realized this was crazy and started looking at other options.
AMBuild 1: Alpha-Gen Building
I evaluated two build systems for SourceMod: CMake and SCons. CMake was a contender for “worst scripting language ever”. After a lot of futzing I concluded it was impossible to replicate SourceMod’s build, and even if I could, I didn’t want to see what it would look like. SCons came much closer, but it didn’t have enough flexibility. Source binaries required a precise linking order, and at the time SCons had too much automagic behavior in the way.
We needed something where we could template a single component and repeat its build over and over with small changes. We needed reasonably accurate minimal builds. We needed platform-independent build scripts, but also tight control over how the compiler was invoked. So in August 2009, I dove into Python and wrote our own general-purpose build tool, dubbed “AMBuild 1”.
I would describe AMBuild 1 as “rough”. It was definitely extremely flexible, but it was buggy and slow. Each unit of work was encapsulated as a “job”, but jobs did not form dependencies. It had some hardcoded, special logic for C++ to support minimal rebuilds, but since it was special logic, it broke a lot, and because it was recursive in nature (like Make), it was very slow. Since jobs didn’t form dependencies, they couldn’t be parallelized either. Everything ran sequentially. The API was also pretty grody. The root build file for SourceMod was somewhat of a nightmare.
But… it worked! Suddenly, supporting a new Source engine was just a matter of adding a new line in its central build file. It worked so well we began migrating all of our projects to AMBuild.
Over time, as always, things began to break down. By 2013 there were twenty versions of Source, and AMBuild 1 was nigh unusable. Minimal rebuilding was inaccurate; sometimes it rebuilt the entire project for no reason. Sometimes it produced a corrupt build. Builds were also EXTREMELY slow. A complete Windows build could took two hours, and multiple cores did nothing. There were so many headers in the Source SDK, and so many file scans for dependency checks, that merely computing the job set for an update could take 15 seconds.
AMBuild 2: Modern Era
Sometime in 2013 I found Tup and was immediately inspired. Tup “gets it”. Tup knows what’s going on. Tup knows what it is. Tup is what build systems should aspire to be. In short, it has two philosophies:
- Builds should be 100% accurate. Not 99% accurate, or 37% accurate, or “accurate if you clean or clobber”. They should be 100% accurate all the time.
- Builds should be fast, because waiting for builds is a waste of time.
The design of Tup is simple on the surface. Rather than recursively walking the dependency graph and comparing timestamps, it finds a root set of changed files, and then propagates a dirty bit throughout the graph. This new “partial graph” is what has to be built. In practice, this is much faster. In addition, Tup caches the graph in between builds, so if your build files change, it can compute which parts of the graph changed.
Well, that’s awesome. But… I didn’t want to take on Tup as a dependency for AlliedModders. I have a few reasons for this, and I’m still not sure whether they’re valid:
- Tup requires file system hooks (like FUSE) and I was worried about its portability.
- Tup is a C project, and not yet in Linux distros, so we’d have to tell our users to install or build it.
- AMBuild’s API, and of course Python, together make for a much better front-end than Lua.
- Tup didn’t seem to include external resources (for example, system headers/libraries or external SDKs) in its graph.
So I sat down and began rewriting AMBuild with Tup’s principles in mind. The result was AMBuild 2. With a real dependency graph in place, the core of the build algorithm became much simpler, and parellelizing tasks was easy. The API got a massive overhaul and cleanup. It took about two months to work all the kinks out, but in the end, my SourceMod builds took about a minute locally instead of 15. And with some smarter #include’ing, our automated builds went from 45 minutes to 2 minutes. The build scripts don’t look too bad, either.
What’s next?
AMBuild 2 isn’t perfect. Its biggest problems are scalability and internal complexity. I’m a huge fan of Python, and it’s nearly everywhere now. But it’s slow. Dog slow. AMBuild 2 would not scale to a codebase the size of Mozilla or Chromium. In addition, Python’s concurrency support is very bad, even taking into account the multiprocessing module. Most of AMBuild’s complexity is in its IPC implementation.
Early on in its development a common suggestion I got was to separate AMBuild into a front-end (the build API) and back-end (the build algorithm). I heard this theory float around for Mozilla’s build system too. CMake is structured this way. Separating the components out means they are replaceable: AMBuild 2’s frontend could generate Makefiles instead of an AMB2 DAG, or it could read Makefiles and parse them into a DAG.
I eventually concluded that didn’t make much sense. The goal of AMBuild 2 is not to provide a general backend or general frontend: it’s to produce correct builds as quickly as possible. In fact, it considers most other build systems to be broken and inferior. Taking an AMBuild project and generating Makefiles defeats its entire purpose.
However, there is something very useful that comes out of this separation, which is the ability to generate IDE project files for casual use or local development. Keeping this in mind, AMBuild 2 did separate its front-end from its back-end, but the API makes too many assumptions that don’t have clean mappings to IDE files. As we experiment with Visual Studio and XCode generators, we’ll probably have to iterate on the API.
Lastly, it’s worth nothing that part of what AMBuild solves is papering over an early design flaw in SourceMod: with proper abstractions, it wouldn’t need 20 different builds. In fact, over time we have moved a large amount of code into single-build components. Unfortunately, due to the complexity of what’s left and the Source SDK in general, it would be nearly impossible to completely eliminate the multiple-build steps.
Ok
There are very few projects I’m truly proud of, where I look back and have very few regrets. (One is IonMonkey, but that’s a story for another day.) AMBuild is one of them. It’s not particularly stellar or original, but we needed to solve a difficult problem, and it does the job very well. I wouldn’t be surprised if it takes another 10 years of iteration to make it perfect – but that seems to be how things work.
I see a lot of interesting articles on your page. You have to spend a lot of time writing, i know how
to save you a lot of work, there is a tool that creates unique,
google friendly posts in couple of seconds, just type in google – laranita’s free content source
An outstanding share! I have just forwarded this onto a coworker
who had been conducting a little research on this.
And he in fact bought me dinner because I found it for him…
lol. So allow me to reword this…. Thank YOU for the meal!!
But yeah, thanks for spending time to discuss this subject here on your internet site.
I read a lot of interesting posts here. Probably you spend
a lot of time writing, i know how to save you a lot of work, there is
an online tool that creates high quality, SEO friendly articles in seconds, just
type in google – laranitas free content source
With havin so much content and articles do you ever run into any issues of plagorism
or copyright infringement? My site has a lot of unique content I’ve
either authored myself or outsourced but it seems a lot of it is popping it up all over the internet without my
agreement. Do you know any ways to help stop content from being stolen? I’d genuinely appreciate it.
my blog: air conditioning service lauderhill
It’s not my first time to pay a visit this web site, i am visiting this web page dailly aand taqke good facts from here daily.
Youre so grand, man! I cant have faith in I missed this blog for accordingly long. Its just abundant ram every cycle. Your propose, man…too amazing! I cant wait to read what youve got subsequently. I fondness all that youre saying and want additional, added, MORE! Keep this positive, man! Its immediately exceedingly good.
If you desire to take much from this piece of writing then you have to
apply such strategies to your won website.
Thank you, I’ve recently been searching for information approximately this subject for
ages and yours is the best I’ve came upon so far. However, what concerning the conclusion? Are you
certain about the supply?
Thanks, this site is really practical
Many thanks! It is an wonderful site!.
Reparación de electrodomésticos, calderas de gas, calentadores y termos eléctricos de todas las marcas. Es entre las mayores multinacionales que opera en los mercados de generación, comercialización y distribución de energía eléctrica y cuenta con seiscientos setenta clientes del servicio en España. Costo rebajado veintidos por ciento hasta el día quince en todo género de reparaciones y también instalaciones.
Varias empresas y servicios técnicos denunciaron estas prácticas que perjudicaban gravemente a otras empresas, ya que las reparaciones que se efectúan a través de las empresas captadoras no ofrecen al usuario las menores garantías de fiabilidad,” contratan y rescinden contratos habitualmente con técnicos por toda España. En caso que no aparezca consulte nuestro servicio de atención al cliente para obtener más información.
domestic violence
To think that it would be better to grab every opportunities for us to not regret everything in the long run.
Such a great article and truly entertaining and informative! Thanks!
Heya I am a newbie to your website. I found this page and it is really informative. I wish to give something back.
The problems that u have discussed in this post is very much common nowadays . Thank You so much for giving me the tips
The problems which u have discussed in this article is very much common nowadays . Thank You so much for giving me the tips
I do agree with all of the ideas you’ve presented in your post. They’re really convincing and will certainly work. Still, the posts are too short for novices. Could you please extend them a little from next time? Thanks for the post.
Is there anything else I could read to learn more about this? This is an excellent, an eye-opener for sure!
You really make it appear really easy with your presentation however I to find this topic to be actually something that I feel I might never understand. It sort of feels too complex and extremely wide for me. I am having a look ahead to your subsequent post, I will try to get the hold of it!
You’re the best, It’s posts like this that keep me coming back and checking this blog regularly, thanks for the info!
I check your website daily and endeavor to learn one thing from the site. Thank you and waiting for the new submit.
I merely like the methods individuals offer as component of your articles. I will certainly bookmark your internet site in addition to examination out once more right here regularly. I’m relatively specific I will be advised a number of new product remedy here! Best of good luck for an extra!
I believe this is among the most important details for me. And i am glad reading your short article.
Admiring the persistence you put into your website and in depth information you offer. It’s good to come across a blog every once in a while that isn’t the same outdated rehashed material. Great read! I’ve bookmarked your site and I’m adding your RSS feeds to my Google account.
Having actually read this I thought it was definitely helpful. I enjoy you discovering the time and attempt to place this insightful article with each other. I once more find myself investing a substantial volume of your time both analysis as well as leaving behind reviews.
Many thanks for the post and terrific tips. even I likewise assume that effort is actually the most essential component from obtaining excellence.
Admiring the time and effort you put into your blog and
Excellent post however I was wondering if you could write a litte more on this subject? I’d be very thankful if you could elaborate a little bit more. Thank you!
Здравствуйте! Advances in technology and the explosion of connected devices and sensors are driving a new era of buildings and their management. With that, facility managers are now under pressure to deal with an unprecedented number of systems and the ensuing complexity they bring.
Appreciating the dedication you put into your site and detailed information you provide. It’s nice to come across a blog every once in a while that isn’t the same out of date rehashed information. Great read! I’ve bookmarked your site and I’m adding your RSS feeds to my Google account.
Thanks so much for the blog article. Fantastic.
Really enjoyed this post.Thanks Again. Much obliged.
Betingslot is a trusted slot site, comes with a 100% new member bonus promo
Betingslot online is easy to win the biggest jackpot in Indonesia, it has indeed become a famous game since its inception. Online slot gambling sites are easy to win
Brokers in Indonesia. Betingslot is one of the high-quality alternatives for you because Betingslot already has an reputable license
and it is registered on the middle international Online Betting organisation.
Inside addition, Betingslot also presents merchandise which can be entire with era which was analyzed by World Gambling
professionals.
Services Supplied by Betingslot
Betingslot is certainly one of the largest Online Gambling Official Agents that offer the first-rate provider, Betingslot provides your centers
to be served right away by expert and pleasant customer support.
In addition , Betingslot Consumer Service is likewise in rate of keeping you secure at gambling, and now not most effective the service supplied
is high-quality however the events given to individuals are also wonderful, of direction.
Brokers in Indonesia. Betingslot is one of the high-quality alternatives for you because Betingslot already has an reputable license
and it is registered on the middle international Online Betting organisation.
Inside addition, Betingslot also presents merchandise which can be entire with era which was analyzed by World Gambling
professionals.
Services Supplied by Betingslot
Betingslot is certainly one of the largest Online Gambling Official Agents that offer the first-rate provider, Betingslot provides your centers
to be served right away by expert and pleasant customer support.
In addition , Betingslot Consumer Service is likewise in rate of keeping you secure at gambling, and now not most effective the service supplied
is high-quality however the given to individuals are also wonderful, of direction.
Brokers in Indonesia. Betingslot is one of the high-quality alternatives for you because Betingslot already has an reputable license
and it is registered on the middle international Online Betting organisation.
Inside addition, Betingslot also presents merchandise which can be entire with era which was analyzed by World Gambling
professionals.
Services Supplied by Betingslot
Betingslot is certainly one of the largest Online Gambling Official Agents that offer the first-rate provider, Betingslot provides your centers
to be served right away by expert and pleasant customer support.
In addition , Betingslot Consumer Service is likewise in rate of keeping you secure at gambling, and now not most effective the service supplied
is high-quality however the events given individuals are also wonderful, of direction.
Brokers in Indonesia. Betingslot is one of the high-quality alternatives for you because Betingslot already has an reputable license
and it is registered on the middle international Online Betting organisation.
Inside addition, Betingslot also presents merchandise which can be entire with era which was analyzed by World Gambling
professionals.
Betingslot Consumer Service is likewise in rate of keeping you secure at gambling, and now not most effective the service supplied
is high-quality however the given to individuals are also wonderful, of direction.
Services Supplied by Betingslot
Betingslot is certainly one of the largest Online Gambling Official Agents that offer the first-rate provider, Betingslot provides your centers
to be served right away by expert and pleasant customer support.
Betingslot is one of the high-quality alternatives for you because Betingslot already has an reputable license
and it is registered on the middle international Online Betting organisation.
Inside addition, Betingslot also presents merchandise which can be entire with era which was analyzed by World Gambling
professionals.
Betingslot online is easy to win the biggest jackpot in Indonesia, it has indeed become a famous game since its inception. Online
Betingslot online is easy to win the biggest jackpot in Indonesia, it has indeed become a famous game since its inception. Online slota gambling sites are easy to win
n By Spiritual Renewal and Why I Seek It | Black Coffee Reflecti
Betingslot is Number one the Greatest Official Internet gambling
Brokers in Indonesia. Betingslot is one of the best choices for you because Betingslot already has an official license
and it is registered at the center worldwide Online Betting company
Betingslot is Number one of the Greatest Official Internet gambling
in Indonesia. Betingslot is one of the best choices for you because Betingslot already has an official license
and it is registered at the internsional Online Betting company
Betingslot is Number one of the Greatest Official Internet gambling
in Indonesia. Betingslot is one of the best choices for you because Betingslot already has an official license
and it is registered at the internsional Online Betting company
Reply ↓
Mas68 is one of the best and most trusted online gambling sites in Indonesia, which is here to provide a variety of security and convenience for online gambling lovers.
Also get a variety of attractive bonuses that are offered on the bettingslot site that you don’t get on other sites. Enjoy the various games available on the site