SVN Externals are Evil; Use Piston or Braid

I've recently spent a considerable amount of time rectifying problems caused by SVN externals. In one of the codebases I work on, it had been developed with a heavy number of Rails plugins as SVN externals. In general, it was a good approach as these were external code, or shared code, etc. This I think is at least better than directly checking the code in, as you have a more precise record of where it's from, etc. I should also note that our externals were all set to specific tags or branches specific to our code (i.e. not to trunk, where you'd be getting updates without your control). Sounds good, what about this "evil"?

The problem comes in when you need to make changes to the code of an external. You might think, well, go change the root code and then adjust your tag, etc. In some cases you can't do that - maybe it's not code you have commit rights to, or you're making a change that's specific to your app and can't be done another way, or, as was often in the case I had, we were on a much older version, and the trunk and other tags had major differences that I didn't want to integrate.

Thus, what I needed to do was remove this as an external, and check the code in directly. Another approach would be to branch it from where you were and modify that, etc. I wasn't able to do that due to various Subversion permissions (probably not a common case, but I had no choice). This action itself (remove external, add code) is not a real problem in SVN. But, it IS a problem when you go to update. A simple "svn up" on other machines failed. That is pathetic. Instead, what I had to do was go delete the existing (svn externaled) directories, then do "svn up". This of course broke our continuous integration server, and I also had to go manually fix this up on machines I was deploying to. Crappy, but if that was the end of it, I'd probably not be as unhappy...

When it comes to merging these kinds of changes into branches, watch out! This is where SVN just flails. First if you happen to use svnmerge.py to manage your branch merging, forget it. It just can't deal with it, and will leave you with a partially complete merge. Doing it manually, even with things like --ignore-ancestors, does not work either. I had to do something similar as to the "svn up" fix: I had to go in and delete all the directories that were previously svn externals, and then do my merge. And note, do NOT delete the parent directories. For example, if all of your Rails app's plugins were externals, do go and nuke "vendor/plugins". It will then be totally confused and just not do anything, and fail. Nope, you need to specifically delete each offending svn external directory. I make extensive use of branches (I do most work on a branch for daily work), so you can multiply these problems across the number of branches you might need to be merging to, etc.

Having said all that, this problem isn't really all that illogical. I don't know how SVN works internally, but the whole svn:externals thing seems a bit like a hack, or at least not a first class citizen in SVN land. SVN merge or update, should be able to see: hey, you were up to date (for your current revision) on directory X, but this update is going to replace that with new code with the same dir name. But, it doesn't, maybe because it doesn't look at the externals properly in relation. I don't know, and I don't care, since it's broken, and my fix is that I'm moving to Git soon enough :) Also, as another point of view, I know Perforce handles this kind of thing just fine (we used remote mounted Perforce depots all the time at Adobe, and made seriously extensive use of branches (in fact, we required working on a branch)).

Now that I've spent entirely too much time on the build-up, what's the solution? Simple: use Piston (or Braid if using Git). What Piston does, is to not use svn:externals, and instead check the code in directly, yet maintain linkage to the external it came from. My take is this is really probably how svn:externals should've worked (I presume that constantly updating an external is actually a rarely desired trait). You import an svn external using Piston, and it will pull the latest code from whatever SVN URL you supply. In this case, you could use trunk, or you could as usual use a tag or branch. But then it's fixed - it will not update that anytime you do "svn update". Instead, it is up to you to explicitly tell it to update. This avoids svn externals as far as your daily operations go, and also causes zero problems for merges. It does more though.

The second benefit of Piston is that you can then modify the external code, but still bring down updates from the external, allowing a synergy between using external code and your app's specific needs. This is exactly what I needed on a couple of plugins we use, where those plugins' code had deviated significantly from our codebase so I couldn't use a newer version, but I needed to make some changes.

To summarize, the evil is SVN itself not handling changing of externals (i.e. to/from an external) in basic operations like updates and merges, which may cause a lot of manual work on your end, and break automated builds or similar. The solution: use Piston or Braid and get the best of everything.