Thursday, February 16, 2017

Parta Games Tech Post #1 Automating Builds & Steam uploads with Unity Cloud Build, Jenkins and Steampipe

Intro

As our blog has mostly been focused on other things, the thought of making a tech post about a specific topic has been on mind for some time. A couple of days ago I finally got around to researching and implementing a thing I've been putting off for too long: automating the Windows build of a Unity game with Unity Cloud Build and subsequently sending the build artifacts to Steam with Steampipe using the Jenkins continuous integration server, essentially making the two servers work together to solve a problem.

Why?

I strongly believe making software builds manually starts becoming a redundant task really fast when the need arises to continuously distribute builds to third-parties, other services or for example QA. This includes the separate platform-specific builds of a single game as well as builds for libraries etc. I also wanted to automate the final step of sending binaries to a store vendor's service, in this case Steam. The sooner a build pipeline is automated the more time is saved when instead of making the builds manually the developer can focus on other things and trust that things are scheduled or triggered conveniently. Creating the whole pipeline from scratch might take a day or two at worst but I believe the patience starts to get rewarded quickly.

Services & Tools

Unity Cloud Build was selected because it removes the need for our own dedicated build servers, has a handy web-api with sufficient documentation and was very easy to set up with our existing version control solutions. It's also a no-brainer if you're making a Unity game because the service is provided for everyone. This guide is created with a Unity Plus license, but that shouldn't change anything, as the service is also provided to free version users with a lower usage priority.

Jenkins is a free & great (if not the best) continuous integration server with awesome support for especially "freestyle" builds. Freestyle essentially means I can configure a series of build steps using whatever method/script/plugin is available to me. I use it for a lot of things from automating static website uploads to creating final software distribution builds. Jenkins is very popular in Java software development and the server actually is a Java software itself (needing Java runtime to function), but don't let that scare you. It's plenty powerful, stable, battle-tested and very configurable. There are also tons of plugins for myriad of use-cases.

Guide requirements:
  1. A Unity 5.x project (source controlled, our choice is Git)
  2. No prior knowledge of Unity Cloud Build
  3. No prior knowledge of Jenkins CI server
  4. Steam developer account and working Steampipe configurations. Essentially I assume you know how to upload a game build to Steam and can publish it.

Setting up the game build in Unity Cloud Build


Start by creating a new project.


As you probably know, a Unity developer account is needed to access the Unity Cloud Build service. If you've recently created your project, chances are it already exists in the Unity projects page and you can just navigate to it and enable build service. If not, create a new project. Just make sure it uses the correct organization and references the correct project ID (the same you can see in Unity services tab).

When presented with the option, enable Unity Cloud Build

Enter the source control information and give UCB the proper rights to access it. If you're somewhat familiar with source control security, setting this up should be a non-issue.


Enter source control information


The next step is to add the new platform target. As we're automating a Steam build, win32 platform is needed. Default platform config should work, but you may need to change the executable name to something sensible. In my case the default executable name was something else and I noticed it only after downloading the resulting ZIP-artifact.


Default config with a sensible executable name
And that's it! Unity Cloud Build setup is now done and you can start the first build. For me it took a couple of tries for some reason (I guess the service needed to warm up a bit), but without changing a thing the build started working. If everything goes well you should see rows of something like this appear in the History tab:

A successful win32 build
Although setting up UCB is probably the easiest step in this guide it solves a very important problem: automating the game build from source control and providing us the build artifacts.

Testing the Unity Cloud Build API

Now that you have a working UCB project you can start querying the API for information about the build. This is a convenient way for the Jenkins build to fetch crucial bit of information: the download links to the build artifacts with proper access rights. The relatively unknown UCB API solves this problem quite elegantly.

1) First find out what your API key is. Instructions to locate it can be found in the Unity Cloud Build API documentation. Keep the key handy as you will need it later in this guide, but note that it's not public information as it allows anyone to access your Unity projects through the APIs.

2) You are now ready to query some basic information from the API. I used a terminal & CURL to get the results, but you can use the HTTP client of your choice. The following query will get you information about all successful builds in UCB (note the authorization header with the API key):

curl -H 'Authorization: BASIC YOUR_API_KEY_HERE' https://build-api.cloud.unity3d.com/api/v1/orgs/your-organization-here/projects/your-project-name-here/buildtargets/default-windows-desktop-32-bit/builds?buildStatus=success&platform=standalonewindows

The result should look something like this:
The essential part in the response JSON is the value of path links/download_primary/href. It contains the direct download URL of the main build artifact (which in this case is a ZIP-file) with AWS credentials. This is exactly what you need to have Jenkins download the data and upload it to Steam.

Setting up Jenkins CI-Server

Now that you have fetched the information available through the Unity Cloud Build API you can continue setting up the Jenkins CI Server. Jenkins is available for many server platforms. In this guide I use Ubuntu Server 14.04 64-bit for which Jenkins is directly available through the default apt package manager. For installing Jenkins on other platforms please consult the official site.

sudo apt-get install jenkins

And that's it. Jenkins is installed as a linux service daemon and is starting up. The server is used through a web-interface which should start accepting requests at http://your-jenkins-address:8080.

Setting up the Steampipe build in Jenkins

Start by clicking "New item" from the leftward menu. Type the build name and select "Freestyle project" and press OK. We will work based on the assumption that this build will be automatically triggered by the successful cloud build.

Note: The way I configure and create the Steampipe app and depot configuration files during the next step is not a perfect solution, but for the sake of this guide it was fairly quick to set up. In the future it would probably be better to pull the files from Git or similar location.

Create a new "Execute shell" build step and add the following configuration:


Next you'll fetch the latest successful build data from UCB. You've already tested how the UCB API works so it's quite easy to parse the data out of the response JSON. Groovy scripts in Jenkins builds are quite versatile so it's very convenient to have the script unzip the files too. You'll probably need to install the "Groovy" Jenkins plugin in the "Manage Jenkins -> Manage Plugins" section for the build step to appear as an option. Create a new "Execute System Groovy Command" build step and add the following script:

The final step is to let Steampipe do its magic. Create another "Execute shell" script and add the following command (where $WORKSPACE corresponds to the Jenkins build workspace folder):

/opt/tools/steamcmd_linux/steamcmd.sh +login your_login_name +run_app_build $WORKSPACE/app_build_123456.vdf +exit

Note: I was running Jenkins on Linux, so I had to log in to steamcmd manually as the "jenkins" user for it to cache the login credentials. This way you need to submit the two-factor authentication code only once and the script will be able to login automatically from that point on.

Now the build is complete. It won't trigger automatically just yet, but you can run it to make sure everything works. The script will first create the config files. Then it will download the build from UCB and send it to Steam.

Setting up webhook to trigger Jenkins build

Now that you have created the two builds that work individually, it's time to make them sing together. You'll set up a webhook in Unity Cloud Build that creates a HTTP POST request to Jenkins triggering the Steampipe-build automatically, every time the cloud build is successful. First you need to configure a couple of security settings.

Jenkins security

Install a Jenkins plugin called "Build Authorization Token Root Plugin". Do this in "Manage Jenkins -> Manage plugins" page. This plugin makes it relatively easy to give UCB access to Jenkins' REST API while using a single security token (UCB does not support HTTP authentication, which is by default needed by Jenkins).

Second, you need to configure Jenkins to receive "Anonymous" requests through the built-in REST API. Go to "Manage Jenkins -> Configure Global Security" page. If security is not enabled, you should enable it now. For the purpose of this guide, make sure matrix based security with Jenkins' user account database is used. (Access control should be secure, with user accounts only for people who need access)

The "Anonymous" user needs access to builds. Check the box labeled "Build" (can be found from the "Job" column) and then click "Save".

Build checkbox for user "Anonymous" in the lower left corner
Finally, you need to add a build-specific authentication token. Go to the build page and click "Configure". From the "Build triggers" section, check "Trigger builds remotely". Think up and enter a secure token and save the configuration.

Entering the authentication token
That's it for Jenkins. You can make sure everything works by making a POST request to Jenkins with CURL. The build should start with this:

http://your-jenkins-address:8080/buildByToken?job=your-jenkins-steampipe-build&token=YOUR_SECRET_TOKEN

Unity Cloud Build webhook

In UCB, navigate to the build's "Notifications" page and add a new webhook. Copy the previous URL, check the "ProjectBuildSuccess" event, make the hook active and save. For this guide I didn't use a secret or SSL verification.

Configuring the UCB webhook that triggers the Jenkins build
Congratulations! You've successfully automated the build pipeline from Unity Cloud Build to Steam! You can now start up the cloud build and enjoy the completely automated Steam build process! Please give feedback in the comments!

Note: After a while of using completely automated builds (that triggered every time we pushed something, however trivial) we started to feel frustarated about the amount of builds adding up to Steam. We resorted to running the Jenkins build nightly and sometimes manually when needed. To use nightly builds, the only thing you need to do is disable the UCB webhook and add a "Build periodically" configuration to Jenkins. That way UCB keeps the continuous integration running while Jenkins only uploads Steam builds every night.


4 comments:

  1. Hey there, first off great guide! I'm having a bit of trouble though when it comes to the step of connecting Jenkins to Steam. I'm running Ubuntu on Windows and when I'm trying to run the execute shell script "/opt/tools/steamcmd_linux/steamcmd.sh +login your_login_name +run_app_build $WORKSPACE/app_build_123456.vdf +exit" the file steamcmd.sh can't be found. I've tried changing the path to the folder where I've found this file(steamcmd.sh wasn't where it was in this guide for me) and it's still unable to locate the file. Is this an issue related to file paths regarding Linux/Windows, or with the Ubuntu Shell and its access to files? Thanks!

    ReplyDelete
    Replies
    1. It's worth noting I've replaced the placeholders with my information.

      Delete
    2. There are multiple possibilities that might cause this.

      Maybe the file running permissions are incorrect for the user - have you checked the command works when running directly from the shell outside Jenkins?

      It's also worth noting I downloaded the linux-specific steamcmd package from the developer site. (and not the one the developer page offers by default which also includes some linux executables). I couldn't get the default package working for some reason but the linux-specific package worked immediately.

      Delete
  2. Thanks for this great articles. Im a unity mobile game developer, automating the build process really helps a lot!

    May I ask can I deploy the apk to real android devices after downloading the zip from UCB?

    ReplyDelete