Thursday, February 16, 2017

Parta Games Tech Post #1 Automating Builds & Steam uploads with Unity Cloud Build, Jenkins and Steampipe

Intro

As our blog has mostly been focused on other things, the thought of making a tech post about a specific topic has been on mind for some time. A couple of days ago I finally got around to researching and implementing a thing I've been putting off for too long: automating the Windows build of a Unity game with Unity Cloud Build and subsequently sending the build artifacts to Steam with Steampipe using the Jenkins continuous integration server, essentially making the two servers work together to solve a problem.

Why?

I strongly believe making software builds manually starts becoming a redundant task really fast when the need arises to continuously distribute builds to third-parties, other services or for example QA. This includes the separate platform-specific builds of a single game as well as builds for libraries etc. I also wanted to automate the final step of sending binaries to a store vendor's service, in this case Steam. The sooner a build pipeline is automated the more time is saved when instead of making the builds manually the developer can focus on other things and trust that things are scheduled or triggered conveniently. Creating the whole pipeline from scratch might take a day or two at worst but I believe the patience starts to get rewarded quickly.

Services & Tools

Unity Cloud Build was selected because it removes the need for our own dedicated build servers, has a handy web-api with sufficient documentation and was very easy to set up with our existing version control solutions. It's also a no-brainer if you're making a Unity game because the service is provided for everyone. This guide is created with a Unity Plus license, but that shouldn't change anything, as the service is also provided to free version users with a lower usage priority.

Jenkins is a free & great (if not the best) continuous integration server with awesome support for especially "freestyle" builds. Freestyle essentially means I can configure a series of build steps using whatever method/script/plugin is available to me. I use it for a lot of things from automating static website uploads to creating final software distribution builds. Jenkins is very popular in Java software development and the server actually is a Java software itself (needing Java runtime to function), but don't let that scare you. It's plenty powerful, stable, battle-tested and very configurable. There are also tons of plugins for myriad of use-cases.

Guide requirements:
  1. A Unity 5.x project (source controlled, our choice is Git)
  2. No prior knowledge of Unity Cloud Build
  3. No prior knowledge of Jenkins CI server
  4. Steam developer account and working Steampipe configurations. Essentially I assume you know how to upload a game build to Steam and can publish it.

Setting up the game build in Unity Cloud Build


Start by creating a new project.


As you probably know, a Unity developer account is needed to access the Unity Cloud Build service. If you've recently created your project, chances are it already exists in the Unity projects page and you can just navigate to it and enable build service. If not, create a new project. Just make sure it uses the correct organization and references the correct project ID (the same you can see in Unity services tab).

When presented with the option, enable Unity Cloud Build

Enter the source control information and give UCB the proper rights to access it. If you're somewhat familiar with source control security, setting this up should be a non-issue.


Enter source control information


The next step is to add the new platform target. As we're automating a Steam build, win32 platform is needed. Default platform config should work, but you may need to change the executable name to something sensible. In my case the default executable name was something else and I noticed it only after downloading the resulting ZIP-artifact.


Default config with a sensible executable name
And that's it! Unity Cloud Build setup is now done and you can start the first build. For me it took a couple of tries for some reason (I guess the service needed to warm up a bit), but without changing a thing the build started working. If everything goes well you should see rows of something like this appear in the History tab:

A successful win32 build
Although setting up UCB is probably the easiest step in this guide it solves a very important problem: automating the game build from source control and providing us the build artifacts.

Testing the Unity Cloud Build API

Now that you have a working UCB project you can start querying the API for information about the build. This is a convenient way for the Jenkins build to fetch crucial bit of information: the download links to the build artifacts with proper access rights. The relatively unknown UCB API solves this problem quite elegantly.

1) First find out what your API key is. Instructions to locate it can be found in the Unity Cloud Build API documentation. Keep the key handy as you will need it later in this guide, but note that it's not public information as it allows anyone to access your Unity projects through the APIs.

2) You are now ready to query some basic information from the API. I used a terminal & CURL to get the results, but you can use the HTTP client of your choice. The following query will get you information about all successful builds in UCB (note the authorization header with the API key):

curl -H 'Authorization: BASIC YOUR_API_KEY_HERE' https://build-api.cloud.unity3d.com/api/v1/orgs/your-organization-here/projects/your-project-name-here/buildtargets/default-windows-desktop-32-bit/builds?buildStatus=success&platform=standalonewindows

The result should look something like this:
The essential part in the response JSON is the value of path links/download_primary/href. It contains the direct download URL of the main build artifact (which in this case is a ZIP-file) with AWS credentials. This is exactly what you need to have Jenkins download the data and upload it to Steam.

Setting up Jenkins CI-Server

Now that you have fetched the information available through the Unity Cloud Build API you can continue setting up the Jenkins CI Server. Jenkins is available for many server platforms. In this guide I use Ubuntu Server 14.04 64-bit for which Jenkins is directly available through the default apt package manager. For installing Jenkins on other platforms please consult the official site.

sudo apt-get install jenkins

And that's it. Jenkins is installed as a linux service daemon and is starting up. The server is used through a web-interface which should start accepting requests at http://your-jenkins-address:8080.

Setting up the Steampipe build in Jenkins

Start by clicking "New item" from the leftward menu. Type the build name and select "Freestyle project" and press OK. We will work based on the assumption that this build will be automatically triggered by the successful cloud build.

Note: The way I configure and create the Steampipe app and depot configuration files during the next step is not a perfect solution, but for the sake of this guide it was fairly quick to set up. In the future it would probably be better to pull the files from Git or similar location.

Create a new "Execute shell" build step and add the following configuration:


Next you'll fetch the latest successful build data from UCB. You've already tested how the UCB API works so it's quite easy to parse the data out of the response JSON. Groovy scripts in Jenkins builds are quite versatile so it's very convenient to have the script unzip the files too. You'll probably need to install the "Groovy" Jenkins plugin in the "Manage Jenkins -> Manage Plugins" section for the build step to appear as an option. Create a new "Execute System Groovy Command" build step and add the following script:

The final step is to let Steampipe do its magic. Create another "Execute shell" script and add the following command (where $WORKSPACE corresponds to the Jenkins build workspace folder):

/opt/tools/steamcmd_linux/steamcmd.sh +login your_login_name +run_app_build $WORKSPACE/app_build_123456.vdf +exit

Note: I was running Jenkins on Linux, so I had to log in to steamcmd manually as the "jenkins" user for it to cache the login credentials. This way you need to submit the two-factor authentication code only once and the script will be able to login automatically from that point on.

Now the build is complete. It won't trigger automatically just yet, but you can run it to make sure everything works. The script will first create the config files. Then it will download the build from UCB and send it to Steam.

Setting up webhook to trigger Jenkins build

Now that you have created the two builds that work individually, it's time to make them sing together. You'll set up a webhook in Unity Cloud Build that creates a HTTP POST request to Jenkins triggering the Steampipe-build automatically, every time the cloud build is successful. First you need to configure a couple of security settings.

Jenkins security

Install a Jenkins plugin called "Build Authorization Token Root Plugin". Do this in "Manage Jenkins -> Manage plugins" page. This plugin makes it relatively easy to give UCB access to Jenkins' REST API while using a single security token (UCB does not support HTTP authentication, which is by default needed by Jenkins).

Second, you need to configure Jenkins to receive "Anonymous" requests through the built-in REST API. Go to "Manage Jenkins -> Configure Global Security" page. If security is not enabled, you should enable it now. For the purpose of this guide, make sure matrix based security with Jenkins' user account database is used. (Access control should be secure, with user accounts only for people who need access)

The "Anonymous" user needs access to builds. Check the box labeled "Build" (can be found from the "Job" column) and then click "Save".

Build checkbox for user "Anonymous" in the lower left corner
Finally, you need to add a build-specific authentication token. Go to the build page and click "Configure". From the "Build triggers" section, check "Trigger builds remotely". Think up and enter a secure token and save the configuration.

Entering the authentication token
That's it for Jenkins. You can make sure everything works by making a POST request to Jenkins with CURL. The build should start with this:

http://your-jenkins-address:8080/buildByToken?job=your-jenkins-steampipe-build&token=YOUR_SECRET_TOKEN

Unity Cloud Build webhook

In UCB, navigate to the build's "Notifications" page and add a new webhook. Copy the previous URL, check the "ProjectBuildSuccess" event, make the hook active and save. For this guide I didn't use a secret or SSL verification.

Configuring the UCB webhook that triggers the Jenkins build
Congratulations! You've successfully automated the build pipeline from Unity Cloud Build to Steam! You can now start up the cloud build and enjoy the completely automated Steam build process! Please give feedback in the comments!

Note: After a while of using completely automated builds (that triggered every time we pushed something, however trivial) we started to feel frustarated about the amount of builds adding up to Steam. We resorted to running the Jenkins build nightly and sometimes manually when needed. To use nightly builds, the only thing you need to do is disable the UCB webhook and add a "Build periodically" configuration to Jenkins. That way UCB keeps the continuous integration running while Jenkins only uploads Steam builds every night.


33 comments:

  1. Hey there, first off great guide! I'm having a bit of trouble though when it comes to the step of connecting Jenkins to Steam. I'm running Ubuntu on Windows and when I'm trying to run the execute shell script "/opt/tools/steamcmd_linux/steamcmd.sh +login your_login_name +run_app_build $WORKSPACE/app_build_123456.vdf +exit" the file steamcmd.sh can't be found. I've tried changing the path to the folder where I've found this file(steamcmd.sh wasn't where it was in this guide for me) and it's still unable to locate the file. Is this an issue related to file paths regarding Linux/Windows, or with the Ubuntu Shell and its access to files? Thanks!

    ReplyDelete
    Replies
    1. It's worth noting I've replaced the placeholders with my information.

      Delete
    2. There are multiple possibilities that might cause this.

      Maybe the file running permissions are incorrect for the user - have you checked the command works when running directly from the shell outside Jenkins?

      It's also worth noting I downloaded the linux-specific steamcmd package from the developer site. (and not the one the developer page offers by default which also includes some linux executables). I couldn't get the default package working for some reason but the linux-specific package worked immediately.

      Delete
  2. Thanks for this great articles. Im a unity mobile game developer, automating the build process really helps a lot!

    May I ask can I deploy the apk to real android devices after downloading the zip from UCB?

    ReplyDelete
    Replies
    1. It's possible to hook UCB or Jenkins to send the finalized build to a service that deploys APK's to devices, for example Hockeyapp.

      Delete

  3. We offersbest java training in chennai.We at CodingCub not only concentrate on training, we also help our students with placements. We have a separate team of people working with the HR in various top MNC companies to find the right job for each and every candidate with us and help them in getting their dream career.

    ReplyDelete
  4. CodingCub was started by a most struggled IT professional to make it easy for the talented people to reach out their dream without great struggle. There are lot of people who are striving hard to get their career start but we at CodingCub make it easy for our students to get the career start in a couple of months with the best Software Testing Training in Chennai.

    ReplyDelete


  5. We offers best java training in chennai.We at CodingCub not only concentrate on training, we also help our students with placements. We have a separate team of people working with the HR in various top MNC companies to find the right job for each and every candidate with us and help them in getting their dream career.

    ReplyDelete


  6. Colors Institute of Interior Designing to help you learn interior designing. Theory classes, 3D visualization classes, material study classes, designing classes, internships, industrial visits, guest lectures to keep you updated on the latest happening, color selection process, and real hands on through live projects is all include in the training for diploma in interior designing.

    ReplyDelete


  7. Colors Institute of Interior Designing to help you learn interior designing. Theory classes, 3D visualization classes, material study classes, designing classes, internships, industrial visits, guest lectures to keep you updated on the latest happening, color selection process, and real hands on through live projects is all include in the training for diploma in interior designing.

    ReplyDelete
  8. we offersbest java training in chennai.We provide a quality of training, certifitcate and 100% placement.

    ReplyDelete

  9. Colors Institute of Interior Designing to enable you to learn inside outlining. Hypothesis classes, 3D perception classes, material examination classes, outlining classes, temporary positions, mechanical visits, visitor addresses to keep you refreshed on the most recent happening, shading choice process, and genuine hands on through live undertakings is all incorporate into the preparation for certificate in Diploma in interior designing.

    ReplyDelete
  10. CodingCub provides the best Software Testing Course in Chennai with a restriction to batch size to make sure that we can provide attention on each and every individual and help them to learn Software testing in-depth.

    ReplyDelete
  11. We give the best Java Training Courses in Chennai and we have the best mentors who are taking a shot at Top MNC organizations with right around 10 years of involvement in Java. we at CodingCub being turned out to be the best Java Training Institute in Chennai causes you get learning alongside the activity.

    ReplyDelete
  12. The best career for people with this mind is fashion designer. To start a career, first step is to join a Fashion Designing Course in Chennai and the course is best offered at Colors School of Fashion Technology. There is lot of things like in you then step into Colors School of Fashion Technology to start your career as a fashion designers and become a world class fashionista through the Fashion Designing Course in Chennai.

    ReplyDelete
  13. Colors School of Fashion Technology started by a group of top fashion designer in Chennai and now with 10 years of success looking forward for more successful years in providing the best fashion designing course in Chennai. An achievement of making a lakh of successful world-class fashion designer with a great career in the top fashion companies.

    ReplyDelete
  14. We offers interior designing course in chennai. We also provide a quality of training,certificate and 100% placement.

    ReplyDelete
  15. This comment has been removed by the author.

    ReplyDelete

  16. Colors Institute of Interior Designing to enable you to learn inside outlining. Hypothesis classes, 3D perception classes, material examination classes, outlining classes, temporary positions, mechanical visits, visitor addresses to keep you refreshed on the most recent happening, shading choice process, and genuine hands on through live undertakings is all incorporate into the preparation for certificate in Diploma in interior designing.

    ReplyDelete
  17. Colors School of Fashion Technology started by a group of top fashion designer in Chennai and now with 10 years of success looking forward for more successful years in providing the best fashion designing course in Chennai. An achievement of making a lakh of successful world-class fashion designer with a great career in the top fashion companies. No interest in books, papers but thinking about how to make a great career for yourself. No worries, Colors School of Fashion Technology, the Best Fashion Designing Institute in Chennai is here to build a great career for you in the fashion industry and make you a world-class designer.

    ReplyDelete
  18. Colors Institute of Fashion Technology with the best Fashion Designing Course in Chennai is here to shape your interest on fabrics and colors to make you get a career path in fashion design industry.

    ReplyDelete
  19. Colors Institute of Fashion Technology with the best Fashion Designing Course in Chennai is here to shape your interest in fabrics and colors to make you get a career path in the fashion design industry.

    ReplyDelete
  20. CodingCub provides the best Software Testing Course in Chennai with a restriction to batch size to make sure that we can provide attention on each and every individual and help them to learn Software testing in-depth. We also conduct one-on-one on a regular basis to clarify the doubts and understand the pros and cons of the training improving on your training process.

    ReplyDelete
  21. Colors Fashion Schoohas started the Fashion Designing Course in Chennai with an aim to teach the students from the ABC of fashion designing to the expert level fashion designing.

    ReplyDelete



  22. Colors Institute of Interior Designing to enable you to learn inside outlining. Hypothesis classes, 3D perception classes, material examination classes, outlining classes, temporary positions, mechanical visits, visitor addresses to keep you refreshed on the most recent happening, shading choice process, and genuine hands on through live undertakings is all incorporate into the preparation for certificate in Diploma in interior designing.

    ReplyDelete
  23. CodingCub provides the best Software Testing Course in Chennai with a restriction to batch size to make sure that we can provide attention on each and every individual and help them to learn Software testing in-depth. We also provide 100% job guarantee Selenium Training Course in Chennai to make the job availability to the students easily and we have a separate team of HR working with the MNC companies to the best career path to our students.

    ReplyDelete


  24. we build portrays your business effectively in the market. We utilise the World Class Technologies to give you the outstanding and most reputed digital asset.Getting online is easy. Succeeding online is a different story. You’ll need more than just a beautiful website to stand out these days. Online marketing solutions. Conversion-based web design coupled with a lead generating marketing plan, your online success is inevitable.

    ReplyDelete
  25. CodingCub provides the best Software Testing Course in Chennai with a restriction to batch size to make sure that we can provide attention on each and every individual and help them to learn Software testing in-depth. We also conduct one-on-one on a regular basis to clarify the doubts and understand the pros and cons of the training improving on your training process.

    ReplyDelete



  26. we build portrays your business effectively in the market. We utilise the World Class Technologies to give you the outstanding and most reputed digital asset.Getting online is easy. Succeeding online is a different story. You’ll need more than just a beautiful website to stand out these days. Online marketing solutions. Conversion-based web design coupled with a lead generating marketing plan, your online success is inevitable.

    ReplyDelete
  27. CodingCub has been rated to be the best Software Training Institute in Chennai and we strive towards growth of our students which in-turn grows us. Our student’s success is the success to our institute and thus we always strive towards our student’s success. CodingCub has collaborated hands with the leading technical architects and leads to be a part of our training and provide our students the knowledge they have got throughout their career.

    ReplyDelete
  28. We also provide 100% job guarantee Selenium Training Course in Chennai to make the job availability to the students easily and we have a separate team of HR working with the MNC companies to the best career path to our students.

    ReplyDelete
  29. CodingCub make it easy for our students to get the career start in a couple of months with the best software testing training in Chennai. CodingCub has been rated to be the best Software Training Institute in Chennai and we strive towards growth of our students which in-turn grows us. Our student’s success is the success to our institute and thus we always strive towards our student’s success.

    ReplyDelete