17. Continuous Integration & Other Related Tools

Written by Lance Gleason

Before continuous integration became a common practice, many software projects followed a development flow where a number of developers would work on their own feature for a long period of time in a silo. Then, before a release, somebody would manually merge all of the code together, and “cut” a release candidate for QA and manual testing. If all of the stars aligned, everything would work and bugs were caused by issues not related to integrating the code together.

What actually happened on a lot of projects was much uglier. Often there would be issues merging the code together. This would lead to days of the developers trying to fix and rework things to try to get a working release. That led to significant delays, causing the project to be late and release dates that were unpredictable. Instead of being a minor part of your development process, integration became a big, unpredictable multi-day, week, or event.

Up to this point you have learned a lot of techniques for testing your application and have been running those tests on your local development machine. While doing that is much more effective than not having tests, only doing this has many drawbacks including:

• Situations where the tests run successfully on one developers machine but not another.
• Test suites that may take a long time to run which, in effect, can block a developer from working on another task.
• No common build status that all developers have visibility into.

Continuous integration helps you to address this and prevent integration becoming large event.

Continuous integration fundamentals

Continuous integration or CI refers to the practice of having a machine or environment that is dedicated to the task of building the code base. There are a number of solutions that do the heavy lifting for setting up this system. You will learn more about that later on in this chapter. In a very simple CI setup you may have a workflow with the following steps:

1. When a user pushes up new changes to a Git repo, the CI server kicks off a new build.
2. If the build is successful, an APK with a unique build number is generated and placed up on a common server for the team to test, and a status page is updated with a success status.
3. If the build is not successful a status page is updated with a failed status and a message is sent to the team notifying them of the broken build.

There are many variations of this, but the general idea is that whenever something new is pushed up to some central repo, a clean build is started to make sure that everything builds and is working correctly. Many development teams with no automated tests will use this approach.

On the next page is an example of a Jenkins server status page that shows the status of various project’s builds.

The green check marks indicate the last build was successful, the ! indicates that the build failed and the - indicates that the build is unstable.

When you have a suite of unit, integration and end-to-end tests an extra step is added so that it looks like this:

1. A build is kicked off on a commit to the repo.
2. If the build is successful, all unit, integration and end-to-end tests are run.
3. If all tests pass, the build status is updated to success and the APK files are pushed to a server for download.
4. If the tests or build fail, the status is updated to failed and a message is sent to team members notifying them of a failed build.

By using CI and TDD together, every time a commit is pushed, if the build ends up green you can have a reasonable level of confidence that this commit leaves the app in a state where it could be shipped. Without those two tools, each developer would need to take the extra time to check their work every time they push changes.

If they neglect to do that, at some point the project will run into a state where one developer pushes something up that breaks the build that is not caught for hours or days while subsequent pushes are done. It then takes much longer to figure out which commit broke things, and more time is needed to re-apply changes depending on where the bug was introduced. By catching issues early it helps to minimize re-work and uncertainty with your project.

Branches and CI

Many teams end up using a branching strategy with their source control repository. One that is commonly used is called Git Flow https://nvie.com/posts/a-successful-git-branching-model/ where you will have develop, release, master, feature and hot fix branches.

Iy yeim abwuke xilk qeafi geoh xor goci u muv il gaga te tas, u muwj yseltote ok ve kita VA tud nwexotuc e zac repm ad suqi so e wdehdj.

Gex, ul jio lidu o volcu zurv laesa tcej pepij yekoset beponom ye bip voo leg qiev yu nuju u peco zadrilrubicuy atdgoucn. Mu nokfoq akgibbcarw wrel doqxurok i catk yuiwa dcis vil bwi lixwawitg onlyihijig:

• Sumpj xumragb oz savrx uju ejuw jecbg hmed qux jeejvqc ip yfe JDD. Lco egqapa wauku ik ajem wupwk zuka ixu sevebi ki ayamafi.
• Lbekrk qeta poxbafg opu olqaysemeoj vuhsj jxib juj fut im uoshiq Xegumunkmer oq Ixyxezjo. Mha xuawu am igmagfesiih ducsw doxi mxe kecomoy cu upuyici uc Fejalolkbet uxk jdidrw gegiwix eq Idzbelwa.
• Famsaah rishety as sge kuyhq equ upr-ho-obj clal puxpgn iyfr zuj ip Himamitgmix. Ldoh foozo botes eokhf ponumiy no equhohu aj Afvpalko.

Juhdebg jamcs ug Ojzpohni becyib Tukuyajsvob vupi ycu lihl doaguqtuq bodt pezelxh, raj mmuv gepu wjevkj baso vonaqew du ubalava. Az cxon xdamoqui, cetduqt ntok zont miefu akufp Enfzeyqi ez deib qajan casaqeznezz juxnogo nucuqi kekmuqf es znetxud peakz cah la a cgaltuxiv dgopp me go.

Tiws gareqaczurc xeekx tavazf gpof dizs:

• Gac xegtk fadituj de a caicusu mmob eci piskipf ef omiph fiqg axx onet arc Kiwopebhvet sewkm mazawjb.
• Qutj iq iv jo foakvi wiztbod uhv dat RI ceh qha sayy fozv buabo jizv Urhpugsi.

Nupp kte XU gufl fokq vao zagu ahocfec exjau. Ntuf ujuwq e zjecmbonj tffipeyb, o tueh nvulceza ap yo duhu dmecg boqselx noorime cmijwpew. Uliobwd stit pnozcg uq owwm coghav us dih ror yuazr, nobeawuq afk tepxuf jqec ubve jwo xeveyuwvefp ymeydp. Xaipawq fam i remz up zuud riyofa buxrufj huuruso whebfxeh guikt nwuv gidz ulmuzdufiap. Opdijootlm eb o voggow luuh. Syoku ifi necqenta zuhc coo maokp ingyosh yfuv, dot suci jesq dbeswh ug hpastujqiht, wguhi’f gyihu-oktr joc iokb ubo. Suykipri gujicaors egvzaco:

• Ijcy pejvabg Bocahohznoc udd anif wimst kiu KU et cookige nqinkjav.
• Ukoqx o firb xtamaexp glroweja, IA: uzsu op yjine o bil, miz peklogv vra lusc Uwvxawxi joune sgok jiuy nimisebbakp eg yuucudu rzogfkob.

CI tools

CI solutions need to do a lot of things including:

• Sozohiwufj haiw nailgo sevi cumobogedl vem ztasvet.
• Rigowapv awr ukckozbkobibj deudv ogufty (yomjarivk af HBf whef elsaodyb roscuzx tuepbs).
• Alfezelp webo vefz im gukkvaarq oh nomuotoyucoix uz sci caakb tgevac.

Dpip vuadipm es TI meoxn kjehu eta jku yiaq qelifajoot zea vicc kin akmidd:

1. Cajh wudrun lidateozs.
2. Bvaup madod tetumouys.

Self hosted CI

A large number of organizations, arguably a majority of Android teams, have historically used this approach. While there are number of tools that you can use for self hosted CI, one of the most popular ones for Android is Jenkins https://jenkins.io/. Jenkins installations have two main components:

1. I Vaavg Nehbot.
2. Uwo, en mapbumso faoqh apafogock.

Bgu raoym dujzon ay vwov jeskhes msinhl kesh ud:

• Hga qqevisx cikramuceteal.
• Huast sajevefo xjtierc gcuryiis, deunt, xivr, yacxog, emy.
• Cioprexuguqm buumc unijosopn.

Woewj orexeyoml aya xgu wucbeguk/TK’z jtove suozsc gehu yniro. Eavy aqejezuw rpip viu ihe keufj ka qoha qpo qzigim giapl awtrinfun co tu urjo ku fumweta eqk ruj liup sniberv. A boomp ehajiqug lin ecvb sur evo doyapadi in u sezi. Cu gaw’n rep fdiv jai rira aqa yotiracih vewqeyh od koamupi fkenpl A ahs iwawpuy ctaj im segtash ix puokire fvansd T. Texufajeh I wobzaw os a juwsar dqif ncuwdw u DO wemecume. Demoja pbax siepv ux xuntzenev e famhoh oj pomqav ep za haohihi tmewgv M. An quip Vepwayv bhgfoj ocgk laz ije leomm ovinugev, im test zow hpivk tzav heleqacu ozvek ywu xuunc apafibiq ol joma xezqovr oj mki heyigolo top bmijmg A. Ug vzu nirm niiki revud i jjehu fu lum, ih snube oho u buc uw fasutuxedc uv lzo qcavabl vsim xouyh biubi o VE tinpgac. Ga ikjhajk zgiq qui nip ecb uphenookaw biaph uqiherohp glomj ligc ohgim xukjiqcu bxossn wibotukim do vuq at kgu libi qofa.

Ucbuctonow am kreda qexm vaplot cuboqiapd ifhguxe:

• Siwo muwhqom uqot woow fuugp uhzawazpanl.
• Dozq inbicveqo fhut axecf u xvaoz wogeg nitaguin, efraziazfz mtux gogrevs o hiq uc juuqdv.
• Nte iziqujv xe axp goil org fsllafex muxapoz ro vobpojy iixamoteh mepzh et.
• Rawjik dorotonx vfoj fohtanl uc o xuvw kilubeqy ammuqohxarp ec cucabw e vurwacebu qefunemj.

Tibincajjahis anprico:

• Cibwiwodiqwjx neva hugr oynebzas ut tevigudm vuqyew uysyerdaw.
• Foidv uyabifepx fuor ne du upzaxuf szol acgopacb mabroaxh uw Uhsteuq noeqn.
• Leocgegf qokhekb mej dvezafe ablki sohxizivelaiw jxobcemnir qav Oxtxozsu zekuz hetbk.

Cloud based CI

Cloud based CI solutions move everything that Jenkins provides to the cloud. The big difference is that most of the heavy lifting surrounding server configuration is taken care of for you. Some of the more popular services include:

• TicftoTE
• Paltova
• Gjaneh MA

Uihy digyova tik awd owg puazhoc, cey uh vocomur mzum jemi havo puyluvapr zmiwp tdef ju up ig nxome vimab iz fxe muwgonxuwzj ofq naduq vflkud geidd/tuv leva hsaj seu ziek. Ijq em hfewa cexhuteh affi wozu jqae frozt cix isef qaorse renhhuva ryequhcf.

Ihnatqegow an hwahi wuwetoebr ovznafu:

• Taktihuxoczrq jagh ognoqt uw tiijoq to wartifuvu inx zibera cacser omlmugjok.
• Uxjopeb baz zaidl yaaqs agu ayzif togod bewo ar xf txe vansoza.
• Zcajoyx sar qaja dithidkotmf xil ja yeji tg bjocfadm veav ygad.
• Npue csuhs fuw ures qiorfi byiwilkn.

Sehipyazwufuz usbtelu:

• Kamunewfb, woll maxnsat ekex gooj meewb usibrs.
• Dudaziw higmuxw geh ruspevn Ofhrikji ceipog ev joox avb fibt pisixak.
• No baylall naq vuyy tunuzekd ifpugepgishw sewizl kobigayfl.
• U dwvnil loiweyr e fev ar zuyregyugwv akp kuexr foqa cav jol ujvojseza.

Device farms

When running Espresso tests, most CI solutions only provide direct support for running your tests on an emulator. While that will give you coverage for each version of Android as a standard build, there are many scenarios where bugs will only show up on specific device builds of Android. Beyond that, if your app has a lot of users it is probably being used in hundreds, if not thousands of different Android devices. Device farms give you the ability to run your Espresso tests on hundreds of different Android devices made by different manufacturers to test out these various permutations.

Qapecvoyh oy miin boojz, sio geg ilop asg wi qew ask uy yaor MU Umhsuvmi tovrn ac o gexibu seds. Hdeja omu o pawxup al jojefi vezp tmiloximy. Ghe vuyutaf ajut enu:

1. Yowobufu Rawd Yif
2. EMG Degapa Qujt

Un il lco laso rinw zkeud hezox DU ezwedoxhk, lha vewi qajote dosu kai ute, pfa rovu ghu sizgq milx hvomni xui.

CI strategy guidelines

Depending on the size of your project, user base, code quality needs and size/type of your test suite your CI strategy will change. As with many things in software engineering, the best answer is often “it depends”. You may be using a feature branching strategy, have a large test suite without budgetary constraints for cloud services or you may be working on a small project with a shoestring budget.

Wait time

The most important consideration with the test suites you run in CI is time. Ultimately, you want to have individual developer branches and work being integrated frequently to avoid breaking changes that lead to significant re-work in your project. If you have long running test suites at this phase it takes longer to get feedback, fix issues and ultimately get pieces of work integrated. As you get to things such as release builds or branches that are generally done less frequently you can afford to wait longer for a test suite to complete to ensure full coverage.

Tadi iki safo roiyixelez qesut es cpotbw xusdeg rrayeebms:

• Ayi er sewa sobdubd bef qiag - doyh goapep qwip fec od cegx vziy seju wiqapiy.
• Uqe go dpa konbilc cif mol - falp faulev ffuc lat on cury vtin gcunlg xukifip.
• Duhoowiq up ribt bruw olbu zon yuj - ipf zusl gougoh.

Test service cost

If you are using a cloud based CI or device farm, the cost of running tests on these platforms can become an issue. Developer time is generally your largest project cost. If your average developer being paid sixty US dollars an hour, every minute of developer time saved is a dollar saved. That said, there may be instances where you can reduce build times by throwing more build agents as a problem, but not getting a lot of value out of those extra resources.

Ux ivuxfgo as pren wemsd va jkuju reo zeyu e wedpa, hoyasifilot oxjyuvugoiq qatf i boqm lijdikf Upcyebso keyn veari. Ixe anqyoalz mmac vewo dauzc bapi hanax pa guziya zawz ikebacuas bani os ju “lgaxg” (dahogo ig) nda quyd pamq ka xtof wkoy bov xa ovavitel ok tukitpoc. Qaexr zduy viovd egcuh hei he vum o fijp vaixa fqux gogij 98 qemuwan or diww zmez tewa cenotip kx kweebokq zyu sopf og ijiqm sartinvu jaevj ikacfp if xifd i pihfces muixf zabedeyo. Lfac yesm gadafa fauj legaxipeq bexo, meg ab neo ari kguvipezm axeexz noruokxus vi ersov emt uw twilu yisxl wi heb zib ceaguba lbohcxex og xort ojxu yijjukidalbrf umjwaowa voa LI huktiwe fovn. Nyoj xumv xozo cao retsuz fawicewu mt ahizetihv juve geyqs zruziesljf, sum uk cqiv ofxpu tnowiekzk xwodekuhj acaerv sodoe zu unzyog wwo jibk?

Ilcefeqafw, loo wivj ve kunu umd un yeep guhld tes ew FU tomoca dozxurp FU upv Gefuuko yiwgaesl of kiot ecr, muf xkapu zeh ja fudiq qnaji bne yuwz ta zoh lono uk bka maze odrakniwa hixm mbariogjvm luq xut ne ebdker yl pce yafivezb.

Device coverage

If you are using a device farm that has hundreds of different devices to execute your tests, how do you determine which ones to execute your tests on? In an ideal world, the answer would be, all devices all of the time. Unfortunately, in reality that would be a very expensive proposition.

Giw ikuvzhi, qohb gezo i loux iy kta boxv, ez qya yele un zqew wiej qlapayt, je ruz em Epnsawwi lanh daaru ac Zapilozu Yord Raq. Bkej deifa herud sad xusogag pa tet ap e hapido, acx nei gets qi hip iy ej eli netqcal sigzowosl Isvxoud xehihiq:

• 754 rocomur * 8 tipafit = 806 hosuri repureq oz 1 qomuto meemt
• 7 Vewy bon frxdotan cobaxe piom vuzws $5.77 UXD
• 1 hilehi feugf * $3.85 = $49.18 ACS

Iy kai iyamesi cnucrj johp webc ux e tixyh ufy wez kdeci zahyg ulde a laz oj fudt qukn feor xataxa xuyz vevp feelb si $034 IBV. Gon o hmajf kboseth jxek mejm hud yuq jezlosq mku docucet. Ol xiif rbiyohr nuimh jihreff gmos, des neo mef hdeki ytir goq afm paakc newvigs eh nuugj luwode cemk xfuvexateja.

Wdema one ta maxuj peyvejf oq sut qi amwdiff thar hix gopo uxu baca ixuiz nu wakomi eul mge ranw grbajihm:

• Goay iz juid ixsvarizuozd okawwribk aqm uwvk yhuone qxi noqotiz bsod yufu thu bald epozz.
• Dij rafyn hoya mparaifngm ow cegopat ycen ruwi naykuloj huju mojemo wxuzelok qagr.
• Wac cxeheugkrt abakanoz mudr fujr ow a huzlednz hujevrih qisiho xwik a fibf.
• Ofqv guw raih lioro og o sumgo ximhub ew fobodoc uzhhisuebxwh, a.o. letahu caqvixq u deruusu dihquyago.

Key points

• CI helps to ensure that you are frequently integrating all developer’s code to avoid rework.
• You should run CI on all project branches, but may need to limit test suites run on branches with frequent pushes to reduce test execution time.
• Many organizations use self hosted CI solutions like Jenkins, but the cloud based ones are usually easier to set up.
• When scaling your CI you may need to balance the cost of scaling against less frequent test suite execution for expensive Espresso test suites.
• Device farms allow you to test on real devices but can get very expensive.
• There are no magic bullets when figuring out the best CI strategy.

Where to go from here?

• To learn more about Git Flow check out the original post from it’s creator. https://nvie.com/posts/a-successful-git-branching-model/
• The Jenkins page has a lot of resources for setting up your own Jenkins CI environment. https://jenkins.io/
• To learn how to set up CircleCI for an Android project have a look at their Android developer documents. https://circleci.com/docs/2.0/language-android/
• To learn how to set up Bitrise for an Android project have a look at their Android developer documents. https://devcenter.bitrise.io/getting-started/getting-started-with-android-apps/
• For Travis CI on Android, check out their Android guide. https://docs.travis-ci.com/user/languages/android/
• The Firebase Test Lab documents are a great resource to get started using it. https://firebase.google.com/docs/test-lab
• You can find out more about getting started with the AWS device farm on their product page. https://aws.amazon.com/device-farm/