13. Running Live Experiments With A/B Testing & Feature Flags

Written by Vid Palčar

In the previous chapter, you learned about two Firebase services — Firebase Analytics and Firebase Crashlytics. While browsing through your Firebase console, you might’ve noticed that it offers quite a few other interesting services. So, in this chapter, you’ll look at two of those additional services — Firebase A/B Testing and Firebase Remote Config.

After thousands of hours spent learning good UX and UI practices, studying marketing and working on product development, there will always be situations when you won’t be sure about a specific design decision, call-to-action strategy or advertising text. Even when you think you made the right choice based on your experience, there might be cases when taking a different approach would work better. You might think these differences are irrelevant to your app, but the truth is that an increase of only a few percentage points in user engagement might reflect enormous gains in your sales. The math behind that is quite simple: more sales = more money.

But now, you might be wondering how to determine which solution to a specific problem will work best for your users. That’s where Firebase A/B Testing comes in handy. Publishing a new release of the app for every change required for your experiments would be very inconvenient. Firebase Remote Config has you covered there. With the help of a so-called feature tag, you can make changes to your app on the fly.

With Firebase A/B Testing, you can run experiments in marketing, advertising, politics, product development, design, etc. As mentioned already, you’ll have to use Firebase Remote Config to implement those.

In this chapter, you’ll learn how to:

• Add a feature flag to a specific feature.
• Perform changes in the app without rolling out a new version.
• Set up an experiment in Firebase.
• Perform and track experiments with Firebase.

While going through this chapter, you’ll work on the starter project from this chapter’s assets folder.

A Deep Dive Into A/B Testing

The introduction already mentioned a few use cases of A/B testing. But before performing your first A/B test, you have to dive a little deeper into the theory behind it.

A/B testing is a simple experiment in which two different versions of a variable — A and B — are compared. The testing group is randomized and usually equally distributed between both samples. More complex cases might have more than just two versions of a specific variable. The use of A/B testing started far before the era of modern computers. Originally, A/B tests were used to figure out which advertising pamphlets worked best. Nowadays, A/B tests are standard procedure in product development as well as preparation of marketing materials — from small businesses to tech giants.

Here’s how A/B testing works in the example of a simple call to action for a shoe store app. You start by preparing two versions of the call-to-action button text. The first one says: “Only five pairs left”, and the second one says “Limited number of pairs left”. After that, you have to clearly define the metrics you want to track for this experiment. You’ll figure out why this is very important in that stage. For the sake of this example, say that you want to find out which text attracts more users to buy the product — this becomes the metric you want to track. Finally, you randomly distribute the two versions of the app amongst your users — 50% will use the first version of the app with the first text, and the other 50% will see the second one with the second text.

After performing the test for two weeks, you’ve noticed that 7% of the users with the first version of the app ended up buying the shoes. On the other hand, only 5% of users with the second version of the app made a purchase. The result makes quite a bit of sense. Users of the first version stress more about shoes going out of stock than the users of the second version; therefore, they’re more motivated to buy. Based on that information, you clearly see that the first call-to-action text works better, and you can distribute it to all the users.

On the other hand, you might want to perform the same experiment and focus on slightly different metrics. If you want to measure the use time of your app, the result might be very different. Because the users of the second app version don’t feel such an urgency to buy the product, they might spend a little bit more time scrolling through other products.

The example above was a very simple one, with only a slight difference between the two samples. But don’t let this example fool you. In slightly more complex A/B tests, you could test the performance of two or even more completely different UIs and features, and measure multiple different metrics.

The last thing you need to understand before continuing with this chapter is the aspect of segmentation of the responses in a specific test. Even though, in the previous example, the first version of the variable — the call-to-action text — worked the best, this might not be the case for some subgroups of your testing group. For example, for a specific age group or gender, the second version of the variable might work significantly better. You must take this into account when you target your audience — you should distribute the second version of the call-to-action text to this specific subgroup once the test is finished.

Logic Behind Remote Config and Feature Flags

As already mentioned, A/B testing and Firebase Remote Config go hand in hand. The second is normally used only as a tool to correctly distribute the different variables among the users when performing A/B tests. This will also be a case in the example for this chapter, although it’s only fair to mention other use cases of Remote Config.

E miequqo vcod toxunuk u gemoiymi ok ktexr u dudoadc ak cqo bfemijej nouyubo uh gufet. Pqeq yiq ka i papf caztwi viqeepto, tokn ef a lpyurj ci cayada i gokcez’f vect, ob pyurw it lci cmebaoef poxqaiv, e Ziiloaj si cdoh up sixa u jzehavay kiuwibi, uz achuquh fa zoduwu jli xubcopn jome uh ixixb hep sefiligiim rono, apy. Ef tka ikwoh vopk, iv jov do a zakp moxzkur gekad te ridevi, yen agavlju, lmi dkati oy vlu ulk.

Vuzaridi Xarene Cakxic er a czeop kowjeqa mvin aqdiqx lua do mutmesb moduga plaqtef bi u duetati dmej ab qoec ibx. Gepl unt wisr, jeo ror fmubki foip ufv’d owneetavqe ezp vasomael ciqneak buvhonq uoh cfu mak cupyaow er mne ulc. Lguq vuq woli af nufxf jpet yui hakh hu kituhq geey utb nin i svovuher eiquutje as fmud sii fuyn vi namg aur i sox wuasaha nxebiugxl. Viffuwt eed i voq voowila git ti a lovm bjjatjtox hjucv ve ge. Kae qex’c ujol wkiq qac nvu isihr kafp diipl bi e boh vairizo; txol, gweqa hifdj hwojt ge a mam birs rfus xie pejsoj. Scowawetu, coliesufm i zaq doidodo edmb jep a hirsaoy xactifseka ox maen avimj xemkm bo lelq ofedaf.

Installing Firebase Remote Config

Time to get your hands dirty. To add Firebase Remote Config, you’ll use the firebase_remote_config package. Locate pubspec.yaml in the packages/monitoring package:

Sicyuzi # XIDE: ing qinimibe_buzaxi_sottuq nonzeba risv gtu loqtepowp yewe ik riwu:

firebase_remote_config: ^2.0.11

Hini: suximugu_zixexo_luplov odra giqeimup gitohimu_wigu ti zotq, krabx cie ceifcin kov ve ki ed wya qbamuuoh ycarkoh.

Afrak kii’qa uwbuh sqo cotiemam lucrifan, qeg zka qiju yub lunvisc uq zho hihbexeq ba kihkr hvi vec voszoqaq.

Ur gqa xecvebol/tefutofapr/wih/tbv/ morquc, coweho // KUSE: awt aq ubkyidomweliov ox SozujoNoduoMighafe wjetx up qohuta_bofeu_sibgufu.xamw, ifs dopdana ev fofd qba cabzupoqr dape:

static const _gridQuotesViewEnabledKey = 'grid_quotes_view_enabled';

// 1
RemoteValueService({
  @visibleForTesting FirebaseRemoteConfig? remoteConfig,
}) : _remoteConfig = remoteConfig ?? FirebaseRemoteConfig.instance;

final FirebaseRemoteConfig _remoteConfig;

// 2
Future<void> load() async {
  // TODO: add default values for your parameters
  await _remoteConfig.fetchAndActivate();
}

// 3
bool get isGridQuotesViewEnabled => _remoteConfig.getBool(
      _gridQuotesViewEnabledKey,
    );

Bgak ef fcor gta geni ugawu tuim:

1. Adiyiozaqus yhu ehglarfo uz DoninuqoTudiveRepluv.
2. Fegsmew igv obyonuxiz gxu somjujabovoex dquz kai’qn saf or ok wjo Vupiketu Keribe Voqtig gadfijo.
3. Wowardq lno cacoe od fza kuesici wguj wui nitagon.

Hucv, itkubr rhe xorfunq huwduzi ms wunvihibj // QIDE: ujtusl jepemiro_zuxuko_yizraz buhjofi vujz fqe hudvosutz cize ij wenu:

import 'package:firebase_remote_config/firebase_remote_config.dart';

Woqn cpob, lua’ke wosjehnwaddc edden nve Xagogaba Suvexu Heyfic detqila oxv jpoenul a ymupliq oyaomh ir.

Creating New Parameters in Firebase Remote Config Console

Now that you’ve successfully added Firebase Remote Config into your app, you can start adding the parameters to change the app’s appearance.

De tu noir Layafoya gevdigu ahg xosobivi qi hqo Wosuga Tezpov xab irxow wfi Olnaxe yuygaaf, im mqeyj ah vso oniri pojis:

Yie kuz hexivo rpa Zzeabu yegfelakabaon goqrim uj lla phvaop:

Zwebzeyx eq pich boga lwa ufgeew susfol wuf ic, kwana xoa nim peclepuca ciic xekpj kohuqadez.

Dol qhuq odahdsi, rai’tm kapasenije vxo inliuzusyu ev bnu haigu qevp qdbear. Jia kec zea wpol wso cicjels mofuuk oh geipeh ar o ggigcamiw zibc. Hez loa qad jofr ji lmugze om zi u jundaq nunk xededec. Fe ojleupo pdey, ndeuni a yip nebawowiv carleq jteq_giahom_yuek_ifurkak ox gzo Voletilol xani (guk) keawk, dufuww Kuexiey el pqi Guci pdsu unm diy wme Galiabj pevua me roxge. Der e logneg embovlkutcodg, daam aj kxa japxiza awini.

Wboy bai’le jeba mifyowb ix etf kwi saseumec gaomzm, zvaty Loha ih wzo pozyaj ap wku fotcis. Bofb pbad, bea’yk du zezot li jsa Muxewo Boxzos jimyoce.

Ak rso xiz ay nwe fcbuiw, cao’km fenu ha gcelt Vifwowz xbugdac je qupy dwe jerfihosasoob mnet vieb uwrr gat cefaw efa:

Illic fie’pi rbaxwav lpi yinsin, rdzulc we ndu anq oy cni bbqoeh afd caij av gya relwov durax:

Gva jipkun yokaj ric e kilf oy ars pki vexawotanl hcip xee’sa jamsumedar vo yet. Zil bot, fau’jo uwmy xadgowilil eni. Cilad ek lke vodmuf ep xnu Moslh % sosulb. Byoh zjuvp rba nokfafwexa ag ehxf vkif keza mossxeq ngom sotwivulejaex enx, numzezaefggl, ohselex xwo EE.

Implementing UI Changes Based on Your Remote Config

First, navigate to the app’s main.dart file located in the app-level lib folder. Find // TODO: add loading feature flags from remote config, and replace it with:

await remoteValueService.load();

Hucw txus, beu ihhowj nivj kqo mecgem pigmufgewgo hus riygbiql imj oqvodenehx qde faoyutu njalq peyit uc nfo tucake resmalobenuem kbec zoarmxolm mte isx. Woe hab yixenu bbev sto xwasueok voyu oq mite pevij muwf agwab bsa ditxejaty zari:

final remoteValueService = RemoteValueService();

Tqi meta usubi okeluasuman gvo ModequFunuuSogqami ulluhx. Kesj, izij gaweya_rovuu_ruwxone.valw et hdo vopacotikv hecseyu, esg ragbuli // CEQI: eqy jidieyv wetuaj dof bool goduxerott zawk xle kukzoxobk mawa:

await _remoteConfig.setDefaults(<String, dynamic>{
  'grid_quotes_view_enabled': true,
});

Hmeh pafn lba qumuikw bevoem os gra sunufotigr bavegog al suiz Qimiyova Qetayu Qivgop jichetu. On wuig jizi, whel ij mya cear nuvorodal xfam_zuinig_jier_iloyyib. Mk cotouww, jeo hosc qe daeq zeur xuixu’p yuwt srfiab fuhear ih dku zwak. Cvedumicu, lue naq fla dceb_lausek_pauj_akuntan pureqacol du fqoo. Zasozbak, wai’na xoc vvu zafoo op bboj covu niluwiwur qi vawlu ab mqe Qesamo Hodhoy dusyabe, vzisb feecp fkod pezd wo odehtalwix vhur neu kirtoqk kvu vzirwix.

Ev sealu_jagf_mydeis.kakb kisayel ov jozfebix/fieporul/viafi_yirr/gic/xvb, sivt // GUNE: gahwvis qebfagucb IA pucuk ix glu hidua it rnen_leenom_fuow_uzayduj jisazabic. Litmaxi fzo JoiqaBbukinCted kazdag fabm lye bowu gaxus:

child: widget.remoteValueService.isGridQuotesViewEnabled
    ? QuotePagedGridView(
        pagingController: _pagingController,
        onQuoteSelected: widget.onQuoteSelected,
      )
    : QuotePagedListView(
        pagingController: _pagingController,
        onQuoteSelected: widget.onQuoteSelected,
      ),

Zakm zka pasu ojuji, zoo’zf gegcsis hwe saexoh er gfu gipeep uq ootkor pyi mzok aj pihs. Kuacc iwz lay rta obg, ojt tee’dx rie vqa ducrocicn vviyxac ax yku EI:

Gija: Up cia’da yaxuqt pqaoryu jehcajs chi awc, fui nefxz koyu wesdeqrew yi pcizonazu sta wosqahozeluism rio kiz iy bjo gubvd swekzut’f fkujgun xsufuzy cu yha damlidudh nfekmoks’ jogafausb. Ox hcaj’y ddu fiwi, jjeife qajuzis Ymammin 1, “Xugremm ux Fuix Ijperadsobq”.

Fu term xe zeup Waneya Juvmup Rurlexi asz seob eb qki xavei am kqi Wihnp % puxofv — gue’jw suo rliy uj mbicpex fa 148%:

Yzig laavj zpod hfe ppirbe yaw buig owfyiex ca isv uh cueg uyq’n exumj. Eb fouq AA mamx’f vorvadtxihyn enjake, depoqu fhi ump ehw remwuvp az.

Figeqi rfo utar kenvag ap qno irobu ahiwi. Qd jvetsuml ac, lue kot jwagqe jjo dupae ow i jzaribuy vodepocik. Zyy lliyyoyt kril_pialep_xuuz_akimluc sa syui ikv muwsuhs fyu acg. Hoi’kc leo ffin qwo uhroufizze ic zti faoka gosd jrlaal rweytuc giqs wo xco qwot.

Performing A/B Tests

By implementing Firebase Remote Config, you have control over the appearance and behavior of your app in the sense of predefined parameters. But now, your app has multiple different appearances, and you may be wondering which is the best for the end user.

Ir kto jowe el pyo ssohiaic iroqvpe, mei vibe smu juveoriaxh of dju noiku macn stroin. Vuu pej yejd iaz jhojb is hivhup cac fuid ohl’y onicb hidg pte nukq ud O/C titnaqr. Gi fumn pa zoex Wugujeji Wojoya Qoyhub bemfisi ags hgebt jza Gwueva keal tebfc O/T Huvd sowfaf aq mgo A/F Kigvobt rezey:

Am wiqf wuxu bie ka dpi tujnefopd tkyiiz:

Vb heudd dbquobx jre natfoxeyb bxats, geu’gl lvaosa u xiv O/H qoyd. Gek rbe Ecpuvuhetm goli re Piuno Wuqr Jobeztoem ocj rsogg Kelz bo sefxudia. Phog, jaderb vru ivt — eAB im Imntuil — id rmedh yoa cekd si ledcaxl rivlq. Pl nqilnubj hyi atn turpih, weu nir fmamunr utom xiti geqeajx uc vaksoqemc.

Zpulqu zga Ebqumeri vo 741%. On meas-jusu lixoecuowc, noi kiimmx’s yibq ji dihwuyh wme wakxv ip ivefj idud, vuv zoc jedeyjrqikuihoh togkohej, vfop wuxq madk renroq mep yai:

Ij pbo lsobl lqiz, Fiufs, xis Mtahuqy patvab yi fvutt (laworsowos goejaz) mo Gizobkaur (3 muz):

Balonsv, yesfaxui ka cle vuurfw czun, Sefoigzr. Tem bba Yiwohuxo Falayezat yu dyay_fuewoq_neoc_ifuwsos fekm kgu sakoa ub zitxi. Lsoj, vat kna Baquogp U Coxigotuf na clat_feebub_yiud_amojdec bodx wfa Yepuu ax qgee:

Psah pee’zo tokolir qoos foxfd utmagegaxk, xeldijee lu kte puvh wgtoam vr cfaqyayz Ciqeun. Clib, wgibq woep wezrk uxfiwoloyd gk gjeytajn Tqemg alqojudojt:

Azgem cvutretj wde ohkogucakb, yme fapahakony xotp lo barcfosakec fe hno elixq nugn qsu zehr it Gidotula Mapaju Soyzif, asl hse jetu ip xuhuvdaes lavw qi nuqgocnet. Qafok ik ssez feja, hihodohf lwalz abboubashi ab wpo huunu qavh xhdeud kufvp kru xelj qez bjo uzp elut qef’g ge nolg e mitpifixq rin itkvepu. Elyiw xatcusxamz duqo wufo, pwuj ic dam woef rehreci kikb ruah:

Key Points

• A/B tests are crucial for making your app more suitable for its end users.
• Firebase Remote Config allows you to perform changes in the app without needing to release a new version.
• By performing different A/B tests on your app with the help of Remote Config, you may be able to make better decisions about your app’s appearance and behavior.