Application of A/B Testing Experimentation on Government Digital Products to Enhance Teachers' Skills and Capabilities in Indonesia
DOI:
https://doi.org/10.26740/jpsi.v7n2.p67-75Keywords:
Microlearning, self-determined learning, BayesianAbstract
The Ministry of Education, Culture and Research and Technology (MoECRT) in Indonesia has developed a digital product named “Microlearning” within the Merdeka Mengajar digital Platform. “Microlearning” enables teachers to enhance their skills and capabilities by doing self-determined learning that is followed-up by the submission of Aksi-Nyata reports. We improved users’ journey flow and provided an alternative of user-interface designs in the platform, in which we hypothesized that these improvements could boost the number of teachers participating in the Aksi-Nyata report submissions - the main expected outcome of the “Microlearning” digital product. We conducted a randomized-control-trials experimentation in the digital platform to scientifically quantify its effect on the product, by including 1.6 million users who logged-in to the platform during a two weeks period. This experimentation (known as an A/B Testing) allowed us to simultaneously compare two groups within a specific period and within a controlled-environment. The bayesian-based experimentation revealed that the new developments increased the number of users who started to learn a topic, as well as increased the Aksi-Nyata report submission in the “Microlearning” product, as compared to the legacy-designs. A follow-up study showed even more convincing results, which shows the number of submitted Aksi-Nyata reports increased by 590% up to two months after implementing the analytics-based initiatives for all logged-in users in the platform.
References
Bryan, J., Wickham, H., Posit, P. B. C., Kalicinski, M., Valery, K., Leitienne, C., Colbert, B., Hoerl, D., Miller, E. 2022. Computer Software. [Online]. Available: https://cran.r-project.org/web/packages/readxl/index.html.
Harvard Business Review. 2017. A Refresher on A/B Testing. Cambridge. Last accessed on April 4, 2023 on page https://hbr.org/2017/06/a-refresher-on-ab-testing
Han, J., Yin, H., Boylan, M. 2016. Teacher Motivation: Definition, Research Development and Implications for Teachers. Cogent Education Vol. 3 Issue 1.
Ingarianti, T. M. 2017. Faktor-faktor yang Mempengaruhi Komitmen Karir. Jurnal Ilmiah Psikologi Terapan Universitas Gadjah Mada 5(2): 202-209.
Kemendikbud Ristek RI. 2022. Implementasi Kurikulum Merdeka. Jakarta. Last accessed on December 22, 2022 on page https://kurikulum.gtk.kemdikbud.go.id/.
Kominfo RI. 2020. Hasil Survei PBB, e-Government Indonesia Naik Peringkat. Jakarta. Last accessed on December 22, 2022 on page https://www.kominfo.go.id/content/detail/30024/hasil-survei-pbb-e-government-indonesia-naik-peringkat/0/artikel.
Pedersen, T. L., Wickham, H., Chang, W., Henry, L., Takahashi, K., Wilke, C., Woo, K., Yutani, H., Dunnington, D., Posit, P. B. C. 2021. Computer Software. [Online]. Available: https://cran.r-project.org/web/packages/ggplot2/index.html.
Portman, F. 2021. Computer Software. [Online]. Available: https://cran.r-project.org/web/packages/bayesAB/index.html.
Puza, Borek. 2015. Bayesian Methods for Statistical Analysis. Canberra: The Australian National University eView.
Robert, C. P. 2007. The Bayesian Choice: From Decision-Theoretic Foundations to Computational Implementation Second Edition. Paris: Springer Science & Business Media.
Samaniego, F. J. 2010. A Comparison of the Bayesian and Frequentist Approaches to Estimation. Paris: Springer Science & Business Media LLC.
The Signal. 2022. Users: New vs Returning. San Francisco. Last accessed on April 8, 2023 on page https://mixpanel.com/blog/new-vs-returning-users/.
UN Department of Economic and Social Affair. 2022. UN E-Government Survey 2022 The Future of Digital Government. Jakarta. Last accessed on December 22, 2022 on page https://desapublications.un.org/publications/un-e-government-survey-2022.
Vaughan, D., Wickham, H., Francois, R., Muller, K. 2022. Computer Software. [Online]. Available: https://cran.r-project.org/web/packages/dplyr/index.html.
Wei, William W. S. 2006. Time Series Analysis: Univariate and Multivariate Methods Second Edition. Philadelphia: Pearson Education, Inc.
Westland, J. C. 2022. A Comparative Study of Frequentist vs Bayesian A/B Testing in The Detection of E-commerce Fraud. Journal of Electric Business & Digital Economics Vol. 1 No. 1/2, pp. 3-23.
Wickham, H. 2022. Computer Software. [Online]. Available: https://cran.r-project.org/web/packages/tidyverse/index.html.
Xue, X., Fu, Y., Xia, S., Li, X. 2022. A/B Test Research on the Selection of Push Scheme in Network Precision Marketing. ISCTT 2022; 7th International Conference on Information Science, Computer Technology and Transportation, Xishuangbanna, China, 2022, pp. 1-6.
Yen, H. S., Lim, H. E., Campbell, J. K. 2015. Age and Productivity of Academics: A Case Study of a Public University in Malaysia. Malaysian Journal of Economic Studies 52(1): 97-116.
Additional Files
Published
Issue
Section
License
Retained Rights/Terms and Conditions of Publication
1. As an author you (or your employer or institution) may do the following:
- make copies (print or electronic) of the article for your own personal use, including for your own classroom teaching use;
- make copies and distribute such copies (including through e-mail) of the article to research colleagues, for the personal use by such colleagues (but not commercially or systematically, e.g. via an e-mail list or list server);
- present the article at a meeting or conference and distribute copies of the article to the delegates attending such meeting;
- for your employer, if the article is a ‘work for hire, made within the scope of your employment, your employer may use all or part of the information in the article for other intra-company use (e.g. training);
- retain patent and trademark rights and rights to any process, procedure, or article of manufacture described in the article;
- include the article in full or in part in a thesis or dissertation (provided that this is not to be published commercially);
- use the article or any part thereof in a printed compilation of your works, such as collected writings or lecture notes (subsequent to publication of the article in the journal); and prepare other derivative works, to extend the article into book-length form, or to otherwise re-use portions or excerpts in other works, with full acknowledgement of its original publication in the journal;
- may reproduce or authorize others to reproduce the article, material extracted from the article, or derivative works for the authors personal use or for company use, provided that the source and the copyright notice are indicated, the copies are not used in any way that implies the Department of Public Administration, Universitas Negeri Surabaya endorsement of a product or service of any employer, and the copies themselves are not offered for sale.
All copies, print or electronic, or other use of the paper or article must include the appropriate bibliographic citation for the articles publication in the journal.
2. Requests from third parties
Although authors are permitted to re-use all or portions of the article in other works, this does not include granting third-party requests for reprinting, republishing, or other types of re-use. Requests for all uses not included above, including the authorization of third parties to reproduce or otherwise use all or part of the article (including figures and tables), should be referred to the Department of Public Administration, Universitas Negeri Surabaya by going to our website at https://journal.unesa.ac.id/index.php/jpsi
3. Author Online Use
- Personal Servers. Authors and/or their employers shall have the right to post the accepted version of the article pre-print version of the article, or a revised personal version of the final text of the article (to reflect changes made in the peer review and editing process) on their own personal servers or the servers of their institutions or employers without permission from the Department of Public Administration, Universitas Negeri Surabaya, provided that the posted version includes a prominently displayed the Department of Public Administration, Universitas Negeri Surabaya copyright notice and, when published, a full citation to the original publication, including a link to the article abstract in the journal homepage. Authors shall not post the final, published versions of their papers;
- Classroom or Internal Training Use. An author is expressly permitted to post any portion of the accepted version of his/her own articles on the authors personal website or the servers of the authors institution or company in connection with the authors teaching, training, or work responsibilities, provided that the appropriate copyright, credit, and reuse notices appear prominently with the posted material. Examples of permitted uses are lecture materials, course packs, e-reserves, conference presentations, or in-house training courses;
- Electronic Preprints. Before submitting an article to JPSI, authors frequently post their manuscripts to their own website, their employers site, or to another server that invites constructive comments from colleagues. Upon submission of an article to JPSI, an author is required to transfer copyright in the article to the Department of Public Administration, Universitas Negeri Surabaya, and the author must update any previously posted version of the article with a prominently displayed the Department of Public Administration, Universitas Negeri Surabaya copyright notice. Upon publication of an article by the Department of Public Administration, Universitas Negeri Surabaya, the author must replace any previously posted electronic versions of the article with either (1) the full citation to the work with a Digital Object Identifier (DOI) or link to the article abstract in the JPSI homepage, or (2) the accepted version only (not the final, published version), including the Department of Public Administration, Universitas Negeri Surabaya copyright notice and full citation, with a link to the final, published article in journal homepage.
4. Articles in Press (AiP) service
The Department of Public Administration, Universitas Negeri Surabaya may choose to publish an abstract or portions of the paper before we publish it in the journal. Please contact our Production unit immediately if you do not want us to make any such prior publication for any reason, including disclosure of a patentable invention.
5. Author/Employer Rights
If you are employed and prepared the article on a subject within the scope of your employment, the copyright in the article belongs to your employer as a work-for-hire. In that case, the Department of Public Administration, Universitas Negeri Surabaya assumes that when you sign this Form, you are authorized to do so by your employer and that your employer has consented to the transfer of copyright, to the representation and warranty of publication rights, and to all other terms and conditions of this Form. If such authorization and consent have not been given to you, an authorized representative of your employer should sign this Form as the Author.
6. The Department of Public Administration, Universitas Negeri Surabaya Copyright Ownership
It is the formal policy of the Department of Public Administration, Universitas Negeri Surabaya to own the copyrights to all copyrightable material in its technical publications and to the individual contributions contained therein, in order to protect the interests of the Department of Public Administration, Universitas Negeri Surabaya, its authors and their employers, and, at the same time, to facilitate the appropriate re-use of this material by others. The Department of Public Administration, Universitas Negeri Surabaya distributes its technical publications throughout the world and does so by various means such as hard copy, microfiche, microfilm, and electronic media. It also abstracts and may translate its publications, and articles contained therein, for inclusion in various compendiums, collective works, databases and similar publications
Every accepted manuscript should be accompanied by a "Copyright Transfer Agreement" prior to the article publication.
Abstract views: 259
,
PDF Downloads: 0
