class: center, middle, inverse, title-slide # Binary R Packages for Linux: Past, Present and Future ## cran2copr project and bspm ### <p><span style="display:inline-block;width:100%"> <span style="float:left"> <br><strong>Iñaki Ucar</strong><br> .font80[Universidad Carlos III de Madrid | Fedora Project]<br> .font80[<a href="mailto:inaki.ucar@uc3m.es" class="email">inaki.ucar@uc3m.es</a> | <a href="mailto:iucar@fedoraproject.org" class="email">iucar@fedoraproject.org</a>] </span> <span style="float:right"> <br>Dirk Eddelbuettel<br> .font80[University of Illinois at Urbana-Champaign | Debian Project | R Project]<br> .font80[<a href="mailto:dirk@eddelbuettel.com" class="email">dirk@eddelbuettel.com</a> | <a href="mailto:edd@debian.org" class="email">edd@debian.org</a>] </span> </span></p> ### <p><br>Regular talk for <em>useR! 2021</em> <br>9 July 2021</p> --- # Motivation .pull-left-33[   ] .pull-right-66[.font120[ ## Comprehensive R Archive Network (CRAN) - More than **17,700** packages (!) - Many of them use compiled code (Fortran, C, C++...) - Compilation offers performance, integration, use of libraries... ]] -- .pull-right-66[.font120[ - Pick a package at random:<br>**80% chance of requiring compilation** - Up to 50% of the 1% most popular require compilation ]] -- .pull-right-66[.font120[ - Compiling requires an appropriate **toolchain** and **libraries** - Compiling requires a lot of **time** - Debugging build and compilation failures requires **skills** ]] --- # Motivation  -- <div style="position:absolute; bottom:20px; padding:5px; background-color:white; left:5%; font-weight:bold"> Windows & macOS desktop users </div>  -- <div style="position:absolute; bottom:20px; padding:5px; background-color:white; left:52%; font-weight:bold"> Linux desktop users </div>  -- <div style="position:absolute; bottom:50px; padding:5px; background-color:white; left:52%; font-weight:bold"> Cloud deployments, containers, CI/CD systems... </div> --- # We can do better .left-column[ ### Entire tidyverse<br><br>in **36 seconds**<br><br>on Fedora ] .right-column[  ] --- # Challenges .pull-left[.font120[ ## OS support - Many distributions - Different package managers - Different philosophies, conventions - Different update cycles <br> ## Scalability - Build and distribute thousands of packages - Accomodate constant growth at CRAN - Accomodate new platforms (hello, arm64) ]] .pull-right[.font120[ ## Dependency management - External dependencies - Static vs. shared (system libraries) <br><br><br> ## Integration - Built-in `install.packages` - Dependencies via system package manager - Need for admin privileges ]] --- # Related work .left-column[ ### Official repos ] .right-column[.font120[ Most distributions maintain R and a number of R packages. In Fedora, ```bash $ dnf repoquery --repo=fedora-source R-* | wc -l ``` ``` ## 384 ``` ]] -- .right-column[.font120[ .pros[ - Dependency management and multi-tenancy - Works out-of-the-box, no additional repos required ].cons[ - **Does not scale**: every R package requires a peer review (even for CRAN packages!) - **No integration**: mixed used of distro-specific package manager (with admin privileges) and `install.packages` ] ]] --- # Related work .left-column[ ### Official repos ### cran2deb<br>cran2obs ] .right-column[.font120[ - `cran2deb` started by Albrecht Gebhard, picked up by Dirk et al - Significantly extented and polished by Charles Blundell - Don Armstrong maintained a parallel setup called `debian-r` - **Michael Rutter** maintains the `c2d4u` repos, based on `cran2deb`, in Launchpad (~ 4.7k packages for several R versions) for Debian/Ubuntu - **Detlef Steuer** maintains the `autoCRAN` repo, based on `cran2obs`, in OBS (~16k packages) for openSUSE ]] -- .right-column[.font120[ .pros[ - Dependency management and multi-tenancy - Better scalability: scripts for automated conversion + PPA / OBS ].cons[ - **No integration**: mixed used of distro-specific package manager (with admin privileges) and `install.packages` ] ]] --- # Related work .left-column[ ### Official repos ### cran2deb<br>cran2obs ### RSPM ] .right-column[.font120[ - **RStudio's Public Package Manager**, announced on July 1, 2020 - Based on the proprietary product **RSPM** (RStudio Package Manager) ]] -- .right-column[.font120[ <br> .pros[ - Better OS support: builds for multiple distributions - Better scalability: builds probably based on Docker containers - Better integration: drop-in replacement for CRAN repo for `install.packages` ].cons[ - **No dependencies**: - system package manager still required to install system dependencies - version mismatches can happen ] ]] --- class: inverse, center, middle # Meet cran2copr... <div style="padding-bottom:120px">  <svg class="heart" viewBox="0 0 32 29.6" style="padding:65px"> <path d="M23.6,0c-3.4,0-6.3,2.7-7.6,5.6C14.7,2.7,11.8,0,8.4,0C3.8,0,0,3.8,0,8.4c0,9.4,9.5,11.9,16,21.2 c6.1-9.3,16-12.1,16-21.2C32,3.8,28.2,0,23.6,0z"/> </svg>  </div> # ... and bspm --- # Meet cran2copr .left-column[ ### A primer ] .right-column[ ```bash $ sudo dnf install 'dnf-command(copr)' $ sudo dnf copr enable iucar/cran $ sudo dnf install R-CoprManager ``` ] -- .right-column[ ```r install.packages("units") ``` ``` ## Install system packages as root... ## (1/3): R-CRAN-units-0.6.7-3.fc32.x86_64.rpm 5.2 MB/s | 787 kB 00:00 ## (2/3): R-CRAN-Rcpp-1.0.5-2.fc32.x86_64.rpm 2.0 MB/s | 2.0 MB 00:00 ## (3/3): udunits2-2.2.26-6.fc32.x86_64.rpm 420 kB/s | 617 kB 00:01 ## Preparing : 1/1 ## Installing : udunits2-2.2.26-6.fc32.x86_64 1/3 ## Installing : R-CRAN-Rcpp-1.0.5-2.fc32.x86_64 2/3 ## Installing : R-CRAN-units-0.6.7-3.fc32.x86_64 3/3 ## Running scriptlet: R-CRAN-units-0.6.7-3.fc32.x86_64 3/3 ## Verifying : R-CRAN-Rcpp-1.0.5-2.fc32.x86_64 1/3 ## Verifying : R-CRAN-units-0.6.7-3.fc32.x86_64 2/3 ## Verifying : udunits2-2.2.26-6.fc32.x86_64 3/3 ``` ] --- # Meet cran2copr .left-column[ ### A primer ### Under the hood ] .right-column[.font120[ - **cran2copr** uses the Fedora **Copr buildsystem** to build packages and publish RPM contributed repos - (Almost) the **entire CRAN available** (~17.3k packages and growing) - **Automated daily updates** via GitHub Actions - **Full integration** with `install.packages` via **bspm** (`CoprManager`) ] <br> | | Official repos | cran2deb | cran2obs | cran2copr | |------------:|:--------------:|:--------:|:--------:|:---------:| | OS support| ● | ● | ● | ● | | Dependencies| ✔ | ✔ | ✔ | ✔ | | Scalability| | ● | ✔ | ✔ | | Integration| | | | ✔ | ] --- # Meet bspm .left-column[ ### A primer ### Under the hood ### The magic ] .right-column[.font120[ - `bspm`: **Bridge to System Package Manager** - Extensible multi-distro integration (currently supports DNF and APT) - cran2copr's `CoprManager` is just a _branded_ version of `bspm` ]] -- .right-column[ <div style="margin-top:-35px"> .font120[ - Available on CRAN, and as Debian/Ubuntu package (`r-cran-bspm`) - Available in the **rocker** project (`rocker/r-bspm`) - Provides integration to other projects ] | | Official repos | cran2deb | cran2obs | cran2copr | |------------:|:-----------------------------------:|:-----------------------------------:|:-----------------------------------:|:---------:| | OS support| ● | ● | ● | ● | | Dependencies| ✔ | ✔ | ✔ | ✔ | | Scalability| | ● | ✔ | ✔ | | Integration| <span class='badge'>✔</span> | <span class='badge'>✔</span> | <span class='badge'>✔</span> | ✔ | </div> ] --- # Meet bspm <div class="mermaid" style="padding-top:20px"> flowchart LR subgraph Debian/Ubuntu cran2deb[cran2deb<br>5k packages] end style Debian/Ubuntu fill:#CE0056,stroke:none,color:white subgraph openSUSE cran2obs[cran2obs<br>16k packages] end style openSUSE fill:#73BA25,stroke:none,color:white subgraph Fedora cran2copr[cran2copr<br>17k packages] end style Fedora fill:#3c6eb4,stroke:none,color:white subgraph bspm[bspm package] subgraph system service backend --- apt backend --- dnf end enable["enable()"] is["install_sys(...)"] is --> |desktop/server<br>via D-Bus|backend is --> |containers<br>via root/sudo|backend end style bspm fill:none subgraph R[R session] ip["install.packages(...)"] ip -.-o enable ip --> is is --> |not available|ip end apt --> cran2deb dnf --> cran2obs dnf --> cran2copr </div> --- # Example: rspm vs. bspm .left-column[ ### Missing dependencies ] .right-column[ <video id="video" src="img/RSPMvsFedoraBSPM.webm"></video> <span class="font80" style="position:relative;top:-540px;"> See another cool Ubuntu-based animation at https://dirk.eddelbuettel.com/blog/2020/08/26/ </span> ] --- # Example: geospatial stack via bspm .left-column[ `rocker/geospatial`: - 25 packages Dependencies: - 139 packages - 69 system libraries Size: - ~300 MB download - ~800 MB on disk ### **68 seconds** ] .right-column[  ] --- # Thanks .pull-left[ [](https://twitter.com/rdataberlin/status/1408379823970557954) ] .pull-right[ [](https://twitter.com/rmflight/status/1409684842703327234) Links: - https://github.com/cran4linux/cran2copr - https://github.com/cran4linux/bspm - https://github.com/rocker-org/rocker/tree/master/r-bspm ] --- class: inverse, center, middle <div style="margin-top:-65px">  <svg class="heart" viewBox="0 0 32 29.6" style="padding:65px"> <path d="M23.6,0c-3.4,0-6.3,2.7-7.6,5.6C14.7,2.7,11.8,0,8.4,0C3.8,0,0,3.8,0,8.4c0,9.4,9.5,11.9,16,21.2 c6.1-9.3,16-12.1,16-21.2C32,3.8,28.2,0,23.6,0z"/> </svg>  </div> --- # Appendix: installation instructions .left-column[ <h3 style="color:#3c6eb4">Fedora:</h3> <h3 style="color:#CE0056;padding-top:45px">Ubuntu:</h3> <h3 style="color:#73BA25;padding-top:90px">openSUSE:</h3> <h3 style="color:#1A292C;padding-top:125px">then...</h3> ] .right-column[ <div class="fedora"> ```bash $ sudo dnf install 'dnf-command(copr)' $ sudo dnf copr enable iucar/cran $ sudo dnf install R-CoprManager ``` </div> <div class="debian"> ```bash $ sudo add-apt-repository ppa:marutter/rrutter4.0 # R v4.0 and higher $ sudo add-apt-repository ppa:c2d4u.team/c2d4u4.0+ # R packages $ sudo apt-get update $ sudo apt-get install r-cran-bspm python3-{dbus,gi,apt} $ echo "bspm::enable()" | sudo tee -a /etc/R/Rprofile.site ``` </div> <div class="opensuse"> ```bash $ sudo zypper ar -r https://download.opensuse.org/repositories/devel:/languages:/R:/patched/openSUSE_Tumbleweed/devel:languages:R:patched.repo $ sudo zypper ar -r https://download.opensuse.org/repositories/devel:/languages:/R:/autoCRAN/openSUSE_Tumbleweed/devel:languages:R:autoCRAN.repo $ sudo zypper install R-patched python3-dnf python38-dbus-python python3-gobject $ sudo ln -s /etc/zypp/repos.d /etc/yum.repos.d $ sudo Rscript -e 'install.packages("bspm", repos="https://cran.r-project.org")' $ echo "bspm::enable()" | sudo tee -a /usr/lib64/R/etc/Rprofile.site ``` </div> ```r install.packages(...) # enjoy! ``` ] --- # Appendix: installation instructions .left-column[ ### Other ] .right-column[.font120[ ### General procedure If you are trying `bspm` in another distro, or you are packaging it as a system package, please refer to the [general procedure](https://github.com/cran4linux/bspm#general-procedure). ### Containerized applications (e.g., a Docker images) Refer to the [`rocker/r-bspm` images](https://github.com/rocker-org/rocker/tree/master/r-bspm). ]]