Scientific computing

(for the rest of us)

Environment and projects

In this previous module, we did not load a single package: everything we wanted to do was provided by Julia “out of the box”. In most applications, we will need to get functionalities from other packages, and this is where Julia’s package manager shines.

There are two families of packages – some are part of the standard library, also known as Base, which is documented in the manual; others are developped by members of the community. Together, these packages form the broad “Julia ecosystem”.

This module is usually delivered as an interactive demo. It doesn’t work quite as well as text, but the documentation is still largely enough to get you up to speed. Again, reading the documentation is a core way to build up your skills.

The Julia package manager (also known as Pkg) is a complex but really well documented beast. For this reason, this module is primarily meant to point you towards the documentation, for everything related to installing and updating packages.

Nevertheless, there is one fundamental concept that is worth saying very explicitely: never work in the default environment. In most cases, this means that you will navigate to your working folder, and call Julia using

julia --project

Alternatively, after starting Julia, you can drop to the Pkg mode of the REPL by pressing ], and calling the activate command. Most of the time, this will be done as activate ., which means “activate an environment in the current directory”.

The . notation is a shortcut to mean here, .. means “the folder immediately above here”, and ~ means “home”. In the vast majority of cases, we will navigate our fileystem using Julia’s pathing utilities.

Of course, because Pkg is a package, it can be called programmatically, i.e. as part of a script:

import Pkg

It offers the same commands as the package mode from the REPL; for example, we can activate an environemnt in a temporary directory with a nifty little trick:

Pkg.activate(mkpath(tempname()))

  Activating new project at `/tmp/jl_L45WnEhuXi`

This draws on concepts we will see (much) later, during the module on working with files and paths.

We can query the status of this environment:

Pkg.status()

Status `/tmp/jl_L45WnEhuXi/Project.toml` (empty project)

This project is currently empty, meaning that we have not added dependencies (i.e. packages we will use). Let’s fix this, by installing the Distributions package:

Pkg.add("Distributions")

    Updating registry at `~/.julia/registries/General`
    Updating git-repo `https://github.com/JuliaRegistries/General.git`
   Resolving package versions...
    Updating `/tmp/jl_L45WnEhuXi/Project.toml`
  [31c24e10] + Distributions v0.25.79
    Updating `/tmp/jl_L45WnEhuXi/Manifest.toml`
  [49dc2e85] + Calculus v0.5.1
  [d360d2e6] + ChainRulesCore v1.15.6
  [9e997f8a] + ChangesOfVariables v0.1.4
  [34da2185] + Compat v4.4.0
  [9a962f9c] + DataAPI v1.13.0
  [864edb3b] + DataStructures v0.18.13
  [b429d917] + DensityInterface v0.4.0
  [31c24e10] + Distributions v0.25.79
  [ffbed154] + DocStringExtensions v0.9.2
  [fa6b7ba4] + DualNumbers v0.6.8
  [1a297f60] + FillArrays v0.13.5
  [34004b35] + HypergeometricFunctions v0.3.11
  [3587e190] + InverseFunctions v0.1.8
  [92d709cd] + IrrationalConstants v0.1.1
  [692b3bcd] + JLLWrappers v1.4.1
  [2ab3a3ac] + LogExpFunctions v0.3.19
  [e1d29d7a] + Missings v1.0.2
  [77ba4419] + NaNMath v1.0.1
  [bac558e1] + OrderedCollections v1.4.1
  [90014a1f] + PDMats v0.11.16
  [21216c6a] + Preferences v1.3.0
  [1fd47b50] + QuadGK v2.6.0
  [189a3867] + Reexport v1.2.2
  [79098fc4] + Rmath v0.7.0
  [a2af1166] + SortingAlgorithms v1.1.0
  [276daf66] + SpecialFunctions v2.1.7
  [82ae8749] + StatsAPI v1.5.0
  [2913bbd2] + StatsBase v0.33.21
  [4c63d2b9] + StatsFuns v1.0.1
  [efe28fd5] + OpenSpecFun_jll v0.5.5+0
  [f50d1b31] + Rmath_jll v0.3.0+0
  [0dad84c5] + ArgTools v1.1.1
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [f43a241f] + Downloads v1.6.0
  [7b1f6079] + FileWatching
  [b77e0a4c] + InteractiveUtils
  [b27032c2] + LibCURL v0.6.3
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [ca575930] + NetworkOptions v1.2.0
  [44cfe95a] + Pkg v1.8.0
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA v0.7.0
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays
  [10745b16] + Statistics
  [4607b0f0] + SuiteSparse
  [fa267f1f] + TOML v1.0.0
  [a4e569a6] + Tar v1.10.1
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [e66e0078] + CompilerSupportLibraries_jll v0.5.2+0
  [deac9b47] + LibCURL_jll v7.84.0+0
  [29816b5a] + LibSSH2_jll v1.10.2+0
  [c8ffd9c3] + MbedTLS_jll v2.28.0+0
  [14a3606d] + MozillaCACerts_jll v2022.2.1
  [4536629a] + OpenBLAS_jll v0.3.20+0
  [05823500] + OpenLibm_jll v0.8.1+0
  [83775a58] + Zlib_jll v1.2.12+3
  [8e850b90] + libblastrampoline_jll v5.1.1+0
  [8e850ede] + nghttp2_jll v1.48.0+0
  [3f19e933] + p7zip_jll v17.4.0+0

Note that the package manager informs us on the specific dependencies of the package we want to install. This management of the network of dependencies is largely invisible to the user, but is reported at install (and removal) time.

We can check that it has been installed:

Pkg.status()

Status `/tmp/jl_L45WnEhuXi/Project.toml`
  [31c24e10] Distributions v0.25.79
The reason we can do import Pkg without installing Pkg first is that Pkg is a part of the standard library, and comes bundled with Julia. It is, in fact, one of the most central pieces of the language.

We can now use the Distributions package:

using Distributions
Normal(0.0, 0.2)

Distributions.Normal{Float64}(μ=0.0, σ=0.2)

In some cases, we may want to remove an existing package from our environment, because we have found an alternative solution. It is good practice to only keep the dependencies you actually need and use.

Pkg.rm("Distributions");
Pkg.status("Distributions")

    Updating `/tmp/jl_L45WnEhuXi/Project.toml`
  [31c24e10] - Distributions v0.25.79
    Updating `/tmp/jl_L45WnEhuXi/Manifest.toml`
  [49dc2e85] - Calculus v0.5.1
  [d360d2e6] - ChainRulesCore v1.15.6
  [9e997f8a] - ChangesOfVariables v0.1.4
  [34da2185] - Compat v4.4.0
  [9a962f9c] - DataAPI v1.13.0
  [864edb3b] - DataStructures v0.18.13
  [b429d917] - DensityInterface v0.4.0
  [31c24e10] - Distributions v0.25.79
  [ffbed154] - DocStringExtensions v0.9.2
  [fa6b7ba4] - DualNumbers v0.6.8
  [1a297f60] - FillArrays v0.13.5
  [34004b35] - HypergeometricFunctions v0.3.11
  [3587e190] - InverseFunctions v0.1.8
  [92d709cd] - IrrationalConstants v0.1.1
  [692b3bcd] - JLLWrappers v1.4.1
  [2ab3a3ac] - LogExpFunctions v0.3.19
  [e1d29d7a] - Missings v1.0.2
  [77ba4419] - NaNMath v1.0.1
  [bac558e1] - OrderedCollections v1.4.1
  [90014a1f] - PDMats v0.11.16
  [21216c6a] - Preferences v1.3.0
  [1fd47b50] - QuadGK v2.6.0
  [189a3867] - Reexport v1.2.2
  [79098fc4] - Rmath v0.7.0
  [a2af1166] - SortingAlgorithms v1.1.0
  [276daf66] - SpecialFunctions v2.1.7
  [82ae8749] - StatsAPI v1.5.0
  [2913bbd2] - StatsBase v0.33.21
  [4c63d2b9] - StatsFuns v1.0.1
  [efe28fd5] - OpenSpecFun_jll v0.5.5+0
  [f50d1b31] - Rmath_jll v0.3.0+0
  [0dad84c5] - ArgTools v1.1.1
  [56f22d72] - Artifacts
  [2a0f44e3] - Base64
  [ade2ca70] - Dates
  [f43a241f] - Downloads v1.6.0
  [7b1f6079] - FileWatching
  [b77e0a4c] - InteractiveUtils
  [b27032c2] - LibCURL v0.6.3
  [76f85450] - LibGit2
  [8f399da3] - Libdl
  [37e2e46d] - LinearAlgebra
  [56ddb016] - Logging
  [d6f4376e] - Markdown
  [ca575930] - NetworkOptions v1.2.0
  [44cfe95a] - Pkg v1.8.0
  [de0858da] - Printf
  [3fa0cd96] - REPL
  [9a3f8284] - Random
  [ea8e919c] - SHA v0.7.0
  [9e88b42a] - Serialization
  [6462fe0b] - Sockets
  [2f01184e] - SparseArrays
  [10745b16] - Statistics
  [4607b0f0] - SuiteSparse
  [fa267f1f] - TOML v1.0.0
  [a4e569a6] - Tar v1.10.1
  [8dfed614] - Test
  [cf7118a7] - UUIDs
  [4ec0a83e] - Unicode
  [e66e0078] - CompilerSupportLibraries_jll v0.5.2+0
  [deac9b47] - LibCURL_jll v7.84.0+0
  [29816b5a] - LibSSH2_jll v1.10.2+0
  [c8ffd9c3] - MbedTLS_jll v2.28.0+0
  [14a3606d] - MozillaCACerts_jll v2022.2.1
  [4536629a] - OpenBLAS_jll v0.3.20+0
  [05823500] - OpenLibm_jll v0.8.1+0
  [83775a58] - Zlib_jll v1.2.12+3
  [8e850b90] - libblastrampoline_jll v5.1.1+0
  [8e850ede] - nghttp2_jll v1.48.0+0
  [3f19e933] - p7zip_jll v17.4.0+0
[ Info: We haven't cleaned this depot up for a bit, running Pkg.gc()...
      Active manifest files: 2 found
      Active artifact files: 57 found
      Active scratchspaces: 0 found
     Deleted no artifacts, repos, packages or scratchspaces
Status `/tmp/jl_L45WnEhuXi/Project.toml` (empty project)
The package manager would report the dependencies that have been removed, but we silenced this by adding ; at the end of the line. This works for all commands, and is a great way to prevent some output from taking up too much space.

Another thing that the package manager can do is garbage collection, i.e. cleaning the versions and environments that are not used anymore:

Pkg.gc()

      Active manifest files: 2 found
      Active artifact files: 57 found
      Active scratchspaces: 0 found
     Deleted no artifacts, repos, packages or scratchspaces

This will look through all the environments Julia knows about, and reclaim disk space by removing versions of packages that are not used anymore, or dependencies of projects that have not been used for a long time.

There is a lot more that you can achieve using the package manager, including creating your own packages. Reading the documentation is key here, and will make maintaining your projects far more efficient.